_estimate_mtp_overhead_bytes is also reached for the separate-drafter spec modes
(draft-simple / draft-eagle3) through _user_draft_via_extras. Those modes load a
small distinct drafter with its own KV -- already counted in the draft KV +
weights -- and keep no duplicated full target context; only MTP runs a second
context over the target model's own KV geometry (llama.cpp ctx_tgt). Charging the
~main-KV-sized f16 copy there over-reserved by tens of GiB on an MLA model and
needlessly shrank the advertised context, the same under-advertising #6312 set
out to fix.
Thread mtp_keeps_target_ctx through _estimate_mtp_overhead_bytes (True for MTP,
False for separate-drafter modes) and derive _engaged_is_mtp at the fit call site
so the target copy is added only when the engaged mode is actually MTP. MLA + MTP
(GLM-5.2 / DeepSeek / Kimi) is unchanged, so the GLM-5.2 OOM fix is preserved;
non-MLA and the draft-simple / draft-eagle3 paths no longer pay the copy.
test_mtp_mla_target_ctx.py adds a case asserting the separate-drafter reserve
collapses to the draft KV (no target copy) while the default MTP path keeps it.
* Studio: Auto disables MTP for MLA models (GLM-5.2 et al.); UNSLOTH_MLA_MTP_ENABLED to re-enable
Studio's Auto speculative mode promotes any embedded-MTP model >=3B to
--spec-type draft-mtp. For MLA models (GLM-5.2/DeepSeek/Kimi) that is a
regression: llama.cpp's MLA/DSA MTP path keeps a duplicated full target-KV
context and recomputes the sparse-attention indexer every draft step, so it
runs ~2x slower than no speculation (GLM-5.2 UD-IQ1_S bench: 27 vs 45 tok/s,
flat across draft depth 1..6 and 96-100% acceptance, on both prose and code).
vLLM/SGLang get a speedup from the same model, so this is a llama.cpp
implementation gap, not a model property.
Auto now drops embedded MTP for MLA models and falls back to ngram-mod (or
spec-off when the binary lacks ngram-mod), mirroring the existing sub-3B
fallback. The metadata separator is kv_lora_rank: it is present on MLA models
and absent on non-MLA embedded-MTP models (Qwen3.x-MTP), whose MTP module is
structurally identical but fast, so a "full layer" heuristic cannot tell them
apart. Qwen MTP, separate drafters (Gemma, --model-draft), and non-MTP models
are unchanged.
Explicit overrides still engage the slower MTP route: choosing MTP / MTP+Ngram
in Settings, or passing --spec-type in extra args. UNSLOTH_MLA_MTP_ENABLED=1
re-enables Auto promotion for MLA once the upstream path is optimized.
A new spec_fallback_reason value "mla_mtp_disabled" surfaces this as an
Auto-mode policy downgrade (not a binary/update problem), with a settings
banner that points users at the MTP override. It is deliberately kept out of
the "Update llama.cpp" affordance since updating does not help.
Tests: resolver-matrix rows for MLA->ngram-mod / MLA-no-ngram->off /
non-MLA-Qwen->draft-mtp / MLA-separate-drafter->draft-mtp /
non-MTP-MLA->default / forced mtp|mtp+ngram on MLA->draft-mtp / env flag;
kv_lora_rank metadata fixtures; and reload-skip coverage (Auto ngram-mod is
idempotent, forced mtp bounces a reload).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: reserve the duplicated MTP target KV context for MLA models
GLM-5.2 UD-IQ1_S advertised its native 1,048,576-token context, loaded, then
crashed cublasCreate on the first generation with "CUDA error: the resource
allocation failed" on a 2x B200 box. The model loaded fine; the decode OOMed.
Cause: when MTP speculative decoding is engaged, llama.cpp keeps a second full
copy of the target model's KV context for draft verification (ctx_tgt=yes in the
spec log), at f16. On an MLA model that copy is ~the main KV again -- for GLM-5.2
at 1M ctx llama.cpp sized it at ~97.5 GiB -- but the auto-fit reserve only
counted the tiny embedded draft head (~2 GiB), 46x too low. So weights (~202 GiB)
+ main KV (~83 GiB) + a 2 GiB reserve looked like it fit in 2x182 GiB, when the
real footprint with the ~97 GiB MTP copy is ~382 GiB and overruns the cards.
Disabling speculative decoding removed the copy and the same context ran fine.
_estimate_mtp_overhead_bytes now adds the duplicated target context (the main KV
re-estimated at f16) for MLA models, so auto-fit backs the context off (or selects
more GPUs) instead of advertising one that OOMs. It is gated strictly on MLA
(kv_lora_rank present), which is exactly the family that keeps the extra copy
(GLM-5.x, DeepSeek, Kimi-K2); non-MLA MTP (Qwen, Gemma) is byte-for-byte
unchanged. The reserve stays deterministic from GGUF dims, matching #6312.
test_mtp_mla_target_ctx.py covers it: the MLA reserve includes the f16 target
copy and dominates the draft head, the copy is f16 regardless of the main cache
type and scales with context, non-MLA embedded heads keep overhead == draft KV,
and _fit_context_to_vram on the GLM-5.2 / 2x B200 budget now returns a context
below the requested 1M where the old draft-only reserve kept the full 1M.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stub structlog/loggers in MTP-MLA test so it is import-order-independent
The new test imports core.inference.llama_cpp, which pulls in orchestrator ->
structlog. In the lightweight test env structlog is absent, so when this file is
collected before test_mtp_vram_budget.py (it sorts first) or run directly,
collection aborted with ModuleNotFoundError. Install the same loggers/structlog
(+ conditional httpx) stubs the sibling MTP tests use before the import, matching
the established per-file convention.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Bypass Permissions menu freeze and show decimal GB for model sizes
Bypass Permissions freeze: the warning dialog lived inside the composer
"+"/More dropdown and kept the menu mounted via onSelect preventDefault,
so confirming or cancelling the dialog left both popovers frozen open.
Lift the dialog out of the menu into a store-driven
BypassPermissionsConfirmDialog mounted at a stable spot in the composer.
The menu item now closes normally on select and just toggles a new
bypassConfirmOpen store flag, so the popovers dismiss as expected.
Model search sizes: formatBytes divided bytes by 1024 but labelled the
result "GB", so unsloth/GLM-5.2-GGUF:UD-IQ1_S showed 201.8 GB where
Hugging Face reports 217 GB. Switch the search display to decimal
(base-1000) units to match what Hugging Face reports. The GPU-fit math
stays base-1024 since VRAM capacity is binary.
* Studio: address review feedback and add GLM-5.2 high/max/disabled thinking
Review feedback on the Bypass Permissions and size-format changes:
- Mount the Bypass Permissions warning dialog once at the chat-page root
instead of inside each Composer. It is driven by global store state, so
the per-composer mount meant Compare mode (multiple composers) rendered
duplicate dialogs and the shared-composer menu had none. A single root
mount fixes both.
- Defer opening the dialog past Radix's menu-close focus restoration with
setTimeout(0), so the dropdown does not steal focus back and break the
dialog's focus trap.
- Clamp the unit index in formatBytes so units[i] cannot go out of bounds
past TB (and to absorb log() float error at exact powers of 1000).
GLM-5.2 reasoning levels:
GLM-5.2's template gates thinking with enable_thinking and also reads a
reasoning_effort level ('high' or 'max'), so it needs high / max /
disabled rather than the binary toggle it got before (its style was
detected as enable_thinking, which made 'high' unreachable). Add a new
reasoning style 'enable_thinking_effort' that reuses the effort dropdown
but, unlike gpt-oss, can be fully disabled:
- detect_reasoning_flags classifies a template that has both
enable_thinking and reasoning_effort, extracting the discrete levels
from the quoted effort literals it branches on. Templates with only one
of the two (gpt-oss, Qwen3, DeepSeek, GLM-4.6) are unchanged.
- _request_reasoning_kwargs maps the new style to enable_thinking plus an
in-range reasoning_effort; disabling sends enable_thinking=false. The
gpt-oss reasoning_effort path is left untouched.
- The backend reports reasoning_effort_levels on the load/status response;
the frontend carries them through to the effort dropdown and sends
enable_thinking + reasoning_effort for this style.
Verified: backend reasoning kwargs render the real GLM-5.2 template to
"Reasoning Effort: High/Max" (thinking) and an empty <think></think>
(disabled); tsc, eslint, i18n parity and the production build all pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback on reasoning effort and formatBytes
- chat-adapter localReasoningEffort: accept 'minimal' so a template that
branches on it (extracted into reasoning_effort_levels) is sent through
instead of being coerced to 'low' and then dropped by the backend.
- formatBytes: return '0 B' for non-finite / non-positive sizes (missing
metadata -> NaN, Infinity, negatives) and clamp the unit index lower
bound to 0, so sub-1-byte values can't produce a negative index.
* Studio: hybrid reasoning none gate and decimal GB in load progress
- _request_reasoning_kwargs: for enable_thinking_effort models, treat a
raw reasoning_effort='none' (OpenAI 'no reasoning' sentinel) as the
enable_thinking=false off gate, so a direct API caller can disable
thinking even without passing enable_thinking. The frontend already
sends enable_thinking=false; this only affects raw API callers.
- use-chat-model-runtime: the download / 'X of Y GB in memory' load
progress divided bytes by 1024**3 but labelled GB, so it disagreed with
the model picker and Hugging Face. Use decimal GB (1e9) to match.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry hybrid reasoning levels on all load paths and harden formatBytes
Review follow-ups on the enable_thinking_effort work:
- Every model-load path now copies reasoning_effort_levels and derives
supportsReasoningOff, via a shared reasoningCapsFromLoad() helper. The
shared/Compare composer load and the three chat-adapter auto-load paths
previously set only reasoningStyle, so a GLM-style hybrid model loaded
through Compare or first-chat auto-load fell back to the default
low|medium|high and lost its Max / Off controls.
- The local send path clamps the effort to the loaded model's advertised
levels (clampReasoningEffortToLevels) instead of a hard-coded list. A
stale "max" carried over from an external provider no longer reaches a
pure reasoning_effort (gpt-oss) model that only accepts none|low|medium|
high, where the backend would have dropped it.
- formatBytes divides iteratively instead of via Math.log, which has float
error at exact powers of 1000 (log(1e12)/log(1000) = 3.9999... would
label 1 TB as "1000 GB"). Keeps the non-finite/non-positive guard.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fall back to text-only when a vision projector hard-crashes llama-server
The text-only mmproj fallback (#6075) only fired when llama-server printed a
recognizable projector-format error ("Unknown projector type", exit -6). An
installed llama.cpp that predates a model's projector can instead SIGSEGV
(exit -11) with no parseable output, e.g. unsloth/Qwen3.5-4B-MTP-GGUF +
mmproj-F16 on an older gfx1151 prebuilt: llama-server crashes on load, the
--fit off retry crashes the same way, and load_model gives up with a hard 500
instead of dropping vision.
Generalize the decision: a vision (--mmproj) launch killed by a signal (POSIX
returncode < 0, e.g. -11 SIGSEGV / -6 SIGABRT; Windows 0xC0000000+ access
violation) is treated like a projector incompatibility, so the load retries
once text-only. The retry is skipped if a cancel/unload is pending, mirroring
the MTP guard. Clean non-zero exits (bad GGUF, port bind) and hung processes
keep their own handling; non-vision launches are unaffected.
Reproduced and verified on gfx1151 (Radeon 8060S, ROCm 7.2.1): a current
prebuilt (llama.cpp b9596) loads the exact model + args fine, confirming the
crash is a stale prebuilt. With a wrapper that SIGSEGVs on --mmproj, Studio now
recovers: the load returns 200 (is_vision=false) and serves at ~31 tok/s
text-only instead of failing. New _is_signal_crash helper plus tests pin the
decision.
Also normalize a few em-dashes to ASCII punctuation in existing comments.
* Studio: refine mmproj hard-crash fallback (signal scope + last argv)
- Limit _is_signal_crash to genuine program faults (SIGSEGV, SIGABRT,
SIGILL, SIGFPE, SIGBUS) and Windows 0xC0000000+ statuses. SIGKILL,
SIGTERM and SIGINT no longer count, so an OOM-killer, unload or
supervisor kill is not masked as a projector incompatibility.
- Strip --mmproj from the last attempted argv so the text-only retry
keeps --fit off / --spec-default instead of resurrecting the original
spec flags (matters for MTP vision models on an older llama.cpp).
- Drop stray temp files committed by mistake and gitignore the "~" dir
so they cannot be re-added.
* Studio: tighten comments in mmproj hard-crash fallback
* Studio: retry --flash-attn off before dropping vision on a startup crash
When llama-server hard-crashes at startup, the recovery chain now tries the
least-destructive mitigation first. Flash-attention kernels SIGSEGV at load on
some ROCm/GPU builds (often inside the vision tower's attention); disabling
flash attention keeps BOTH vision and MTP, so a hard program fault with
--flash-attn on now retries once with --flash-attn off before the MTP-drop or
the text-only (mmproj-strip) fallbacks. _is_signal_crash already gates this to
genuine faults (SIGSEGV/SIGABRT/SIGILL/SIGFPE/SIGBUS), so an OOM-kill or unload
(SIGKILL/SIGTERM/SIGINT) does not trigger a retry.
Field context: a gfx1151 user crashes loading a vision GGUF even on the latest
prebuilt, so an update cannot help, and the same model and args load fine on
another gfx1151 box, pointing at a runtime/flash-attn fault. New
_with_flash_attn_off helper plus tests. Verified on hardware with a wrapper
that SIGSEGVs on --flash-attn on: Studio recovers with is_vision=true (vision
and MTP intact) instead of failing or losing vision.
* Studio: name the OOM kill on a too-large model load
When the OS kills llama-server with no diagnostic output (SIGKILL/SIGTERM,
almost always the OOM killer, e.g. a BF16 model too large for the WSL VM's
RAM cap), the recovery ladder correctly does not retry an external kill, so
this is the message the user sees. It fell through to the generic "is the
GGUF valid / out of memory" text. Make it actionable: name the signal and
point at a smaller or more quantized GGUF, a lower context length, or raising
the WSL memory limit. Output-based diagnoses still win and a hard fault keeps
the generic fallback.
* Studio: refuse a model too large for system RAM on a unified-memory APU
On gfx1150/gfx1151 APUs the weights load into shared system RAM (GGML
unified memory). _get_gpu_free_memory reports the full ROCm/APU budget as
free (often ~100 GB), but under WSL the VM's RAM cap is the real ceiling.
Studio trusted the budget, spawned a load larger than RAM, and the OS killed
it mid-flight, taking the Studio process with it (a silent "Terminated" with
no error, the model resident in RAM not VRAM).
Add a pre-flight guard on the APU path: if the weights exceed available
system RAM (psutil, then /proc/meminfo), refuse before spawning with a clear
message (smaller/more-quantized GGUF, lower context, or raise the WSL memory
limit). Weights only so KV/context auto-reduction is not double-counted;
unknown RAM never refuses; non-APU and discrete-GPU paths are untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine recovery ladder (keep diagnosed errors, flip all flash-attn)
Two review points on the hard-crash recovery ladder:
1. The signal-only text-only fallback stripped --mmproj on any hard fault,
even when llama-server had already printed a non-projector cause (an OOM
such as "cudaMalloc failed: out of memory", an unsupported architecture, or
a tensor-parallel limit). That masked the real error and told the user to
update llama.cpp for vision. New _output_has_nonprojector_diagnostic gates
the signal path: it fires only when no such marker is present, so a bare
SIGSEGV with no output still retries text-only, but a diagnosed OOM surfaces
the real error instead of silently dropping vision.
2. _with_flash_attn_off only flipped the first --flash-attn. llama.cpp is
last-wins, so a leftover enable from extra_args (--flash-attn on, -fa on, or
the = form) could keep flash attention on and re-crash the retry. It now
flips every occurrence and returns None only when nothing is flippable.
test_llama_cpp_mmproj_fallback.py and the classification/APU suites: 103 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pass the text-only retry's exit code to the failure classifier
When the text-only fallback retry itself fails, read its exit code before
_kill_process() clears it and forward it to _classify_llama_start_failure, so an
OS-killed retry surfaces the actionable out-of-memory message instead of the
generic one (matching the primary failure path).
* Studio: scope APU RAM guard to selected GPUs, count MTP drafter, neutral SIGTERM
Three refinements to the startup recovery work in this PR:
- The unified-memory APU RAM guard fired whenever any visible GPU was a
gfx1150/gfx1151 APU, so on a mixed APU+dGPU host it could refuse a valid
load placed on the discrete GPU. Scope _amd_apu_wants_unified_memory to the
selected gpu_indices (physical ids, mapped via CUDA_VISIBLE_DEVICES like
_is_datacenter_gpu); None still means every visible GPU. Applied to both the
RAM guard and the GGML_CUDA_ENABLE_UNIFIED_MEMORY env set.
- The RAM guard counted only the main GGUF plus mmproj, so a separate MTP
drafter (also resident in unified system RAM, even when offloaded to CPU)
could push the load past the RAM cap and still get OS-killed mid-load. Add
the drafter weights to the APU RAM total.
- The startup classifier reported SIGTERM (-15) as 'most likely out of memory',
but SIGTERM is also how an unload/cancel or a supervisor stops the server.
Keep the OOM wording for SIGKILL (-9, the OOM killer) and report -15
neutrally.
Tests updated/added accordingly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review on the APU guard and decode-probe ladder
- Map APU physical ids via the active ROCm mask (HIP, then ROCR, then CUDA),
mirroring _get_gpu_memory, so a HIP_VISIBLE_DEVICES-selected APU is matched.
- Only add the MTP drafter to the APU RAM total when MTP will actually engage,
so a stale LLAMA_ARG_SPEC_DRAFT_MODEL cannot refuse a non-MTP load.
- After an MTP first-decode hard fault, retry --flash-attn off (keeps MTP)
before dropping speculative decoding, matching the startup rung.
- Fold the --flash-attn= / -fa= rewrite into one branch.
Tests: tensor-parallel decode-probe assertion updated for the FA-off rung.
* Studio: tighten two comments in the APU guard and RAM preflight
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refine flash-attn retry and APU RAM guard per review
- _with_flash_attn_off now decides on the effective last-wins value: it returns
None when FA is already off (no wasted retry), and neutralizes a bare
--flash-attn / -fa (which llama.cpp reads as on) so the retry cannot re-enable
it. Length is preserved so downstream index slices stay valid.
- _amd_apu_wants_unified_memory uses 'gpu_indices is not None' so an empty
selection is respected (not treated as all-visible).
- The APU RAM refusal now checks the base model only (main + mmproj); an
optional MTP drafter is dropped by the existing MTP-drop fallback rather than
causing a hard pre-spawn refusal of an otherwise loadable model.
Tests: bare-flag / effective-off / empty-selection / HIP-mask cases added.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: free chat model VRAM at training start only when the GPU is tight
The training start route unconditionally tore down the transformers/MLX
inference subprocess before training, and never stopped the llama.cpp GGUF
server at all, so a loaded GGUF chat model kept holding VRAM for the whole
run. Conversely the HF model was always unloaded even when there was plenty
of room to keep it.
Make the unload VRAM aware and cover every inference backend:
- Add routes/training_vram.py with summarize_resident_chat(),
can_keep_chat_during_training() and free_chat_models_for_training(). The
keep/unload decision reuses the same estimator and live per device free
VRAM reader the training GPU selection already uses (auto_select_gpu_ids,
estimate_required_model_memory_gb, get_visible_gpu_utilization), so the
probe agrees with the placement computed later in start_training.
- When a chat model is resident and training fits alongside it with a
conservative margin (required_gb * 1.15 + 4 GB), keep it loaded so the
user can train and chat at the same time; on a multi GPU box training
lands on a different GPU and both coexist. Otherwise unload the HF/MLX
orchestrator and the llama.cpp GGUF server before training starts.
- The export subprocess shutdown stays unconditional and now runs first so
its freed VRAM is reflected in the decision.
Default deny: non CUDA backends, unestimable models, or any probe error
fall back to the previous always unload behavior.
Adds tests/test_training_vram_coexistence.py and updates two existing route
tests in test_gpu_selection.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-GPU floor for explicit GPU lists + don't unload chat on invalid gpu_ids
Address review feedback on the chat coexistence probe:
- Explicit gpu_ids mode now enforces a per-GPU floor in addition to the
aggregate free-VRAM check, mirroring auto_select_gpu_ids' min_per_gpu_N.
Without it, an uneven split such as free [45, 10] for a 40 GB job passed
the aggregate threshold and kept chat loaded even though the 10 GB GPU
could not hold its training shard, risking an OOM.
- Invalid explicit gpu_ids (ids outside the visible set, or a UUID/MIG
mask) make resolve_requested_gpu_ids raise. That request is rejected with
a 400 before training starts, so leave the resident chat model untouched
instead of unloading it.
- Tighten the target_modules / gpu_ids type hints to List[str] / List[int].
Adds tests for the per-GPU floor (uneven split unloads, even split keeps)
and for invalid gpu_ids keeping the chat model loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only free chat VRAM once training will start; handle in-flight and CPU-only chat
Address the second review pass on the chat-coexistence path:
- Run the chat/export VRAM teardown as a before_spawn hook inside
TrainingBackend.start_training, fired only after the start guards pass.
Previously the route freed chat VRAM before calling start_training, so a
refused start (e.g. a lingering pump thread) would tear down the resident
chat model even though no training job began.
- Treat an in-flight HF chat load (loading_models set, no active model yet)
as not safely sizeable: free it rather than risk both OOMing as the load
keeps allocating after training starts.
- Do not count or tear down a GGUF llama-server confirmed to run entirely on
CPU (_gpu_offload_active is False): it holds no VRAM, so killing it cannot
help training fit.
Adds tests for the before_spawn hook (runs on start, skipped when a
subprocess is alive or a pump thread will not die, survives a hook error),
the in-flight load flag, and the CPU-only GGUF exclusion in both the resident
summary and the unload path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat any in-flight chat load (HF swap / mid-start GGUF) as unsafe to keep
Tighten the in-flight detection in summarize_resident_chat so the keep check
never sizes a load that is still allocating:
- Flag loading on ANY non-empty loading_models, not only when active_model_name
is empty. load_model adds the new model to loading_models before clearing the
old active_model_name, so a replacement load during a swap was previously
sized as a normal resident and could OOM as the new model finishes loading.
- Flag a GGUF server that is active but not yet healthy (is_loaded False) as
in-flight: it is still mmaping/offloading layers, so its final VRAM footprint
is unknown.
Consolidates the signal into a single resident["loading"] flag; the route frees
the chat model whenever it is set. Adds tests for the replacement HF load and
the mid-start GGUF cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments in chat/training VRAM coexistence (comments only)
* Studio: run before_spawn VRAM hook only after GPU-selection validation
Reviewers found the before_spawn hook fired before prepare_gpu_selection
validated gpu_ids (and before config build), so a refused start (invalid
gpu_ids -> 400, or a bad grad-clip value) could still tear down chat/export
VRAM. Move the hook to immediately before proc.start(), once all synchronous
validation and process construction have passed. This also fixes the route's
in-flight-chat loading branch, since that teardown runs inside the same hook.
Add test_hook_skipped_when_gpu_selection_rejects.
* Studio: recompute GPU auto-selection after the before_spawn VRAM hook
Codex P2: with before_spawn moved after prepare_gpu_selection, placement was
frozen against the pre-teardown VRAM state while the hook freed export/chat
afterward. Auto-selection could pin training onto a GPU the hook then cleared
(or onto a kept chat model). Split validation from placement: explicit gpu_ids
are still validated before the hook (raise -> 400, no teardown; explicit
placement is VRAM-independent), but VRAM-dependent auto-selection now runs
after the hook so it sees the freed memory.
Add test_auto_placement_runs_after_hook and test_explicit_placement_validated_before_hook.
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard) (#6335)
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard)
The sidebar disabled New Chat, project, and home navigation while a training
run was active, so users could not chat during training even though the backend
serves inference fine alongside a run. This removes that gate and adds a backend
guard so the one genuinely risky operation, loading a new local chat model
mid-training, is refused with a clear 409 when it would not fit beside the run.
Frontend (app-sidebar.tsx): drop the chatDisabled = isTrainingRunning gate and
its consumers. Navigation triggers no model load on its own, so chat stays
usable during training.
Backend (routes/training_vram.py, routes/inference.py): add
can_load_chat_during_training plus a load/validate guard that sizes the same
effective load the loader performs (LoRA 4-bit to 16-bit resolved first, HF auto
placement via auto_select_gpu_ids, explicit multi-GPU per-GPU floor, GGUF sized
from on-disk shards and companions or the selected remote variant). It is a
no-op when training is inactive, never blocks external providers or
already-resident models, and default-denies only on a CUDA sizing failure so a
load can never OOM the run. Validate refuses early with the real settings so the
frontend does not unload the resident chat model for a load that would be
rejected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback for chat-during-training load guard
- Run the load/validate VRAM guard via asyncio.to_thread so the sync
nvidia-smi + HF metadata work never blocks the event loop.
- Size the GGUF KV cache at the requested context (_estimate_gguf_kv_gb)
and add it to the local GGUF estimate so large-context picks are not
under-counted.
- Keep the requested quantization when adapter_config.json is malformed
(not a JSON object) instead of raising in _effective_load_in_4bit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the training load guard at the launcher's effective GGUF context
The GGUF KV-cache estimate used max_seq_length only, but the llama.cpp
launcher honors a user --ctx-size/-c in llama_extra_args. A load such as
max_seq_length=4096 with --ctx-size 131072 was sized against a 4k cache
while the server allocates 131k, so the guard could approve a long-context
GGUF load that then OOMs training. Size the guard's KV at the larger of
max_seq_length and the parsed --ctx-size (reusing the launcher's own
parse_ctx_override), keeping the conservative f16 cache so the estimate is
never smaller than what the server allocates.
The chat model picker also validated with the raw max_seq_length while
/load sizes with resolveLoadMaxSeqLength, so validate could pass, unload
the current model, then have /load reject the native-context load. Validate
now uses the same effective context; the load path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the GGUF training guard at the server parallel-slot count
The KV-cache estimate assumed a single slot, but llama-server allocates the
cache across --parallel slots (app.state.llama_parallel_slots). On a Studio
launched with --parallel N>1 the guard under-sized the cache N-fold and could
approve a GGUF chat load that then OOMs training. Thread the same slot count
the loader uses into the guard's KV estimate; default 1 leaves single-slot
setups unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments for chat-during-training guard
* Studio: keep chat generation alive across navigation; Train spinner + Return to Chat
Hoist the base chat runtime above the routed outlet so navigating to Train (or any tab) no longer aborts an in-flight generation; only an explicit Stop cancels. Add a Train sidebar spinner and swap New Chat to Return to Chat while a run is active, with a lightweight completion watch so the spinner clears from any tab. Also respawn a chat llama-server killed mid-session and guard unreadable HF cache dirs that 500'd the hub model list.
* Studio: show Return to Chat on the Train tab whenever a chat is live
Previously the top sidebar item only swapped to Return to Chat while training was running; on the Train tab with an idle/just-finished run it stayed New Chat, which started a fresh thread and cancelled an in-flight generation. Show Return to Chat (and navigate back, preserving the run) whenever a generation is running or its thread is still active, or training is in progress.
* Studio: keep a running chat alive when starting a New Chat
Starting a New Chat (or switching threads) while a generation was in flight
remounted the single-chat runtime provider, which detached the in-flight run
and cut the previous chat off (it showed up frozen / empty when reopened).
Key the single-chat view by project instead of by thread or new-chat nonce so
the provider stays mounted and assistant-ui switches to a fresh thread in place.
The previous generation keeps streaming in the background and autosaves on
completion, and returning to that thread reattaches the live run instead of
reloading a half-saved one.
Also:
- "Return to Chat" now lands on the thread that is still generating rather than
the empty new chat that became active after New Chat.
- Skip the explicit /inference/cancel POST when an abort comes from a runtime
detach (navigation / background switch) rather than an explicit Stop, so a
backgrounded generation is never cancelled behind the scenes.
* Studio: make model export non-blocking and inline
The Export tab opened a full-screen modal that trapped focus, could not be
closed or cancelled while running, and showed no progress. It also stopped
training and unloaded the chat model before loading, so export could not run
alongside them.
Export now mirrors the training runtime pattern:
- Inline panel embedded where the Export Model button was, with no modal or
backdrop, so the rest of the UI stays usable during an export.
- Global export runtime store plus an app-root lifecycle hook, so a run keeps
going and streaming across navigation and is reflected on the Export nav item
from any tab.
- The worker log stream now stays connected across the load to export phase
boundary instead of stranding on "Waiting for worker output".
- Progress bar driven by phase and quant index (quant N of M for GGUF), with
elapsed time and a working Cancel.
- load-checkpoint no longer stops training or unloads inference; export loads in
its own subprocess in parallel and surfaces out-of-memory as a clear error.
- Add POST /api/export/cancel and is_export_active on /api/export/status.
* Studio: show Return to Chat on the Export tab too
Extend the New Chat to Return to Chat swap to the Export route so leaving a
running chat for Export offers a way back to the live generation, matching the
Train tab.
* Studio: smooth out Export animations and polish the panel
- Drop the height-based reveal animations (source switch, run panel, quant
picker, hub fields) that caused flashing and reflow; use instant swaps and
quick opacity fades instead.
- Method and quant cards now transition colors only, with no transition-all or
hover lift, so selecting a method or quant is crisp instead of jumpy.
- Auto-scroll the export panel into view when it opens and add a scroll-to-bottom
button when its output is below the fold, like Chat.
- Show Return to Chat on the Export tab while an export is running, matching how
training drives it on the Train tab.
- Surface the current phase or stage in the live output before the first worker
line arrives so the panel never looks stuck while progress is advancing.
* Studio: show Return to Chat on every non-chat tab
Generalize the Return to Chat swap from just Train/Export to any non-chat route
(Recipes, Projects, Hub, ...) so a running or active chat is always one click
away, instead of showing New Chat there.
* Studio: stream export logs over the Cloudflare tunnel; drop janky export animations
Exporting over a --secure Cloudflare quick tunnel showed "connecting..." with no
logs while the progress bar advanced. Cloudflare buffers text/event-stream and
only flushes when the stream closes, so the SSE log stream never reached the
browser during the run (direct localhost is unaffected, which is why this only
showed up over the tunnel).
Add a tunnel-safe JSON poll fallback (GET /api/export/logs?since=) that the
runtime lifecycle hook polls while a run is active. Short JSON responses are not
buffered by the proxy, so logs show up in near real time over the tunnel. It
shares the orchestrator's monotonic seq cursor with the SSE stream and the store
de-dupes by seq, so the two transports run together (SSE on localhost, poll over
the tunnel) without double-printing. A successful poll marks the panel
"streaming" instead of leaving it stuck on "connecting...".
Also remove the framer-motion AnimatePresence reveals from the export config and
run panel (quant picker, hub fields, the inline run panel, and the live log
section). The expand/slide animations flashed and felt clunky; the sections now
render in place.
* Studio: recover export over the Cloudflare tunnel when the blocking POST times out (524)
A model export over a --secure Cloudflare quick tunnel showed "Request failed
(524)" even though the export succeeded on the backend (the GGUF was written).
Cloudflare returns 524 when a single request takes longer than ~100s to respond,
and a GGUF conversion routinely runs for minutes, so the blocking per-method
export POST is cut off while the backend keeps going.
Confirm completion via short status polls instead of relying on the long POST
response (the same approach that fixed log streaming):
- The orchestrator records each finished op's outcome (status / output_path /
error) with a monotonic seq, exposed on GET /api/export/status.
- parseJson now preserves the HTTP status; a 524/520/522/523/502/503 or a
status-less network drop is classified as a recoverable transport error.
- runExport wraps each phase (load, every export method, each GGUF quant): on a
recoverable failure it keeps the run alive (logs keep streaming, the panel
shows "reconnecting...") and polls status until the still-running op finishes,
then settles from the recorded result, recovering the output path for the
success banner. A real 4xx still fails immediately; localhost still uses the
fast POST response. applyBackendStatus also settles a reloaded run from the
last-op record.
Verified over the tunnel: a 3m14s gemma-4-E4B-it GGUF export now ends on the
success banner with the output path instead of 524.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the export method + logs visible after navigating away mid-export
While an export was running, navigating to another tab and back to Export
remounted the page and reset the local form state (exportMethod, quant levels),
so the method card showed unselected and the run panel's log area was hidden
until the card was re-clicked. The run itself lives in the global store and was
unaffected.
Seed exportMethod / quantLevels from the active run's summary via lazy useState
initializers on (re)mount, and gate the panel's log area on the live run
(isExporting / logLines / the run's method) rather than only the local form
selection. The card stays selected and the logs/progress stay visible across
navigation; nothing changes when no run is active.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: address export/training review findings
- Export: guard Start against an empty GGUF quant selection so an inline-panel
run with no quant can't settle as success with no file produced.
- Export: thread the source HF token into the background load so gated/private
HF source exports (and gated bases) authenticate, matching the consent path.
- Export: only settle a recovered (non-owned) run as a finished export when the
last backend op was an export, not a standalone load_checkpoint.
- Training: free the export subprocess whenever an export is active, not only
once a checkpoint is loaded, so an in-flight export load can't race training
for VRAM (current_checkpoint is unset during the load phase).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: show an actionable message when the GGUF runtime is missing
Selecting a GGUF model with no llama-server installed surfaced a generic
"Invalid model" in the UI, because validate_model's catch-all discarded the
real cause. Add LlamaServerNotFoundError (a RuntimeError subclass) raised by the
GGUF preflight in ModelConfig.from_identifier, and catch it in the validate
route so users get an actionable message: run `unsloth studio setup` to
download the prebuilt llama.cpp runtime. Other validation failures keep the safe
generic message. Adds a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: also map missing GGUF runtime to a 400 in load_model
validate_model already surfaces the actionable 'install the runtime'
message for LlamaServerNotFoundError; load_model fell through to the
generic 500 'Failed to load model'. Catch it there too so a GGUF load
without llama-server gives the same install hint instead of a 500.
* Trim comments for PR #6327
* Studio: fix stale validate test after #6398 and surface missing GGUF runtime on /load
- test_other_runtime_errors_do_not_get_gguf_message: after merging #6398,
validate_model surfaces a RuntimeError's own message, so a plain RuntimeError
no longer returns "Invalid model". Assert it does not receive the GGUF
install message instead (the prior assertion was stale after the main merge).
- Raise LlamaServerNotFoundError (not a plain RuntimeError) at the backend
load-time missing-binary branch, after diffusion routing, so /load returns the
actionable 400 like remote validation, instead of a generic 500.
- Share LLAMA_SERVER_NOT_FOUND_DETAIL between the from_identifier preflight and
the load-time raise so the message stays in sync.
- Add a propagation regression test for the non-tensor load path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Reap Studio child processes when the parent dies abnormally
Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.
Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.
Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real Windows kill-on-job-close integration test
Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).
* Fix Win64 handle truncation in the Job Object calls
Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.
* Bind multiprocessing workers to parent death; harden the sweep
Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
bind_current_process_to_parent_lifetime(), wired into the shared
run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
shutdown sweep never signals a recycled pid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cross-session backstop to reap a leftover llama-server on startup
Builds on the parent-lifetime reaper: the Windows Job Object / PR_SET_PDEATHSIG
path kills children when the parent dies, and terminate_all() sweeps the living
parent's children. Neither covers an orphan left by an already-dead Studio:
terminate_all()'s registry is in-memory, PR_SET_PDEATHSIG has no macOS
equivalent, and both are best-effort.
This records the spawned llama-server PID to a pidfile under the active studio
root (removed on _kill_process). The startup reaper kills that exact PID first,
verifying it is still a llama-server to guard against PID reuse, then clears the
pidfile. It is path-independent, so it also catches an orphan the install-root
match would miss; the pidfile only ever names a Studio-spawned server, so
unrelated user processes (vllm, games) are never candidates. The existing
root-gated enumeration stays as a further fallback.
Adds tests: kills a recorded live server (real subprocess, verifies the actual
SIGKILL), skips a reused non-llama PID, cleans a stale/missing pidfile, and
clears the pidfile on kill.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only reap a recorded llama-server when it is a true orphan
Harden the pidfile cross-session reaper so it cannot kill a live server
and does not crash on Windows.
- Reap the recorded PID only when its parent is gone (a genuine orphan),
so constructing a second LlamaCppBackend in-process (the helper and
advisor paths each build one) can never kill the active chat server.
The check is topology independent: it holds whether the sweep runs in
the main process or a worker.
- Record pid:starttime and verify the start-time identity before killing,
so a PID recycled to another process is never reaped.
- Fall back to SIGTERM when signal.SIGKILL is undefined (Windows), where
os.kill maps it to TerminateProcess, instead of raising and leaving the
orphan alive while clearing the record.
Update and extend the pidfile tests: a live server with a running parent
is spared and its record kept, an identity mismatch is skipped, the
record-to-reap round trip kills a matching orphan, and the Windows
SIGKILL fallback uses SIGTERM.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Reap Studio child processes when the parent dies abnormally
Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.
Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.
Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real Windows kill-on-job-close integration test
Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).
* Fix Win64 handle truncation in the Job Object calls
Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.
* Bind multiprocessing workers to parent death; harden the sweep
Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
bind_current_process_to_parent_lifetime(), wired into the shared
run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
shutdown sweep never signals a recycled pid.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Disable MTP speculative decoding under tensor parallelism
Follow-up to #6040 (Studio tensor-parallel support).
MTP-draft speculative decoding plus --split-mode tensor crashes the CUDA
flash-attn kernel at decode time. The startup /health probe only checks that
llama-server comes up, so the existing MTP-drop fallback (keyed on startup
health) never fires and the server dies on the first generation instead.
Gate MTP off when a tensor attempt actually engages: this runs before the
VRAM planner (so no drafter memory is reserved) and before the speculative
flag build (so no --model-draft / --spec-type is emitted). Ngram modes use no
draft model and are kept, and mtp+ngram degrades to ngram rather than off. The
layer-split fallback re-runs with tensor_parallel False and restores MTP.
The reason is surfaced as spec_fallback_reason "tensor_parallel" so the
settings sheet explains why MTP is off instead of prompting a llama.cpp update.
Verified on unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL across 4x B200: the
load now emits --split-mode tensor with no MTP flags and generation completes
without the prior decode crash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make tensor-parallel MTP gate test format-independent
The assertion pinned the multi-line `speculative_type = (` form, but ruff
collapses it onto one line, so match `speculative_type =` instead.
* Recover from MTP+tensor-parallel crashes at runtime instead of banning MTP
MTP-draft speculative decoding under --split-mode tensor usually works, but can
crash llama-server's CUDA flash-attn kernel at decode time (the prompt-cache
checkpoint-restore path). The earlier fix statically disabled MTP whenever
tensor parallelism was on, which is not future-proof and gives up the MTP
speedup even though it normally works.
Replace the static ban with a try/recover, mirroring the existing load-time
MTP-drop fallback:
- Load-time decode probe: after the server passes /health under tensor +
MTP, run one tiny /completion to exercise the draft path. A failure flips
the load unhealthy so the existing fallback respawns with --spec-default.
Catches a hard incompatibility that crashes on the first decode.
- Generation-time recovery: snapshot the load kwargs after a healthy load,
and if llama-server exits mid-generation while MTP + tensor parallelism were
active, quietly reload the same model with speculative decoding off (one
single-flight background reload) and surface spec_fallback_reason=runtime_error.
Catches the rare mid-generation crash the probe and load-time fallback miss.
No persistent ban: a later fresh load re-tries MTP, so this self-heals if a
future llama.cpp supports the combo. Verified on gemma-4-26B-A4B + 4x B200:
MTP runs normally, and killing llama-server mid-generation reloads it without
MTP and serves the next request cleanly.
* Address review feedback on the MTP runtime fallback
- Authenticate the decode probe: direct-stream mode runs llama-server with
--api-key, so the unauthenticated /completion probe got a 401 and falsely
dropped MTP. Attach the same bearer auth the other internal requests use.
- Re-check the cancel flag inside the recovery thread after the death poll,
so an /unload that races the reload can't resurrect the dropped model.
- Schedule the no-MTP recovery on the connection-error paths it was missing:
generate_chat_completion's ConnectError branch, the OpenAI passthrough
typed (RemoteProtocolError/ReadError/CloseError) stream catch, and the
Anthropic passthrough generic stream catch. Previously a server that died
before reconnect, or a typed mid-stream error, skipped the reload.
* Cover every request path with the MTP+tensor crash recovery via a watchdog
The runtime MTP-crash recovery only fired from request handlers that
observed the failure, so the direct llama-server proxy endpoints
(/v1/completions, /v1/responses, the OpenAI/Anthropic passthrough
transports) -- and a crash with no request in flight -- could leave a
dead server. Add a single background watchdog, armed only on a healthy
MTP + tensor-parallel load, that polls the subprocess and routes an
unexpected death into the existing single-flight no-MTP reload. It is
stopped inside _kill_process (the one deliberate-termination chokepoint)
so a planned reload/unload is never mistaken for a crash, and re-checks
the stop flag after a detected exit to close the kill-vs-poll race. The
reload turns MTP off, so the replacement server arms no watchdog and the
fallback cannot loop; a later fresh load still re-tries MTP.
* Harden MTP+tensor crash recovery: stale-load race, pass-through MTP, requested mode
Address review findings on the runtime MTP-crash recovery:
- Stale-load race: the recovery thread snapshotted the crashed load, waited up
to 5s for the process to confirm dead, then only checked the cancel flag
before replaying load_model. A concurrent user load clears that flag, so the
stale snapshot could reload the old model over the user's new one. Make the
load lock re-entrant and run the staleness check (cancel + same process +
unchanged snapshot) under it, atomically with the reload.
- Pass-through MTP: MTP can also be requested via a user --spec-type in
extra_args or LLAMA_ARG_SPEC_TYPE, where Studio emits no spec flags and
_speculative_type stays unset, so the probe/watchdog/recovery never engaged.
Track _mtp_runtime_fallback_active from the actual launched config and gate on
it; on the no-MTP reload, append a last-wins --spec-default so the replay drops
MTP regardless of source (and the load-time fallback does the same).
- Requested mode: the off-reload reset _requested_spec_mode to off, so after a
status refresh the UI showed a bare Off with the runtime-error note suppressed
and would not retry MTP. Restore the original requested mode after the reload,
matching the startup MTP fallback.
- Snapshot the extra_args list by value so a caller mutating it cannot corrupt
the recovery snapshot.
Tests: test_tensor_parallel.py + test_llama_server_args.py green (303 passed).
* Trim verbose comments in the MTP+tensor crash recovery
Tighten the docstrings and inline comments added for the runtime MTP recovery
(watchdog, probe, reload, gating) to succinct one/two-line forms; no code
change (verified comment-only).
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix _kill_process AttributeError when _stats_logger is unset
LlamaCppBackend._kill_process references self._stats_logger in its finally
block, but __init__ only sets self._stats_logger = None partway through. If
__init__ raises before that line, or the backend is built via __new__ (as the
kill-path unit test does), teardown crashes with AttributeError instead of
cleaning up the process.
Guard the reference with getattr, matching the existing
hasattr(self, '_chat_template_file') guard in the same finally block. Fixes
test_kill_process_records_timestamp_on_actual_kill.
* Also guard _stdout_thread in _kill_process teardown
Review follow-up: the same finally block also reads self._stdout_thread, which
is unset on a partially-built / __new__ backend. Guard it with getattr like
_llama_log_fh below, and add a test that _kill_process tolerates a backend with
those optional attrs unset. Trim the _stats_logger comment.
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
* Studio: trim serving-log noise and surface llama-server engine stats
Studio prints one structured line per HTTP request, so the SPA's polling and
per-invalidation fan-out bury the lines that matter.
- Dedup identical successful GETs within a short window (default 300ms,
UNSLOTH_STUDIO_ACCESS_LOG_DEDUP_MS) so a burst logs once. The dedup key
includes the query string, so distinct query-driven GETs are not collapsed.
Runs after the response is sent, so it adds no request latency; mutations,
non-2xx, and loading polls are untouched.
- Collapse pure-liveness polls (/api/health, /api/auth/status,
/api/inference/status, /api/inference/monitor) to a longer heartbeat
(default 10s, UNSLOTH_STUDIO_ACCESS_LOG_POLL_DEDUP_MS). The API monitor
console polls /monitor every 1.5s while open.
- Translate llama-server's Prometheus /metrics into a periodic vLLM-style
engine_stats line (generation/prompt throughput and requests in flight) from
a daemon poller, gated on UNSLOTH_STUDIO_ENGINE_STATS. Throughput uses
llama-server's predicted_tokens_seconds / prompt_tokens_seconds gauges, with
a tokens_predicted_total / prompt_tokens_total counter-delta fallback; it does
not use n_decode_total (which counts llama_decode() calls, not tokens). No KV
field is emitted, since llama.cpp does not expose kv_cache_usage_ratio.
--metrics is added only when probe_server_capabilities reports the binary
supports it, so older/custom binaries still load. The poller keeps retrying
through transient scrape failures (stop() drives shutdown) and a malformed
sample cannot crash its thread.
- api_monitor.append_reply: once the preview cap is reached, skip the per-chunk
re-concat (avoids O(n^2) on long generations) while still recording the "..."
truncation marker for a reply that lands exactly on the cap.
- unsloth studio --verbose and unsloth studio run --verbose both restore every
per-request log; --verbose before a subcommand is rejected with guidance
(matching --secure / --parallel). run --verbose still forwards --log-verbose
to llama-server, preserving the pre-existing pass-through verbosity.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: reserve MTP draft VRAM in GGUF auto-fit
Auto-fit advertised a context (for example ~110k for the Qwen3.6-27B MTP
GGUF) that fit on paper but OOMed mid-generation or during tool calls once
MTP speculative decoding was active. The MTP draft path's VRAM was reserved
as a flat 5% of total VRAM, which tracks neither of the two real costs: the
MTP head keeps its own attention KV cache that grows with context, and the
speculative verification buffer grows with --spec-draft-n-max. On the
hybrid Mamba/attention Qwen3.6 models the main KV is small, so auto-fit
happily kept a near-native context while the draft path pushed the load
over budget at runtime.
Replace the flat fraction with a byte-accurate, context- and n_max-aware
reserve sized from GGUF dims: draft KV from nextn_predict_layers and the
attention dims at f16 (llama.cpp's MTP draft context uses f16 KV regardless
of the main cache type), plus a verify buffer per embedding-unit per draft
token. The reserve is evaluated per candidate context inside the fit binary
search and added to every pin/fit check, including the tensor-parallel
planner and its even-split decision. Coefficients were calibrated against
llama-server VRAM measurements on the Qwen3.6-27B MTP GGUF (RMS 14 MiB).
The flat fraction remains as a fallback when GGUF dims are unavailable, so
non-MTP loads are unchanged. The budget now also engages when the user wires
MTP through extra args (--spec-type draft-mtp, including chains), reads the
effective draft depth from --spec-draft-n-max or the legacy --draft-max with
extras taking precedence over the first-class field, reserves a separate
drafter's weights when supplied via --model-draft/--spec-draft-model/-md,
and mirrors _build_speculative_flags so it never reserves for MTP the launch
resolver will not emit (needs a head/drafter and a binary that supports
--spec-type mtp).
Adds tests/test_mtp_vram_budget.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: total-based VRAM budget + deterministic compute-graph buffer
Build on the byte-accurate MTP reserve with three changes that make the
GGUF auto-fit budget deterministic across architectures and recover usable
context, especially for MTP models on a single tight card.
1. Total-based budget. Cap GPU occupancy at a fraction of TOTAL VRAM rather
than a fraction of FREE VRAM, and raise the fraction from 0.90 to 0.95:
budget = free - (1 - 0.95) * total (per GPU, summed for a pool)
The reserve is now absolute (a fixed slice of the card) instead of
shrinking as the GPU fills, so a partly-used GPU keeps a constant cushion
for compute/CUDA/verify buffers instead of over-promising context and
spilling to CPU at runtime. _get_gpu_memory() reads memory.total alongside
memory.free; _fit_context_to_vram, _select_gpus and the load_model pool
loops thread the totals through. Multi-GPU layer-split pools
sum(free_i - 0.05*total_i); tensor mode reserves per device.
2. Deterministic compute-graph buffer. Replace the flat 5 GB/device tensor
reserve (a magic constant that over-reserved about 8x on a 27B model) with
_estimate_compute_buffer_bytes, sized from GGUF dims and the launch flags:
out = n_vocab * n_ubatch * 4 # vocab-width output buffer
act = 4 * n_embd * n_ubatch * 4 # activation scratch
pipeline_per_device = act + out * (n_parallel - 1)
tensor_per_device = 2*act + out * n_parallel
The buffer is context-independent and scales with --parallel (serving
slots), not with how the model is split across GPUs. It is now reserved in
BOTH multi-GPU paths (layer split folds one buffer into the pooled
footprint; tensor mode reserves it per device). The flat 5 GB stays only as
a fallback when vocab/embedding dims are unavailable. Calibrated against
llama-server measurements (parallel 1/2/4/8 give 36/492/1388/3220 MiB on a
single GPU; about 600 MiB/device tensor); the estimate is a small upper
bound.
3. GGUF parsing. Read vocab size (tokenizer tokens array length) and
feed_forward_length for the compute-buffer estimate.
Effect on the Qwen3.6-27B MTP Q6_K case (MTP on): a single 32 GB card at
about 31 GB free advertises f16 23k to 64k, q8_0 44k to 115k, q4_0 82k to
200k; 2x 24 GB tensor mode recovers the full 262k window for f16 (was about
134k). Validated on hardware: 1x 32 GB f16 at 64768 loads at 29.3 GB / 120
t/s; 2x 23 GB tensor f16 at 262144 loads at 22.2 GB/device / 98 t/s; both
within 0.4% of the estimate. Adds test_compute_buffer.py and updates the
KV/context-fit/MTP-budget tests for the 0.95 constant and the new budget.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten comments in the VRAM auto-fit changes
Condense the docstrings and inline comments added by this PR (internal backend
helpers): drop restated-signature docstrings, fold multi-line block comments to
one or two lines, and remove notes that just repeat the code. No behavior change
(AST-verified comment/docstring-only via comment_tools.py); the backend test
suite is unchanged and green.
* studio: address review findings in the VRAM auto-fit budget
Five fixes from a parallel-reviewer pass on this PR; all confirmed against the
real functions and covered by new tests.
- Tensor mode now honors the total-based VRAM cap. _plan_tensor_parallel took
total_by_idx and budgets each GPU at free - (1-frac)*total, mirroring the
layer-split paths; previously it fit against raw free and could spend the 5%
safety cushion on a partly-used multi-GPU box (reproduced ~3.3 GB over).
- Draft K and V cache types are parsed and accounted independently. A one-sided
override (e.g. --cache-type-k-draft q4_0, V left f16) no longer applies the
small quant to both axes and under-reserves the f16 axis. The embedded-head
formula sizes per axis; the separate-drafter path uses the heavier type so it
never under-reserves.
- The compute-graph buffer honors a user --ubatch / --ubatch-size / -ub override
(parsed and threaded into every _estimate_compute_buffer_bytes call and the
tensor planner); it previously always assumed the 512 default, under-reserving
up to ~8x at --ubatch 4096.
- GPU ranking uses the usable budget (free - (1-frac)*total) instead of raw free
in _select_gpus and both auto-context subset loops, so a more-used large card
no longer outranks a less-used small card that has more usable room.
Adds regression tests for each (tensor total cap, ubatch reserve scaling, split
K/V no-under-reserve, --ubatch parser, usable-ranking GPU selection). Full
targeted backend suite green (321 passed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: gate tensor-parallel admission on usable VRAM budget
The tensor-parallel GPU admission filters still used raw free VRAM after
the total-based budget landed, an asymmetric fix: a partly-used large card
can clear the per-device compute-buffer reserve on raw free while its usable
budget (free - (1-frac)*total) does not, so the planner admitted it and the
even split could emit a near-zero weight slice for a GPU that should have
been excluded.
- _plan_tensor_parallel: admit GPUs by usable budget, not raw free (move the
_usable helper above the filter).
- load_model: admit the tensor set by _gpu_usable, and downgrade to layer
split when the pooled usable budget cannot hold weights plus per-device
compute buffers (the planner can only floor the context, not stop an
overcommitted launch).
Adds regression tests: planner drops a GPU whose usable budget is below the
reserve, and a source-level check that load_model admits on the usable
budget and carries the pooled-weight downgrade.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: size the MTP reserve for the user's overriding drafter
A user --model-draft passed in extra_args is appended last and wins at the
llama-server launch, but the VRAM budget preferred Studio's auto-detected
drafter (mtp_draft_path or extras), so a larger custom drafter was
under-reserved. Flip the precedence to extras-first, matching the draft-depth
(n_max) resolution two lines above. Adds a source-level regression test.
* studio: account for MTP reserve in tensor gate, restore 2-col GPU probe
Two issues found by re-review of the prior fix:
- The tensor-parallel capacity gate only checked the model weights against the
pooled budget, not the MTP reserve. A separate-drafter MTP load whose weights
fit but weights + drafter do not could still launch overcommitted in tensor
mode. Add the non-shrinkable MTP reserve (drafter weights + floor draft KV, or
the flat 2 GiB fallback when dims are unavailable) to the gate.
- The nvidia-smi probe was switched to a three-column query (index,free,total)
for the total-based budget but required exactly three columns, so a driver or
mock returning the legacy two-column "index,free" was dropped and the probe
fell through to the real GPUs. Accept two columns (total 0) and treat an
unknown total as the legacy free*fraction in _select_gpus.
Tests: tensor gate asserts the MTP term is included; _get_gpu_memory parses both
two- and three-column output; the existing two-column GPU-detection mocks pass
again.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: keep the VRAM cushion in tensor planning when GPU totals are unknown
_plan_tensor_parallel fell back to raw free VRAM when a GPU's total was
unavailable (a two-column nvidia-smi probe reporting total 0), while
_select_gpus and the load_model ranking both fall back to free*fraction. That
let tensor planning spend the 5% cushion the rest of the fit preserves and
over-advertise context in exactly that path. Align the fallback to
free*_CTX_FIT_VRAM_FRACTION. Updates the no-totals planner test expectations
(now free*frac) and adds a regression test that total 0 keeps the cushion.
* studio: honor LLAMA_ARG_* env overrides and HF draft flags in the VRAM budget
The budget parsed llama-server flags only from the request's extra_args, but the
child process inherits Studio's full environment (child_env_without_native_path_secret
copies os.environ), and llama-server honors LLAMA_ARG_* env vars for the same
options. So a service-level override the child acts on was invisible to the fit,
which could then advertise a context/GPU set that OOMs at load.
- _extra_args_n_ubatch: fall back to LLAMA_ARG_UBATCH (drives the compute buffer;
an unseen 4096 vs the 512 default under-reserves ~8x).
- _extra_args_mtp_draft_path: also recognize the HF draft-repo flags
(--spec-draft-hf/-hfd/-hfrd/--hf-repo-draft) and fall back to
LLAMA_ARG_SPEC_DRAFT_MODEL / LLAMA_ARG_SPEC_DRAFT_HF_REPO. An HF repo isn't a
local file so it can't be sized, but recognizing it routes to the flat reserve
instead of mis-sizing Studio's auto/embedded drafter.
- _extra_args_draft_cache_types: fall back to
LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K/_V per axis.
CLI extra_args win over env (they are appended last at launch). Each parser takes
an injectable env for deterministic tests. Adds env-fallback and HF-flag tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: review polish - drop non-flag --ubatch, harden GPU probe, document buffer
Non-blocking items from a second review pass; no behavior change in the common path:
- _extra_args_n_ubatch: drop --ubatch; the binary only accepts --ubatch-size/-ub,
so parsing --ubatch implied support it does not have (it would over-reserve for a
launch that fails on the unknown flag).
- _get_gpu_memory: skip a malformed nvidia-smi line instead of letting one bad line
raise and drop the whole NVIDIA probe to the torch fallback.
- _estimate_compute_buffer_bytes: document that the per-slot output-buffer model
assumes a small n_outputs_max (chat decode); it would under-count for
embeddings / --logits-all / reranking, which Studio does not run on this path.
* studio: honor LLAMA_ARG_SPEC_TYPE when deciding the MTP reserve
_extra_args_requests_mtp only checked extra_args, but the child inherits Studio's
env and llama-server honors LLAMA_ARG_SPEC_TYPE. So a service-level
LLAMA_ARG_SPEC_TYPE=draft-mtp would run MTP while the fit skipped the draft
reserve and could advertise a context/GPU set that OOMs at load. Recognize the
env value (CLI still wins). Completes the env-override coverage alongside ubatch,
draft model, and draft cache types. Adds an env regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reserve VRAM for non-MTP model-based draft modes too
The draft reserve only engaged for MTP. A user passing a non-MTP model-based
draft mode (--spec-type draft-simple / draft-eagle3) with a --model-draft loads
a separate draft model whose weights + KV consume GPU memory, but the fit
reserved nothing and could OOM at load. Engage the existing drafter reserve for
those modes when extras (or LLAMA_ARG_SPEC_TYPE) name a drafter; ngram-* load no
model and are unaffected. Purely additive (reserves where there was none).
Adds parser + gate tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: floor quantized embedded MTP draft KV at f16; fix two test issues
Address PR review feedback (three findings):
1. Quantized embedded MTP draft KV was underpriced. The embedded head is a
single draft layer, so llama.cpp cannot amortize quantized-KV overhead over
many layers the way the main model does: a quantized draft KV (e.g.
--spec-draft-type-k q4_0) actually fits LESS context than f16, not more
(ggml-org/llama.cpp#24102, where a collaborator recommends f16 for the draft
KV). Pricing q4_0 at 0.5625 of an element (~28% of f16) under-reserved, so a
quantized override could advertise a context that shrinks or OOMs at load.
Floor the embedded draft KV bytes-per-element at f16 (quantized types priced
as f16, f32 still its full 4 bytes). The separate multi-layer drafter, where
quantization does amortize, keeps the user's real type.
2. test_load_model_reserves_for_non_mtp_draft_modes asserted an exact one-line
source substring that pre-commit black wrapped across lines, breaking CI.
Strip whitespace before matching so the check survives any line-wrapping.
3. test_compute_buffer.py installed a partial httpx stub via setdefault that, if
collected before test_kv_cache_estimation.py, leaked into sys.modules without
HTTPError/Response and could break the transformers introspection tier by
collection order. Adopt the sister file's pattern: only stub when real httpx
is absent, and include the full symbol set.
Updates the affected draft-KV tests to assert the f16 floor.
* studio: guard httpx stub in test_mtp_vram_budget too
test_mtp_vram_budget.py installed a partial httpx stub via setdefault that, like
test_compute_buffer.py before it, lacked HTTPError/Response and could leak into
sys.modules ahead of tests that need huggingface_hub/transformers, breaking the
introspection tier by collection order. Apply the same guard used by
test_kv_cache_estimation.py: only stub when real httpx is absent, with the full
symbol set.
* studio: per-device layer-split reserve, effective spec-type, drafter weights, KV restore
Address PR review feedback (four findings in the auto-fit budget):
A. Reserve the per-device layer-split overhead. A layer (pipeline) split allocates
a fixed per-device overhead (CUDA context + per-device compute scratch) on every
participating GPU, beyond the slot-scaling compute buffer that is conserved across
the split. Measured ~0.9 GB/device on the Qwen3.6-27B GGUF (b9625), independent of
--parallel: layer-split TOTAL VRAM grew +894 MiB (parallel=8) / +946 MiB
(parallel=1) per extra GPU, ~linear to +2.6 GB at 4 GPUs. The fit folded a single
compute buffer for all subset sizes, so a k-GPU layer split was short by
~(k-1)*0.9 GB and could pin a context that fits the pool on paper but OOMs a device.
Reserve (k-1) * _PIPELINE_PER_DEVICE_OVERHEAD_MIB per subset in the layer-split fit;
k=1 adds nothing, so single-GPU sizing (and the validated benchmark rows) is unchanged.
B. Track the effective --spec-type. _extra_args_requests_mtp returned true on the
first MTP-ish --spec-type and consulted LLAMA_ARG_SPEC_TYPE even when a CLI
--spec-type was present, contrary to llama.cpp (last CLI value wins; a CLI flag
overrides the env). So `--spec-type draft-mtp --spec-type ngram-mod` or a non-MTP
CLI value with a stale MTP env over-reserved a drafter the launch won't load
(shrinking context / selecting extra GPUs). Route both detectors through a new
_effective_spec_type helper.
C. Keep known drafter weights in the fallback reserve. When a separate drafter's KV
metadata can't be sized, _estimate_mtp_overhead_bytes returned None and discarded
the drafter's known weight bytes, falling back to the flat 5% reserve; a drafter
larger than that cushion could launch over budget and OOM. Reserve the known
weights even when KV sizing fails (None only when nothing is known).
D. Restore quantized KV on tensor->layer-split downgrade. The tensor attempt drops a
quantized KV cache (tensor mode aborts on it). When the GPU-count or capacity gate
then downgrades to layer split -- which supports quantized KV -- the dropped type
was lost and the launch used f16, using more VRAM and shrinking context. Remember
the dropped type and restore it on downgrade (the launch re-emits it from the var).
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: per-device overhead in GPU pin, skip CPU draft, gate env spec-type
Address PR review feedback (three follow-up findings):
F1. Reserve the per-device layer-split overhead in the pin path too. The earlier
per-device reserve was added to the auto-context fit loops but not to
_select_gpus, which the explicit-ctx and file-size-only paths use to PIN GPUs
with -ngl -1 (no --fit fallback). A 2+ GPU pin within ~1 GiB/extra-GPU of the
budget could OOM a device at load. Add a per_device_overhead_bytes arg to
_select_gpus so a k-GPU pin must hold model + (k-1)*overhead; pass the pipeline
overhead at both pin call sites. Single-GPU pins are unchanged.
F2. Don't charge a CPU-offloaded drafter against the GPU budget. A user passing
--spec-draft-ngl 0 or --spec-draft-device none/cpu keeps the separate draft
model's weights + KV on CPU, but the budget still charged the full drafter GGUF
size, auto-reducing context or downgrading GPU selection. Detect the CPU-offload
flags and drop the separate drafter (and its flat fallback) from the budget; an
embedded head follows the main -ngl and is unaffected.
F3. Consult LLAMA_ARG_SPEC_TYPE only when it can reach the child. llama-server's CLI
args override env, and _build_speculative_flags emits a --spec-type/--spec-default
for every UI mode except "off". So a stale MTP env on a non-MTP model (auto mode)
made the fit reserve MTP that the emitted --spec-default disables, shrinking
context / picking extra GPUs. Gate the env consult on "no user --spec-type and UI
mode off"; the MTP-model auto path still engages via Studio's own detection.
Adds regression tests for each.
* studio: drafter budget precedence and --spec-default in effective spec-type
Two spec-precedence fixes surfaced by an independent multi-reviewer pass:
R3. Size the drafter the launch actually loads. _mtp_draft_for_budget consulted
LLAMA_ARG_SPEC_DRAFT_MODEL (via _extra_args_mtp_draft_path's env fallback)
before Studio's resolved mtp_draft_path, but _build_speculative_flags emits
--model-draft mtp_draft_path, which overrides the env at launch. With a stale
(smaller) env drafter, the budget under-reserved and could OOM. Order the
budget by what actually launches: CLI extras --model-draft (appended last,
wins), then Studio's emitted mtp_draft_path (when MTP engages and the user
doesn't own --spec-type), then the env drafter.
R4. Treat --spec-default as a CLI spec override in _effective_spec_type. It only
recognized --spec-type, so extras=["--spec-default"] with LLAMA_ARG_SPEC_TYPE=
draft-mtp fell through to the env and over-reserved MTP, even though the CLI
--spec-default overrides the env to a non-MTP default. Recognize it as a CLI
spec flag (resolves to "default", non-MTP) that suppresses the env fallback.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: refine MTP draft reserve (parallel slots, last-wins, KV cushion, ranking)
Address PR review feedback (five follow-up findings, all edges of this session's
earlier MTP/auto-fit changes):
G1. Price the separate drafter's KV per --parallel slot. _mtp_draft_kv_bytes called
the drafter's _estimate_kv_cache_bytes with the default n_parallel=1, but the
drafter is served under the main model's slot count; a sliding-window drafter
(Gemma) grows KV per slot and was under-reserved. Thread n_parallel through the
draft KV / overhead estimate and the fit closure.
G2. Honor last-wins for the draft-offload flags. _extra_args_draft_offloaded_to_cpu
returned True on the first CPU value, so --spec-draft-ngl 0 --spec-draft-ngl -1
(final = GPU) wrongly dropped the drafter reserve while the server kept it on
GPU -> OOM. Decide on the final value of each flag only.
G3. Keep the flat cushion when only the drafter weights could be sized. The weights
fallback installs mtp_overhead_fn, which made callers drop the flat MTP reserve,
leaving the still-unsized draft KV with no cushion. Keep the flat fraction on in
that weights-only case, on top of the byte-accurate weights.
G4. Rank auto/cap GPU subsets by the active budget fraction. The ranking used a
hard-coded 0.95 while the fit tests _pin_fraction (lowered by the flat MTP
reserve); on mixed-total GPUs that could order subsets differently and pick a
worse plan. Rank with the same fraction the fit uses.
G5. Keep the embedded-head flat reserve under a draft CPU-offload flag. F2's
not-_draft_on_cpu guard also dropped the reserve for an embedded MTP head, which
is part of the main model and stays on GPU regardless of --spec-draft-ngl. Only
suppress the flat reserve for a CPU-offloaded separate drafter (no embedded head).
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: keep GPU on non-integer total, keep tensor flat reserve for weights-only
Two review findings:
- _get_gpu_memory dropped a whole GPU when nvidia-smi reported a non-integer
memory.total ("N/A" on some drivers / MIG / vGPU): index, free and total were
parsed in one try/except that skipped the line on any ValueError, so the GPU
vanished from the probe and the load could silently spill to CPU. Parse index
and free (required) first, then total separately, defaulting to 0 (the fit then
uses the free*frac path for that GPU). Adds N/A and bad-free test cases.
- Tensor planning skipped the flat MTP reserve for a weights-only drafter (file
size known, KV unsizable): the capacity gate used the byte floor whenever
mtp_overhead_fn was set, so it reserved only the drafter weights and no draft
KV. Tensor mode has no --fit valve, so that could overcommit and OOM. Keep the
flat reserve (never below the byte floor) in the weights-only case too, mirroring
the layer-split _mtp_kv_unsized handling. Adds a regression test.
(A third suggestion -- fold --batch-size into the compute-buffer reserve -- was
checked on hardware and declined: -b 8192 -ub 512 used identical VRAM to the
default at -c 64000, so the logical batch does not size the graph buffer; the
estimate correctly uses the physical micro-batch.)
* studio: budget the main KV from LLAMA_ARG_CACHE_TYPE env when Studio emits none
The child inherits LLAMA_ARG_CACHE_TYPE_K / LLAMA_ARG_CACHE_TYPE_V, but Studio
emits --cache-type-k/-v only when the param or extras set the type. When neither
does, a heavier env type (f32) reaches the child while the auto-fit budget
assumed the f16 default, under-reserving the main KV and risking OOM at the
advertised context. This is the one main-KV axis that lacked the env-aware
handling the other axes already have (spec-type, draft model, draft cache type,
ubatch).
load_model now adopts the heavier of the two env types when it exceeds f16 (only
f32 does), and the launch re-emits it so child and budget stay byte-consistent.
Quantized env types are <= f16 and remain safely over-reserved by the default,
so they are left untouched (no change). A single value is used because the
budget's KV estimate has one cache_type_kv knob, matching parse_cache_override's
existing key/value collapse.
Adds _env_main_cache_type_for_budget plus regression tests covering f32 adoption,
the K/V heavier-of collapse, quantized/unknown no-ops, and the load_model source
precedence.
* studio: budget tensor parallel when LLAMA_ARG_SPLIT_MODE env selects it
Studio emits --split-mode tensor only on its tensor branch; the default
layer-split path emits nothing and resolve_tensor_parallel consults only extras.
The child inherits LLAMA_ARG_SPLIT_MODE, so a tensor env on a layer-split plan
silently runs the child tensor-parallel (heavier per-device compute buffer)
while the budget reserved only the layer-split per-device overhead, under-
reserving on multi-GPU.
load_model now flips the plan to tensor when extras do not set a split mode and
the env selects tensor, so Studio plans, reserves, and emits tensor consistently.
The flip is one-directional (guarded on not tensor_parallel and no extras
split-mode) so an existing tensor plan is never downgraded and extras keep
precedence. Other env modes (layer/row/none) are not a runtime-heavier surprise
and are left untouched.
Adds _env_split_mode_is_tensor plus unit and load_model source-level tests.
* studio: reconcile inherited llama.cpp env with the budgeted launch decision
Addresses a review pass over the VRAM auto-fit work. The budget now sizes the
right amount, but the child process inherits LLAMA_ARG_* env (see
child_env_without_native_path_secret), and a few axes could still run the child
in a mode Studio neither chose nor budgeted.
Mixed known/unknown GPU totals over-advertised the pooled layer-split budget.
_pool_budget_mib pooled free and total separately, so an unknown-total GPU
(MIG/vGPU/N/A) contributed its full free with no cushion when mixed with
known-total GPUs (~(1-frac)*free over-advertise, about 500 MiB in a two-GPU
case). It now sums each GPU's own usable budget, and the layer-split fit calls
take that as an absolute budget (budget_frac=1.0, total_mib=None) so the fit and
the footprint check agree. All-known-total pools are unchanged.
LLAMA_ARG_SPLIT_MODE=tensor survived a tensor-to-layer downgrade. The downgrade
only stripped CLI extras, so the inherited env still ran the child tensor while
Studio budgeted layer split. When the final decision is layer split, a non-layer
inherited split mode (and any paired LLAMA_ARG_TENSOR_SPLIT) is now cleared from
the child env.
Inherited quantized LLAMA_ARG_CACHE_TYPE_K/_V crashed tensor mode. Tensor mode
aborts on a quantized KV cache; Studio drops a quantized cache_type_kv for the
tensor attempt but the inherited env reached the child anyway. When the final
decision is tensor split, a quantized cache-type env is now cleared so the child
uses the tensor-safe default that was budgeted.
Env-derived cache budget no longer mutates the emitted launch flags. An env-only
main KV type now informs the budget only; it is not re-emitted, so an asymmetric
K=f32,V=f16 env reaches the child as set instead of being rewritten to symmetric
--cache-type-k/-v f32.
Adds source-level regression tests for all four and confirms the documented
single-GPU/tensor/pipeline numbers are byte-identical before and after.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten comments in the VRAM auto-fit code
Compress the verbose docstrings and inline comments added by this work to
succinct 2-4 line versions, drop restated/obvious ones, and cut duplicated
rationale across the two tensor-downgrade branches. Keeps the non-obvious intent
(env-inheritance precedence, the #24102 embedded-draft floor, the per-device
overhead and pool-budget rationale) while removing roughly 120 lines of comment
text from llama_cpp.py. Also trims the few longest test-comment blocks; concise
per-test scenario notes are left intact.
No logic change: verified with comment_tools.py check --strip-docstrings (code-
only signature unchanged vs the prior commit) and the full backend suite still
passes (824).
* studio: lock in env-drafter engagement for the separate-draft reserve
A review suggested an env-provided LLAMA_ARG_SPEC_DRAFT_MODEL would skip the
draft reserve and OOM. It does not: the gate's _extra_args_mtp_draft_path(extra_args)
call defaults env=None, which consults os.environ, so an env-only drafter still
sets _user_draft_via_extras and is sized via _env_draft_for_budget. Add a source
guard that the gate keeps the env-inclusive form (not extras-only env={}) and a
behavioral test mirroring the reviewed scenario, so a future cleanup can't
regress it. No production change.
* studio: carry the unsized MTP reserve and env split/offload into tensor planning
Addresses a review pass over the multi-GPU and env-inheritance paths.
Tensor planner dropped the unsized draft-KV cushion. When a separate drafter has
known weights but unreadable KV metadata, _plan_tensor_parallel receives a
non-None weights-only mtp_overhead_fn and applied the flat 2 GiB reserve only for
the no-fn case, so its binary search spent the unsized-KV cushion on context and
over-advertised. Add mtp_flat_reserve_bytes (subtracted from the pooled budget and
the even-split check), and pass it from load_model whenever _mtp_kv_unsized. The
layer path and the tensor pre-gate already kept this cushion.
Stale LLAMA_ARG_TENSOR_SPLIT survived in tensor mode. When the planner picks an
even split it emits no --tensor-split, so an inherited tensor-split env reached the
child and overrode the budgeted split. The layer downgrade branch cleared it; the
tensor branch now does too.
Env-only draft CPU offload was ignored. _extra_args_draft_offloaded_to_cpu checked
extras but not LLAMA_ARG_N_GPU_LAYERS_DRAFT, so an env-offloaded drafter was still
charged GPU budget and under-advertised context. It now consults that env (the
device flag has no env), called with env=os.environ.
Layer-split compute buffer had no fallback when GGUF dims are missing. The estimate
returns 0 then, so the layer path folded no buffer while the tensor path falls back
to the flat reserve. Use the flat reserve for the layer path too (a safe upper
bound, since the tensor buffer >= the layer one).
All four are gated on conditions the documented benchmarks don't hit; the
single-GPU/tensor/pipeline reconfirm numbers are byte-identical, and the full
backend suite passes (830) with regression tests for each fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: share the env-aware tensor decision across load and dedup matchers
A review pass found the inherited-LLAMA_ARG_SPLIT_MODE=tensor flip lived only
in load_model, so the two duplicate-load matchers disagreed with it.
Consolidate the decision into _effective_tensor_parallel (extras + toggle, then
flip on when extras set no split mode and the child inherits a tensor split
env). load_model, the backend matcher (_already_in_target_state) and the route
matcher (_request_matches_loaded_settings) now all call it. Before, an env-driven
tensor server compared against resolve_tensor_parallel (env-blind) in both
matchers, so a follow-up load that should dedup was seen as a mismatch and the
healthy server was needlessly killed and reloaded.
Also finish the tensor cache-type handling: when the tensor attempt drops a
quantized KV it now re-adopts a heavier inherited env cache type (f32) for the
budget, mirroring the initial adoption; and the two layer-split downgrades clear
_cache_type_from_env so the restored quantized type is actually re-emitted rather
than left to a stale inherited env.
All gated on inherited env the documented benchmarks don't set; the single-GPU,
tensor and pipeline reconfirm numbers are byte-identical, and the full backend
suite passes (832) with unit + source regression tests for the shared helper and
the route matcher.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: complete the env-aware tensor/spec handling across all paths
A second review pass found the env-aware tensor/MTP handling was applied
asymmetrically: some paths inherited LLAMA_ARG_* env, others didn't. Three real
follow-ups, plus a small consolidation so the env semantics live in one place.
1. Tensor fallback ignored the inherited tensor env. load_with_tensor_fallback
computed its retry gate with the env-blind resolve_tensor_parallel, so an
env-only tensor load (toggle off, no --split-mode extra) that crashed on a
tensor-incompatible GGUF re-raised instead of retrying layer split. It now
uses the env-aware decision; and since the inherited env would otherwise
re-engage tensor on the retry (CLI args persist, the env does too), the retry
forces --split-mode layer (CLI wins over env) so it can't re-crash.
2. Duplicate-load matchers looped reloads after a tensor->layer downgrade. Both
matchers compared the env-expanded tensor decision against the loaded server,
but load_model may downgrade tensor to layer (capacity/buffer) and scrub the
child env. The still-set parent env then made every identical request look
like a mismatch, killing and reloading a healthy layer server. Add
_tensor_parallel_matches_loaded, which only lets an inherited tensor env raise
a match against a server that actually launched tensor; a downgraded server
matches the same request (an identical load would downgrade the same way).
3. MTP binary-capability fallback leaked an inherited LLAMA_ARG_SPEC_TYPE. When
the binary lacks MTP, _emit_mtp degraded but emitted no spec flag, so an
inherited LLAMA_ARG_SPEC_TYPE=draft-mtp still reached the child and attempted
MTP the gate had budgeted off. It now emits --spec-default (CLI wins over env)
like the sibling no-head / non-MTP fallbacks.
Consolidation: moved _env_split_mode_is_tensor / _effective_tensor_parallel into
llama_server_args.py (with the new _tensor_parallel_matches_loaded) so the
lightweight tensor_fallback module can share them without importing llama_cpp;
llama_cpp re-exports them for back-compat.
All gated on inherited env the documented benchmarks don't set; the single-GPU,
tensor and pipeline reconfirm numbers are byte-identical, and the full backend
suite passes (883) with regression tests for each fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: budget the heavier axis of asymmetric --cache-type-k/-v extras
A review pass found the explicit-extras counterpart of the env cache-type fix.
load_model adopts the heavier inherited LLAMA_ARG_CACHE_TYPE_K/_V env for the
reserve, but the explicit-extras path used resolve_cache_type_kv, which collapses
both axes to one last-wins value. So extras such as
--cache-type-k f32 --cache-type-v f16 (lighter axis last) budgeted f16 for both
axes while the child allocates f32 on K, over-advertising context and
re-opening the OOM path this PR closes.
Add parse_cache_override_per_axis (keeps the K/V last-wins values apart) and
_extra_args_main_cache_type_for_budget (the heavier of the two by bytes/elem),
and budget from it. The user's extras are appended last and win per axis at the
child, so this only raises the reserve; the emitted command and the asymmetric
child cache are unchanged, and the common single-axis / symmetric cases resolve
to the same type as before.
Reconfirm numbers (single-GPU table, tensor, pipeline) are byte-identical, and
the full backend suite passes (892) with per-axis parser and heavier-axis budget
regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix tensor-safety masking and strip inherited HF drafter selectors
A review pass found two more env/extras edge cases on the speculative and tensor
cache paths.
Tensor-safety could miss a quantized axis. The previous change budgets the
heavier-by-bytes cache type, but that masks a quantized axis paired with a
heavier one: --cache-type-k f16 --cache-type-v q4_0 resolves to f16, so the
tensor-safety block did not fire and the q4_0 axis survived into tensor mode,
which aborts on quantized KV. Test each explicit --cache-type-k/-v axis (not just
the budget type) so any quantized axis drops the cache for the tensor attempt.
Inherited HF drafter selectors were not stripped. _extra_args_mtp_draft_path
treats --spec-draft-hf / -hfd / -hfrd / --hf-repo-draft as drafter selectors, but
_SPEC_FLAGS only stripped the local --model-draft selectors, so on an inherited-
extras Apply a stale HF drafter survived and last-wins-overrode Studio's
re-derived spec choice. Add the HF aliases to _SPEC_FLAGS. The per-drafter tuning
knobs (--spec-draft-type-*, -ngld, --spec-draft-device) are intentionally left in
place: the VRAM budget reads them via the same parsers the child honors, so they
stay consistent on inherit, and stripping them would silently move a CPU-offloaded
drafter back onto the GPU.
A third flagged item -- that the HF draft env var should be LLAMA_ARG_HFD_REPO --
was a false positive from a stale manpage; the bundled binary's common/arg.cpp
sets LLAMA_ARG_SPEC_DRAFT_HF_REPO for --spec-draft-hf, which the code already
uses, so it is left unchanged.
Reconfirm numbers (single-GPU table, tensor, pipeline) are byte-identical, and
the full backend suite passes (899) with regression tests for both fixes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: preserve asymmetric cache on tensor downgrade and skip CPU-drafter reserve
A review pass found two more tensor-path edges, one a regression from the
per-axis cache change.
Tensor-to-layer downgrade collapsed asymmetric cache extras. The per-axis
tensor-safety check strips an asymmetric --cache-type-k/-v (tensor rejects
quantized KV), but the downgrade restored only the scalar heavier type, so a
layer fallback silently rewrote --cache-type-k q4_0 --cache-type-v f16 to
symmetric f16/f16 even though layer split supports the original. Save the
original extras before the tensor strip and restore them verbatim (minus the
user --split-mode) on both downgrade points; the budget still uses the heavier
scalar, the child gets the real asymmetric cache. Before the per-axis change this
case happened to survive (last-wins was f16, untouched), so this restores that.
Tensor mode reserved GPU VRAM for a CPU-offloaded drafter. The layer path drops
the flat MTP reserve when the only drafter is a separate CPU one with no embedded
head, but the tensor capacity gate and planner still charged it, under-advertising
context. Gate the tensor reserve on the same condition via _mtp_reserves_gpu.
Reconfirm numbers (single-GPU table, tensor, pipeline) are byte-identical (both
fixes are gated on conditions the benchmarks don't hit), and the full backend
suite passes (901) with regression tests for each.
* studio: drop now-unused llama_server_args imports from llama_cpp
The refactor re-pointed load_model and the matchers off resolve_tensor_parallel /
resolve_cache_type_kv and moved the env split-mode helper into llama_server_args,
leaving those three names imported but unused in llama_cpp. The repo's import-hoist
safety-net lint blocks that, so drop them; the env split-mode test now imports
_env_split_mode_is_tensor from its real home (llama_server_args).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add shared Xet-primary download helper with HTTP stall fallback
Xet is the fast default transport in huggingface_hub, but a stalled Xet
transfer hangs with no progress and no exception, and a blocked native thread
cannot be killed. The safetensors inference path already recovers (subprocess
watchdog + respawn with HF_HUB_DISABLE_XET=1); the GGUF and training paths do
not. Add a reusable helper that the in-process paths can adopt.
utils/hf_xet_fallback.py:
- DownloadStallError (moved here from core/inference/orchestrator.py, which now
imports it; behavior unchanged, still a RuntimeError subclass).
- get_hf_download_state / start_watchdog: a no-progress watchdog built on the
sparse-aware hub.utils.hf_cache_state helpers; fires only while a .incomplete
is present and the on-disk byte total is unchanged for stall_timeout.
- hf_hub_download_with_xet_fallback: cached files short-circuit; otherwise the
download runs in a spawn child (own process group) supervised by the watchdog.
On a stall it kills the child, makes the partial safe for HTTP via
prepare_cache_for_transport, and respawns once with HF_HUB_DISABLE_XET=1. Cancel
and deterministic errors (auth/missing/disk) propagate without a fallback.
Tests cover the watchdog state machine, the transport decision logic, and a
regression lock that HF_HUB_DISABLE_XET is honored in a fresh interpreter.
* Studio: route GGUF Chat-Mode downloads through the Xet->HTTP fallback
The GGUF load path (_download_gguf main+shards, _download_companion_gguf for
mmproj/MTP) called a bare blocking hf_hub_download with no recovery, so a Xet
stall hung the Chat-Mode load with no fallback. Route those three calls through
hf_hub_download_with_xet_fallback: Xet stays primary, HTTP is used only if Xet
stalls, per-file so finished shards stay cached. The existing _cancel_event is
threaded through, the Cancelled sentinel is preserved, and companions stay
best-effort (a terminal stall is swallowed to None). Cached files short-circuit
in the helper with no subprocess, so the fast path is unchanged.
The two offline mmproj tests are repointed from huggingface_hub.hf_hub_download
to the new call boundary (the helper) since the download now goes through it.
* Studio: recover a stalled training model-load via Xet->HTTP respawn
Training runs in a spawn subprocess and FastModel.from_pretrained downloads
internally, so the download cannot be wrapped per-file like GGUF. Instead the
worker now watches the HF cache during the model-load phase (emitting
model_load_started / model_load_completed and a stall event), and the parent
recovers a stall by terminating the worker and respawning it once with
HF_HUB_DISABLE_XET=1.
worker.py: set HF_HUB_DISABLE_XET=1 before any HF import when the parent passes
disable_xet (respawn), and wrap trainer.load_model with start_watchdog.
training.py: plumb disable_xet through the config; track the model-load window;
on a first-load stall arm a one-shot respawn (handled on the exiting pump thread,
so no pump self-join) that preserves the DB run row (history is not duplicated)
and re-runs the load over HTTP. A second stall, or a stall outside model-load,
surfaces as a normal error. W&B init happens after model-load, so a pre-load
respawn cannot duplicate it; the dataset is re-formatted in the new worker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add gated test-only fault-injection hook for the Xet stall path
UNSLOTH_HF_XET_FORCE_STALL=1 makes the Xet download attempt write a partial
blob and hang, so the no-progress watchdog and the HTTP fallback can be
exercised end to end against a real repo (never set in production). Used to
verify recovery on real models: a forced Xet stall on a 5.37GB Qwen3.5-35B-A3B
shard triggered the watchdog and the HTTP retry downloaded the correct file
(sha256 verified).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten Xet-fallback comments and consolidate its tests
Trim docstrings and inline comments across the Xet->HTTP fallback code to
the non-obvious why (spawn-not-thread, killpg-not-getpgid, the sparse-partial
HTTP-resume hazard); drop comments that merely restate the code. Verified
comment-only with an AST signature check.
Merge the three helper-level test files (watchdog, transport policy, and the
HF_HUB_DISABLE_XET regression lock) into tests/test_hf_xet_fallback.py, and
prefer the real structlog over a bare stub so test collection order cannot
leak an incomplete module to others that log at import.
Full backend suite: 3455 passed, 14 pre-existing flash-attn failures only.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: omit --threads when unset so llama.cpp picks physical cores
Studio passed --threads -1 when no thread count was set. The intent was physical cores, but an explicit --threads -1 makes llama.cpp's arg parser resolve it to hardware_concurrency() (every hyperthread), which contends on the memory bus and slows CPU and hybrid decode (a user saw about 60-75 fall to about 20-30 tok/s under CPU offload). Leaving --threads unset keeps n_threads at -1, which llama.cpp resolves to physical cores via common_cpu_get_num_math(). Omit the flag when unset; still pin it for an explicit override and the Windows full-offload OpenMP cap.
* Studio: drop inherited LLAMA_ARG_THREADS when omitting --threads
Omitting --threads relies on llama.cpp resolving physical cores via common_cpu_get_num_math. But the child inherits os.environ and llama.cpp also reads --threads from LLAMA_ARG_THREADS, which routes through the arg handler and maps <=0 to hardware_concurrency. So an ambient LLAMA_ARG_THREADS would silently override the physical-core default. Scrub it from the child env only when we omit the flag.
* Studio: warn when a GPU model silently loaded on CPU
llama-server can serve HTTP 200 while running a model entirely on CPU when its GPU backend fails to init, so Studio could run a GGUF on CPU without saying so (#5807 / #5106 / #5830). The silent-CPU warning already exists but stopped firing on current llama.cpp because _classify_gpu_offload keyed only on the dropped 'model buffer size' lines. Add a shared classify_gpu_offload_lines (offloaded N/M counts, GPU model-buffer markers excluding _Host, device_info disconfirm-only) and delegate to it so the warning fires again. Log-only: no install or load behavior changes.
Pure classification of already-captured startup log lines, run once after load; no new subprocess, no slowdown.
* Studio: key the CPU-offload warning on the main model, not a draft
With MTP/speculative decoding llama-server logs 'offloaded N/M layers to GPU' twice: once for the main model and once for the small draft model. The old scan returned True on any non-zero count, so a drafter that fits on GPU while the main GGUF runs on CPU suppressed the warning (the Qwen3.6-27B-MTP case). Decide on the line with the most layers (the main model) instead, so a drafter cannot mask a main model on CPU.
* Fix the libaray path for probe_server_capabilities()
even when running something as simple as `./llama-server --help`,
the binary still requires correct LD_LIBRARY_PATH to work - or it
returns merely an "error while loading shared libraries":
"libllama-server-impl.so: cannot open shared object file: No such file or directory"
For a local installation with no LD_LIBRARY_PATH specifically set,
the probe_server_capabilities() run of `./llana-server --help` should
share the same libaray resolution logic as start_llama_server().
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adjust and readd comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
On a restart the constructor reaps the previous run's orphaned llama-server, but the driver does not reclaim that VRAM synchronously. _kill_orphaned_servers now returns the number of processes it killed, and __init__ arms _last_kill_monotonic when that count is positive, so the first load_model waits for VRAM to settle before ranking GPUs by free memory instead of pinning the model onto the smaller card. The compute-graph / auto-fit reserve is handled separately in #6312.
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: add Bypass Permissions (skip confirmation, disable tool sandbox)
Adds an opt-in Bypass Permissions toggle next to Confirm tool calls. When on,
no tool call shows a confirmation prompt and the python/terminal sandbox is
disabled: safety checks, command blocklist, and resource limits are skipped.
Secret env vars are still stripped and HOME stays repointed at the session
workdir. Default off keeps current behavior, and it takes precedence over
Confirm tool calls. Enabling it requires accepting a warning each time.
* Studio: harden Bypass Permissions secret handling and fix Anthropic tool path
Follow-up to the Bypass Permissions feature. Addresses the review findings:
- Anthropic /v1/messages 500: declare bypass_permissions on
AnthropicMessagesRequest so tool requests that omit the field default to
False instead of raising AttributeError (extra='allow' does not set absent
attributes).
- /proc parent-env leak: stripping the child env did not stop a same-uid
bypassed child from reading /proc/<parent>/environ to recover the
tool-executing process's unfiltered secrets. Clear PR_SET_DUMPABLE on that
process before the first bypass exec so its /proc entries become root-owned.
Hardening is fail-closed: if prctl is denied, bypass execution is refused
rather than run with the parent environ still readable. Mitigation, not a
full boundary; documented in the code.
- Broker/capability vars: strip SSH_AUTH_SOCK, SSH_AGENT_PID, GPG_AGENT_INFO,
GNUPGHOME, KUBECONFIG, DOCKER_HOST so a bypassed tool cannot use the
operator's live agents.
- Credential-bearing URL values: drop any env var whose value embeds URL
userinfo (scheme://user:pass@ and token-only scheme://token@) regardless of
the variable name. Benign proxy/index URLs without credentials are kept, so
proxy-only and internal-index setups still work in bypass mode.
- Windows temp isolation: repoint TEMP and TMP (not just TMPDIR) at the
per-session sandbox dir.
- Frontend: stop persisting bypassPermissions; a reload now starts with the
sandbox/confirmation bypass off and requires re-accepting the warning dialog.
Adds regression tests for each finding in test_bypass_permissions.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip cred-location env vars (HF_HOME etc.) in Bypass Permissions
Repointing HOME did not stop SDKs auto-reading cached creds via vars that
point at the real home/cache/config: HF_HOME (startup always sets it; token
lives under $HF_HOME/token), HF/XDG cache roots, NETRC/BOTO_CONFIG/
PIP_CONFIG_FILE, and Windows HOMEDRIVE/HOMEPATH. Drop those, and repoint
USERPROFILE/APPDATA/LOCALAPPDATA at the per-session workdir. Adds regression
tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: lock in bypass HF token resolution with an end-to-end test
The drop-based fix relies on the whole HF_HOME/XDG fallback chain being
removed so huggingface_hub resolves under the repointed HOME. Add a test
that sets HF_HOME and XDG_CACHE_HOME at a real cache and asserts the
resolved token path lands under the workdir, not the operator's cache.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip npm _auth, MYSQL_PWD, and BASH_ENV from bypass env
Three more credential vectors dodged the bypass scrubber: NPM_CONFIG__AUTH
(npm _auth, base64 so no URL userinfo and no AUTH marker), MYSQL_PWD (markers
use PASSWD, not PWD, since PWD is the cwd var), and BASH_ENV (bash -c sources
it for non-interactive shells, so a startup file can re-export stripped
secrets). Add an AUTH marker, the exact MYSQL_PWD name, and drop BASH_ENV plus
PGPASSFILE. Adds regression tests incl. an end-to-end check that a bypass
terminal call does not source BASH_ENV.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend bypass env scrubber and enforce confirm precedence in loops
From a parallel review pass over the bypass changes:
- Drop more credential-location vars in _build_bypass_env: npm/yarn/git/cargo/
rclone config pointers (NPM_CONFIG_USERCONFIG, NPM_CONFIG_GLOBALCONFIG,
YARN_RC_FILENAME, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM, CARGO_HOME,
RCLONE_CONFIG) and the GIT_ASKPASS/SSH_ASKPASS auth helpers.
- Enforce confirm_tool_calls AND NOT bypass_permissions inside the safetensors
and GGUF tool loops, not just at the route, so a direct internal caller
passing both flags never prompts.
- Soften the toggle hint: environment secrets are stripped, but bypassed code
can still read files and credentials on the machine (no overclaim that keys
stay hidden).
Adds regression tests for the new names and the loop-level precedence.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add GGUF loop test for bypass-over-confirm precedence
The safetensors loop precedence is covered behaviorally; the GGUF loop needs a
live llama-server so add an AST guard asserting its _needs_confirm gate
references both confirm_tool_calls and bypass_permissions, matching the other
llama_cpp source-inspection tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add red Bypass Permissions badge in the composer
When Bypass Permissions is on, show a persistent red pill in the composer
tool-pill row (like the Search/Code pills), matching Claude Code's always-
visible bypass indicator. Clicking it turns bypass off, mirroring the other
composer toggles. Enabling still goes through the settings toggle + warning
dialog. Adds a data-variant=danger style for the destructive-colored pill.
* Studio: show Bypass Permissions badge in the Thread composer too
The empty-state and active Thread render their own composer (thread.tsx),
not shared-composer, so the badge only appeared in the split layout. Mirror
the red dismissible pill in ComposerAction so it shows in every composer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the Bypass Permissions badge visible when the composer is collapsed
The Thread composer only renders the pill row when expanded, so the active-mode
badge vanished on the default (collapsed) empty state. Render it before the
expand gate (it returns null when bypass is off) so the red indicator always
shows while bypass is on.
* Studio: make the Bypass Permissions confirm button a solid red button
The destructive button variant is a subtle 10% tint that read as bare red text
next to the outlined Cancel. Force the solid destructive fill (the variant's
class loses to the tint through AlertDialogAction's Slot merge, so use the !
override the codebase already uses for this case) and shorten the label to
'I understand' so it fits the small dialog's two-column footer.
* Studio: add Bypass Permissions to the composer + More menu
Adds a 'Bypass Permissions' entry to the composer plus-menu (under More by
default) in both composers, so it can be toggled without opening Run settings.
Enabling routes through the same danger warning dialog; disabling is immediate.
A shared BypassPermissionsMenuItem keeps the two composers in sync.
* Studio: harden bypass env scrubber for IMDS opt-out and connection strings
Two gaps in the Bypass Permissions secret scrubber:
- The broad AWS_ prefix also dropped AWS_EC2_METADATA_DISABLED, a non-secret
opt-out. Removing it re-opens the IMDS instance-role credential path that the
operator explicitly disabled, so a bypassed boto/AWS-CLI call could recover
cloud creds. Keep that flag (and AWS_EC2_METADATA_V1_DISABLED) via a keep-list
while still stripping the real AWS credential vars.
- Azure App Service connection strings (SQLCONNSTR_/CUSTOMCONNSTR_/...,
WEBSITE_CONTENTAZUREFILECONNECTIONSTRING) and values like Password=/AccountKey=
/SharedAccessKey= slipped past the name and URL-only value classifiers. Add
CONNSTR/CONNECTIONSTRING name markers and a connection-string value matcher.
* Studio: let Bypass Permissions suppress the confirm-tool-calls guards
The confirm-vs-bypass precedence (confirm and not bypass) was applied at the
loop call sites but not at the earlier request guards, so a client sending
confirm_tool_calls + bypass_permissions together was rejected (stream=true
required / unsupported for external or Anthropic tools) before the precedence
took effect. Gate all four confirm guards on not bypass_permissions so both
flags together proceed with the gate suppressed, matching the documented rule.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): Windows GGUF cancel hang + CPU spinlock overhead (#5692)
Two fixes for Windows-native GGUF inference via llama-server:
**Issue 1 — GPU/CUDA Hang on Stream Cancellation:**
- Add `Connection: close` header to all httpx requests proxying to
llama-server, preventing Keep-Alive from masking downstream socket
closure.
- Introduce `_await_disconnect_then_close` background watcher that
polls `request.is_disconnected()` every 100ms and calls
`resp.aclose()` immediately when the client disconnects. This runs
alongside the existing cancel-POST watcher and covers client aborts
that never reach the /cancel endpoint (tab close, proxy aborts,
Colab, mobile navigation, etc.).
- Change all StreamingResponse `Connection: keep-alive` headers to
`Connection: close`.
**Issue 2 — High CPU Spinlock & KV Cache Backup Overhead:**
- Set OMP_WAIT_POLICY=PASSIVE and OMP_NUM_THREADS=2 in the
llama-server subprocess environment on Windows to prevent OpenMP
from spin-waiting on all logical cores while the GPU decodes.
- Limit `--threads` to 2 on Windows when the model is fully
GPU-offloaded (`-ngl -1`). Auto-detect otherwise.
- Pass `--cache-ram 0 --ctx-checkpoints 0 --no-cache-prompt
--checkpoint-every-n-tokens -1` on Windows to disable prompt-cache
snapshots that copy KV cache to system RAM over the WDDM/PCI-E bus.
Closes#5692.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: use local import to avoid ruff F823 (sys used before assignment)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: address gemini review feedback
- Simplify _fully_gpu_offloaded init: default to False, only set True
in the gpu_indices branch, drop redundant else.
- Log exceptions in _await_disconnect_then_close at debug level instead
of silent pass, per review suggestion.
* Adjust review feedback for PR #5749
- _await_disconnect_then_close: set cancel_event before resp.aclose() so
the streamer's RemoteProtocolError handler treats the watcher-driven
close as cancellation, not an upstream error. Both call sites pass
cancel_event through.
- Windows --cache-ram / --no-cache-prompt / --ctx-checkpoints block: gate
on _fully_gpu_offloaded so CPU and partial-offload Windows runs keep
prompt-cache reuse across turns.
- Windows OMP_WAIT_POLICY / OMP_NUM_THREADS env: same gate so CPU and
partial-offload Windows runs keep default OpenMP parallelism.
* Shorten code comments touched by PR #5749
* Clean up local imports and rename underscore locals in PR #5749
- Drop the function-local `import sys as _sys` introduced as an F823
workaround; remove the redundant in-function `import os`/`import sys`
block so module-level imports resolve sys/os instead. F823 no longer
triggers because no shadowing import remains inside load_model.
- Rename `_fully_gpu_offloaded` and `_t` to `fully_gpu_offloaded` and
`threads_arg`. Underscore-prefixed names usually mean private/module-
level; plain locals match Python style for in-function temporaries.
No behavior change. ruff clean, py_compile clean, 35 studio cancel-
infra tests + 13 launch-gating AST locks + 6 disconnect-watcher locks
+ 4 spoof live-import tests all pass.
* Fix Windows GGUF follow-ups for PR #5749
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix cache flag gating for PR #5749
* Fix Python 3.9 annotations for PR #5749
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Anmol Mishra <anmolx.work@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: decide diffusion routing before the SWA resolver
Loading a DiffusionGemma GGUF could fail with "llama-server failed to start. Check that the GGUF file is valid" while llama-diffusion-cli ran the same model fine.
_read_gguf_metadata set self._is_diffusion after calling _resolve_swa_pattern, both inside one try/except. The resolver reaches into transformers/HF, which can raise for an architecture transformers does not know (diffusion-gemma); the shared except then swallowed it and left _is_diffusion False, so the model was routed to plain llama-server instead of the diffusion runner. It only reproduced where the SWA pattern was not already cached/inline.
Set _is_diffusion right after the KV parse loop (before the resolver) so a resolver error can no longer drop the routing, and skip the resolver for diffusion models, which do not use Studio's SWA pattern.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: keep llama-server discovery from crashing on an access-denied candidate
_find_llama_server_binary probed candidates with Path.is_file(), which raises
PermissionError (WinError 5) when a path exists but is momentarily inaccessible
(antivirus lock, an install replace in flight, an elevated-install ACL),
aborting model validation. Treat a denied-but-present path as the real binary
so discovery returns it; absent paths still skip.
* Retry a transiently locked binary instead of returning a denied path
Returning a still-denied path only moved the PermissionError to the next
is_file() (probe_server_capabilities). Retry briefly so a transient lock
clears and discovery returns an accessible path; on a persistent lock return
nothing rather than a path downstream cannot stat.
* Studio: do not fall back to another llama-server when a pinned one is locked
A denied LLAMA_SERVER_PATH made discovery skip the explicit pin and run a
lower-priority managed or PATH binary, so a load could silently use a stale or
incompatible server. Split the probe into a file/absent/denied status: when the
pinned path exists but stays access-denied, warn and stop rather than falling
back to a different executable.
* Studio: never downgrade past a denied pinned or managed llama-server
Extend the no-fallback rule beyond LLAMA_SERVER_PATH: a present-but-denied
UNSLOTH_LLAMA_CPP_PATH or managed ($STUDIO_HOME/llama.cpp, ~/.unsloth/llama.cpp)
binary now reports temporarily-unavailable instead of silently launching a
lower-priority legacy or PATH server. Shared _scan_pinned/_unavailable helpers;
legacy in-tree and PATH stay genuine fallbacks (a denied candidate there just
continues).
* Studio: let diffusion asset lookup use a locked llama-server path for its dir
DiffusionGemma does not run llama-server; _find_diffusion_assets only needs the
install dir to find the adjacent llama-diffusion-gemma-visual-server. The
no-fallback rule returning None on a transiently locked llama-server therefore
hid an available visual-server and raised 'runner not found'. Add an
include_denied option so diffusion lookup gets the locked path (its dir is all
it needs), while inference keeps the no-denied-path, no-downgrade behavior.
* Studio: report a locked llama-server as temporarily unavailable, not missing
When the pinned/managed binary stays access-denied through the retries, discovery
returns None and load_model raised 'binary not found', a terminal error that
points users at reinstalling rather than retrying a transient AV/install lock.
Reuse include_denied to detect the locked path and raise a distinct
temporarily-unavailable, retry message instead.
* Studio: GGUF preflight treats a locked llama-server as present
The pre-download preflight (and so /api/inference/validate) used the default
discovery, which returns None for a transiently access-denied binary, so it
raised 'binary not found' for a binary that merely needs the lock to clear. Use
include_denied so the existence check counts a locked binary as present; the
load itself still reports a still-locked binary as temporarily unavailable.
* fix: extend llama.cpp first-token timeout
* fix: timeout label pluralization
* studio: distinguish llama stream timeout phases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust timeout handling for PR #5841
* Fix lint failure for PR #5841
* Fix/adjust stream timeout handling for PR #5841
* Fix/adjust first token timeout for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust passthrough timeouts for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust preheader stream cancellation for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust timeout PR diff for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust Python 3.9 stream iteration for PR #5841
* Fix first body timeout for PR #5841
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix first token timeout deadlines for PR #5841
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
A clean install could not run DiffusionGemma: the runner spawns
python -m unsloth_zoo.diffusion_studio.shim, and unsloth_zoo refuses to
import unless UNSLOTH_IS_PRESENT is set (normally by import unsloth). The
shim never imports unsloth, so the subprocess died with
'Please install Unsloth via pip install unsloth!' and the model load
failed with a 500. Set the flag in the runner child env, as unsloth does
on import.
* Studio: account for mmproj VRAM in GGUF fit budget (#5825)
Vision GGUFs load the mmproj projector onto the GPU via --mmproj
alongside the weights, but the context auto-sizing / GPU-selection
budget sized off _get_gguf_size_bytes(model_path), which counts only
the weight file(s). The projector was never added, so the budget was
too optimistic: context got mis-estimated and tight vision loads
spilled to system RAM / OOM'd.
Resolve the launch projector once before GPU selection and fold its
size into the fit budget. The same resolved path feeds both the budget
and the --mmproj launch flag, so the two cannot disagree. The summary
log now reports the projector size separately, keeping "GGUF size"
accurate.
Adds _mmproj_vram_bytes() + unit tests (no GPU / network / subprocess).
* Studio: simplify mmproj summary-log concatenation (#5825)
Address review: the summary log mixed explicit `+` with implicit
f-string concatenation. Extract the optional projector fragment into
`mmproj_note` so the logger.info uses uniform implicit concatenation.
No behavioral change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: trim mmproj VRAM comments
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
DiffusionGemma serves via the visual runner, which streams per-step
canvas frames so the answer resolves live in the bubble. The agentic
tool loop (generate_chat_completion_with_tools) does not forward those
frames, so whenever a tool pill (Search/Code) was on the live canvas
silently vanished while text still streamed. DiffusionGemma is not a
tool-calling target anyway, so report supports_tools=False for it: the
chat always takes the frame-forwarding path, and the Search/Code pills
disable themselves (a local model has no builtin web search either).
Also turn the artifacts canvas on by default for DiffusionGemma so a
full-HTML answer (e.g. a playable game) renders as an interactive
sandboxed card without the user flipping the global artifacts toggle.
* Studio: override chat template for unsloth/gemma-4-*-GGUF with bundled gemma-4.jinja
The chat templates baked into the shipped unsloth/gemma-4-*-GGUF quants predate
Google's gemma-4 chat-template PR #118 and lack the preserve_thinking flag, so
Studio cannot surface the "Preserve thinking" toggle for Gemma 4. Bundle the updated
template and override the embedded one at llama-server launch via --chat-template-file,
scoped to the gemma-4 GGUF family, so users do not need to re-download any quant.
- Add studio/backend/assets/chat_templates/gemma-4.jinja (PR #118 based;
preserve_thinking defaults false, the one deliberate divergence from upstream).
- Add core/inference/chat_templates.py: gemma-4 GGUF matcher plus an
effective-override resolver (explicit user template still wins).
- Wire the resolver into routes/inference.py ahead of the reload-dedup check and
both load_model calls so the live backend and the incoming request compare against
the same template text (no spurious reloads).
- Default preserve_thinking off in the launch-time chat_template_kwargs so direct
API callers match the UI default.
- Ship the asset via package-data and add unit tests.
* Studio: ship E2B/E4B edge variant of the bundled Gemma 4 template
Google ships two distinct gemma-4 chat templates: E2B and E4B omit the empty
"<|channel>thought<channel|>" block on enable_thinking=false, while the
12b/26B-A4B/31B family emits it (confirmed against google/gemma-4-E2B-it,
-E4B-it, -12b-it, -26B-A4B-it, -31B-it; the two families differ only in that
one block). The single PR #118 based template followed the larger-model
behavior, which is wrong for the E2B/E4B GGUFs this feature most targets.
- Add studio/backend/assets/chat_templates/gemma-4-edge.jinja: identical to
gemma-4.jinja minus the empty-thought-block, matching E2B/E4B behavior.
- Route unsloth/gemma-4-E2B-it-GGUF and -E4B-it-GGUF to the edge template;
12b/26B-A4B/31B keep gemma-4.jinja.
- Extend tests for the edge matcher, per-family routing, and the empty-thought
block difference (off for edge, on for standard).
* Studio: address review feedback on the gemma-4 template override
- Normalize owner-less shorthand model ids in the template matcher: a bare
"gemma-4-E2B-it-GGUF" is canonicalized to "unsloth/" the same way
ModelConfig.from_identifier does, so shorthand loads still get the override
(and the preserve_thinking capability) instead of falling back to the
embedded template.
- Scope the test's module stubs with unittest.mock.patch.dict instead of
sys.modules.setdefault, and only stub deps that are missing, so the global
module registry is not polluted for tests that run afterwards.
- Guard the Jinja render tests with pytest.importorskip("jinja2") so the suite
stays runnable in minimal Studio environments where jinja2 is not present.
- Add tests for shorthand resolution.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address 10-reviewer P1 findings on the gemma-4 template override
- /status no longer surfaces Studio's auto-applied bundled template as a
user-authored chat_template_override. The frontend adopts that field as
editable state and would otherwise re-send the gemma-4 template as an explicit
override for a later, unrelated model. /status now reports None when the live
override equals the model's auto-resolved bundled template.
- When a bundled family template is in effect, strip an inherited
--chat-template-file from llama_extra_args too (not only when the raw request
set chat_template_override). Otherwise a stale inherited template, appended
last, shadows the bundled one while Studio reports the bundled template's
capabilities.
- Write the temp chat-template file as UTF-8 explicitly, and keep the bundled
templates ASCII (replaced em dashes), so non-UTF-8 Windows locales cannot raise
UnicodeEncodeError or emit a mis-encoded template. Added an ASCII guard test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: serve DiffusionGemma GGUFs with the on-device visual decoder
* Studio: render the DiffusionGemma denoising canvas live in chat with honest stats
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DiffusionGemma runner resolution (Windows .exe, build/bin lookup, clear stale audio flag, safe PYTHONPATH, Linux-only pdeathsig)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* install.sh: persist ROCm-on-WSL drop-in even when rocminfo already works
_maybe_bootstrap_rocm_wsl calls _ensure_rocm_probe_env (which exports a
transient HSA_ENABLE_DXG_DETECTION + adds /opt/rocm/bin to PATH on the
installer process) right before the "rocminfo enumerates gfx1151 -> already
set up, return early" gate. On any reinstall over an existing /opt/rocm --
the common case, since the uninstaller keeps shared ROCm userspace but
removes /etc/profile.d/unsloth-rocm-wsl.sh -- that probe env makes rocminfo
succeed, so the gate returns 0 WITHOUT ever persisting the drop-in. The
transient env dies with the installer, so the next login shell (Studio,
llama-server) sees no GPU: torch cuda_avail=False, rocminfo finds nothing,
the llama.cpp ROCm prebuilt segfaults on a GPU it can't reach.
Factor the drop-in writer into _persist_rocm_wsl_dropin() and call it before
the early return so the persistent env is restored whenever librocdxg is
present. Idempotent (only writes when the drop-in is missing), gated on
librocdxg so it never fires on non-WSL/non-ROCDXG hosts, root-writes or
sudo-tees like before. The fast-path branch now reuses the same helper.
Reproduced on gfx1151 (Radeon 8060S) under dash (the curl|sh shell):
before the fix a reinstall left the drop-in absent and torch cuda_avail
False; after, the drop-in is persisted and a fresh login shell reports
cuda_avail True. Verified under both dash and bash, and idempotent on
re-run.
* Studio WSL: load system HIP before a prebuilt's bundled runtime (gfx1151)
The lemonade / published llama.cpp ROCm prebuilts bundle their own HIP
runtime (libamdhip64) built for bare-metal Linux. In WSL the GPU is reached
through the system ROCm's librocdxg bridge over /dev/dxg, which the bundled
runtime cannot drive -- it segfaults on the first GPU call. So:
- install_llama_prebuilt.py: the prebuilt's llama-quantize/llama-server
validation runs with the bundle dir first on LD_LIBRARY_PATH, segfaults
(empty stderr), and the install silently falls back to a CPU source build
(which on this host can't even build for GPU -- hipcc absent). The Strix
Halo WSL user ends up on CPU despite a working GPU.
- llama_cpp.py: even if a GPU prebuilt were kept, the serve-time launcher
put the bundle dir first too, so it would crash at load.
Fix: on a ROCDXG WSL host (gated on /dev/dxg + "microsoft" /proc/version +
a librocdxg-providing /opt/rocm), prepend the system ROCm lib dir to
LD_LIBRARY_PATH so the WSL-capable libamdhip64 + librocdxg load first, while
the bundle still supplies libggml-hip / librocblas with the gfx1151 kernels.
Set HSA_ENABLE_DXG_DETECTION=1 alongside. Added _wsl_system_rocm_lib_dirs()
to both modules (kept identical so a prebuilt that passed install validation
runs the same way at serve time). Strict no-op on bare-metal Linux, NVIDIA,
macOS, and Windows.
Verified on gfx1151 (Radeon 8060S) in WSL (ROCm 7.2.1 + librocdxg, Adrenalin
ROCDXG): before, the lemonade gfx1151 prebuilt segfaulted and the install
fell back to a broken CPU build; after, install_llama_prebuilt validates and
keeps the GPU prebuilt (source=published, prebuilt_fallback_used=False), and
Studio serves Qwen3-1.7B-GGUF at 53 tok/s with the model resident in GPU
memory (llama-server device_info: ROCm0 = AMD Radeon 8060S).
* tests: cover the WSL ROCDXG drop-in + system-HIP-ordering fixes
- _wsl_system_rocm_lib_dirs: no-op without /dev/dxg, on bare-metal Linux,
and on WSL without librocdxg; returns the system lib dir on a ROCDXG WSL
host.
- binary_env: prepends the system ROCm lib dir ahead of the bundle and sets
HSA_ENABLE_DXG_DETECTION on WSL; unchanged on bare-metal Linux.
- install.sh: _persist_rocm_wsl_dropin exists, is gated on librocdxg, and the
rocminfo-already-works early return calls it before returning.
- llama_cpp.py: the serve-time launcher prepends the WSL rocm dirs before the
bundle dir (mirrors binary_env).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten WSL ROCDXG fix comments (no logic change)
Condense the drop-in / system-HIP-ordering comments and docstrings added in
this PR. Verified comment-only via AST parse + py_compile + sh/bash -n, the
308-test rocm_support suite, and a dash functional re-run of the bootstrap
(drop-in still persisted, env still set).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/responses): forward chat_template_kwargs enable_thinking to chat request
The /v1/responses translation in _build_chat_request dropped
chat_template_kwargs (e.g. {"enable_thinking": true}) sent via the
Responses extra-body, so reasoning control was silently ignored.
Lift enable_thinking onto the typed ChatCompletionRequest field,
mirroring openai_chat_completions, so both the non-streaming and
streaming Responses pass-through paths honor it.
Fixes#6198
Signed-off-by: Tai An <antai12232931@outlook.com>
* Fix/adjust Responses reasoning for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust reasoning none for PR #6202
* Fix/adjust structured reasoning for PR #6202
* Fix/adjust responses reasoning review findings for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust responses reasoning follow-ups for PR #6202
* Fix/adjust think parsing gate for PR #6202
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add Tensor-Parallel llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Tensor-Parallel fallback and GPU selection
* Studio: reconcile split-mode extras and harden tensor-split planning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode extras in backend duplicate-load guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve inherited non-tensor split modes on reload
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor cancellation in tensor fallback, preserve tensor mode on rollback, and don't raise an explicit small context
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode in reload check and strip it on tensor downgrade
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip --tensor-split alongside --split-mode so inherited ratios don't override the tensor planner
An inherited or stale --tensor-split in llama_extra_args was appended after
Studio's computed --tensor-split and won last in llama.cpp, re-introducing the
asymmetric-GPU OOM tensor mode is meant to prevent. Group -ts/--tensor-split
into the split-mode shadow set so it is stripped on inherit and on the layer
fallback; parse_split_mode_override still keys on the mode value only.
* Drop quantized KV for the tensor attempt and report native max context
Tensor mode aborts on a quantized KV cache, so a user with q8_0/q4_1 etc. who
enabled Tensor Parallelism silently fell back to layer split. Clear the cache
type (and strip inherited/explicit --cache-type) for the tensor attempt only;
the layer fallback re-runs with tensor off and keeps the user's choice.
Also report max_available_ctx from the native context, not an explicit small
-c, so the context slider no longer warns too early in tensor mode.
* Reconcile inherited split-mode extras in the already-loaded check
When a same-model load omitted llama_extra_args, the tensor comparison resolved
the raw (None) request and treated an inherited --split-mode tensor server as a
mismatch, forcing a needless reload. Compare using the stored extras stripped
the same way the reload strips them.
* Pass tensor_parallel through compare-mode loads
The generalized compare path loaded each GGUF without tensor_parallel, so
compare ran layer split even with the toggle on and left the settings sheet
stale. Send the toggle and hydrate the loaded state from the response, matching
the main chat and recipe load paths.
* Add --tensor-parallel flag to unsloth studio run
The headless one-liner could only reach tensor mode by passing --split-mode
tensor as a raw llama.cpp extra. Add a first-class --tensor-parallel/
--no-tensor-parallel option that sets the tensor_parallel field on the
/api/inference/load payload, forwarded through the studio-venv re-exec like the
other polarity flags. Matches the web UI toggle and the API field.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: tune llama.cpp env for data-center GPUs
Detect datacenter/professional NVIDIA GPUs at llama-server launch and set
the llama.cpp env flags that help them, gated so consumer GeForce, AMD/ROCm,
CPU and macOS are never touched.
- GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1 for any DC GPU (FP32 cuBLAS
accumulation). On a B200 this is ~0% throughput cost with identical
perplexity (7.3230 wikitext-2-raw, baseline and on), where on GeForce the
same flag costs real throughput, hence the gate.
- GGML_CUDA_P2P=1 and CUDA_SCALE_LAUNCH_QUEUES=4x for multi-GPU DC boxes.
Benchmarked on 6x B200: +33-51% prompt processing on tensor (row) split and
+8-16% on the default pipeline (layer) split, with no regression on the
other split or on token generation.
Detection uses torch device names (A100/A30/H100/H200/H800/GH200/B200/GB200/
GB300/L40/L4/RTX PRO 6000/RTX 6000 Ada). A mixed box with one consumer GPU in
the selection is treated as non-DC. All writes are setdefault so a user value
always wins, and UNSLOTH_DISABLE_DC_TUNING=1 turns the whole thing off.
37 unit tests cover detection, multi-GPU gating, user-override precedence, the
disable flag and fail-open on error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix data-center GPU detection false positives and physical-id mapping
Two issues in the data-center llama.cpp env tuning gate:
- _is_datacenter_gpu matched the marker allowlist as unbounded substrings, so
workstation/laptop parts "NVIDIA RTX A1000" and "NVIDIA RTX A3000" matched
"a100"/"a30" and were wrongly tuned as data-center GPUs (forcing FP32 cuBLAS
accumulation and the multi-GPU env, which carry a real cost on those cards).
Switch to a word-boundary regex.
- gpu_indices carries physical GPU ids (translated from torch ordinals by
_get_gpu_free_memory via CUDA_VISIBLE_DEVICES), but they were passed straight
into torch.cuda.get_device_properties, which expects mask-relative ordinals.
On a masked host (e.g. CUDA_VISIBLE_DEVICES=4,5,6,7) a selection like [4,5]
fell out of range and silently dropped the tuning, and on a mixed mask it could
probe the wrong GPU class. Build a physical-id to device-name map mirroring
_get_gpu_free_memory, then look up the selection by physical id.
Add regression tests for the A1000/A3000 false positives and for masked-host
physical-id selection (reordered and mixed-class masks included).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten data-center GPU tuning comments
Comment-only pass over the DC tuning block and its tests: shorten verbose
docstrings/comments, drop ones that restate the code, collapse multi-line
blocks. Keep the load-bearing rationale (physical-id vs ordinal mapping, the
word-boundary reason, the B200 benchmark numbers). No code change: verified
with comment_tools.py check --strip-docstrings (code unchanged, comments only).
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: Keep tool group open while a tool call awaits confirmation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool confirmation session scope for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix confirmation follow-ups for PR #5869
* Apply pre-commit formatting for PR #5869
* Fix confirmation cleanup for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden confirmation lookups for PR #5869
* Studio: make the tool-call confirmation decision immutable
resolve_tool_decision accepted a second confirmation for the same approval_id
and overwrote slot["decision"] in the window before the waiter reads it and
pops the slot, so a duplicate or out-of-order POST could flip an Allow to Deny
(and returned a misleading resolved:true). Reject once the slot's event is
already set so the first decision wins. Adds a regression test.
* Fix/adjust tool confirmations for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* fix(studio): inherit llama_extra_args and honor --no-mmproj
Reloading the same GGUF from the UI without gguf_variant no longer drops
CLI pass-through args like --no-mmproj. Skip mmproj download and launch
when --no-mmproj is present in llama_extra_args.
Co-authored-by: Cursor <cursoragent@cursor.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): tighten GGUF llama_extra_args variant inheritance guard
Reject inherited CLI args when the request changes gguf_variant or when
omitted variant resolves differently from the stored extra_args source.
Co-authored-by: Cursor <cursoragent@cursor.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Treat --no-mmproj-auto and --mmproj-auto with last-wins parsing for PR #5902
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
When pinning GPUs for the llama-server child, the ROCm path set both
HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES to the same physical
indices. These masks filter at different layers and stack:
ROCR_VISIBLE_DEVICES reduces the visible set at the HSA/ROCr layer and
re-indexes from 0, then HIP_VISIBLE_DEVICES indexes into that reduced
set. _select_gpus ranks by free VRAM and picks the most-free card, so a
single non-zero pin (e.g. "1") becomes out of range at the HIP layer,
HIP enumerates 0 devices, and the model silently runs on CPU
("ggml_cuda_init: failed to initialize ROCm: no ROCm-capable device is
detected").
Set only HIP_VISIBLE_DEVICES (which narrows correctly on its own) and
clear any inherited ROCR mask so it can't double up.
Verified on a 2x Radeon AI PRO R9700 (gfx1201) host, ROCm 7.1.1: the
same selected=[1] load that fell back to CPU (~7.7 tok/s) now runs on
the GPU (~78 tok/s).
Fixes#6175
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving
A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.
Changes:
- After llama-server health, read default_generation_settings.n_ctx from
/props and adopt it whenever it is below Studio's computed context, with
a warning. The load response, status route, UI value, and the passthrough
max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
the passthrough drops whole middle turn-groups (system prompt, first turn,
and recent turns kept; tool calls stay paired with their results), clips
oversized contents middle-out when group-dropping is not enough, clamps
max_tokens to the generation headroom, and retries. Default stays 'error'
with code=context_length_exceeded so clients running their own compaction
keep full control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)
Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
always passes --parallel N, which silently splits -c into per-slot windows
of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
--fit-ctx <requested> for explicit requests so fit offloads or fails into
the existing --fit off retry instead of silently shrinking the window.
Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
* Studio: enable MTP for sub-3B Gemma separate-drafter GGUFs
The sub-3B auto-drop to ngram-mod was tuned for an embedded draft head
(Qwen), whose per-token cost regresses below 3B. Gemma ships the head as a
separate root mtp-*.gguf drafter, a tiny standalone model that is cheap
enough to win below 3B: B200 Q4_K_XL bench, draft-mtp n=2 vs spec-off,
gemma-4-E2B (2B) = 1.21x (accept ~0.65) while ngram-mod is 1.00x.
Exempt a separate drafter from the sub-3B gate everywhere the threshold is
applied: the resolver (_mtp_too_small), the auto-fit VRAM reserve, the
drafter auto-download decision, and the reload-skip mirror via a
has_separate_drafter flag on _auto_mode_drops_mtp. Embedded sub-3B heads
(Qwen) still drop to ngram-mod. A drafter the binary cannot build (older
prebuilt, or a CUDA kernel limit) still aborts the spawn and the load
retries once without speculative decoding.
Adds the full Qwen3.5 + Gemma-4 (regular and QAT) auto/off/forced resolver
matrix, plus explicit sub-3B exemption tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Always compare the separate drafter in the reload-skip mirror
The sub-3B wrapper around the drafter compare could skip it when the drafter
was deleted out from under a running sub-3B server (detected None, stored set),
leaving a stale launch. The resolved-path compare is cheap and already handles
every case, so drop the _auto_mode_drops_mtp guard (and its now-unused imports)
and always compare when the mode can use a drafter and the user does not own
--spec-type. Addresses review feedback on #6191.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: gracefully disable MTP when the model has no head or drafter
Selecting MTP or MTP+Ngram in Speculative Decoding on a GGUF with no nextn
head and no separate drafter aborted the whole load. llama-server does not
no-op an empty draft-mtp request: it exits with 'failed to measure MTP
context memory: failed to create llama_context', surfaced to the user as a
generic 'llama-server failed to start. Check that the GGUF file is valid
and you have enough memory.'
Build-time fix in _build_speculative_flags: when a forced mtp / mtp+ngram
mode targets a model with no MTP head and no drafter (is_mtp_model is
False), default back instead of emitting draft-mtp. mtp falls back to
--spec-default; mtp+ngram keeps the ngram-mod half, which needs no head.
Real MTP models (embedded head or separate drafter), sub-3B MTP overrides,
and the auto path are unchanged.
Runtime hardening: the existing post-launch MTP retry only fired for
separate-file drafters (--model-draft in spec_flags), so an embedded-head
model that the binary cannot build still hard-failed. Gate the retry on the
spec block requesting MTP, recognise the embedded-head abort strings
('failed to measure MTP context memory', 'failed to create llama_context'),
and make the drafter name None-safe in the warning.
Tests: extend the resolver matrix (forced mtp / mtp+ngram on a non-MTP
model) and add two cases asserting the default-back emission.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Adds an in-app "Update llama.cpp" banner and button to Unsloth Studio. When the installed prebuilt is behind the latest published release, a non-invasive banner appears; clicking Update downloads the latest prebuilt for this host and swaps it in place in the background, with no restart.
Detection reuses the freshness check from #5529. The update re-runs install_llama_prebuilt.py the same way setup.sh and setup.ps1 do after #5963: it forwards the published repo and the AMD gfx target derived from the install marker, and does not pass the removed --simple-policy or the arm64-only --cpu-fallback.
While the installer swaps binaries the backend enters a maintenance state (flag set under the serial load lock, active server unloaded) so a concurrent load cannot start a server from a half-swapped binary; the next load uses the new build. The banner also handles refused responses and jobs started in another tab so it never sticks on "Updating...".
Verified end to end on an NVIDIA B200: installed b9493, detected the update, applied it, and confirmed the binary at the same path advanced to b9585 in the same process. Hermetic backend tests and the frontend type-check pass.
* Studio: fall back to text-only when llama.cpp is too old for a model's vision projector
Loading a GGUF vision model starts llama-server with --mmproj <projector>. When the installed llama.cpp prebuilt predates the model's projector format, llama-server aborts at startup with 'clip.cpp: Unknown projector type' (exit -6), and the whole load failed even though the base GGUF is a fine text/tools chat model. Seen with gemma-4 on a 3-day-old prebuilt (build b9496).
load_model now retries the launch once without --mmproj when the captured startup output indicates a projector-format incompatibility. The retry runs the model text-only, marks the session non-vision (is_vision False, mmproj audio dropped) so the status/capabilities the frontend reads stay consistent, and warns the user to update llama.cpp. Detection is generic, not model-specific, and conservative: OOM, bad GGUF, port-bind, missing-file and other failures keep their existing handling and never retry. If the text-only retry also fails, it errors out with the real reason.
Adds _is_projector_incompatibility and _strip_mmproj_args (unit-tested with the real gemma-4 abort plus negatives) and extracts _start_llama_process so both the initial start and the retry share one spawn path and each logs its argv.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reformat mmproj fallback files to match main (ruff line-length 100 + kwarg spacing)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: support separate-file MTP GGUF drafters (Gemma 4)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix review findings for separate-file MTP drafters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pair local MTP drafters by name and include them in reload dedup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: manage --model-draft in extras and reject MTP/ copies as models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Windows installer winget msstore certificate failure
`winget install` was invoked without `--source winget`, so winget also
queried the msstore source. When msstore fails certificate pinning
(error 0x8a15005e, "The server certificate did not match any of the
expected values") winget aborts and demands `--source`, so the Python
(and uv) install fails even though the package exists in the winget
source.
- Pass `--source winget` to all winget install calls (Python x2, uv).
Both packages live in the winget source, so this is strictly correct
and skips the failing msstore round-trip entirely.
- Add a python.org fallback (Install-PythonFromPythonOrg) that downloads
the official installer and runs it silently per-user (no admin/UAC)
when winget is unavailable or fails for any reason. Mirrors the
existing uv -> astral.sh fallback so Python installs without manual
steps. Resolves the latest 3.13.x from python.org with a pinned
fallback, and selects the amd64/arm64/x86 installer per architecture.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pin remaining setup.ps1 winget calls to --source winget
Two winget invocations in studio/setup.ps1 still queried all sources and
could hit the same msstore certificate-pinning failure (0x8a15005e) that
broke the Python install in install.ps1:
- `winget show Nvidia.CUDA --versions` (CUDA Toolkit version probe)
- `winget install ... ShiningLight.OpenSSL.Dev` (OpenSSL dev for llama-server)
Every other winget call in this file already passes `--source winget`
(Git, CMake, VS Build Tools, CUDA install, Node.js, and setup.ps1's own
Python 3.12 install), so these two were stragglers. Both packages live in
the winget source; pinning it makes setup robust to an unhealthy msstore
source, matching the rest of the file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stop amd-smi GPU probe from popping a DiskPart UAC prompt
On Windows, AMD GPU detection in install.ps1 and studio/setup.ps1 runs
`amd-smi list` / `static --asic` / `version`. amd-smi (shipped in
System32 by the Adrenalin driver) auto-elevates to read GPU/APU memory
details, surfacing a confusing DiskPart UAC prompt mid-install. The
Studio backend already documents and circuit-breaks on this in
studio/backend/utils/hardware/amd.py, but the installers did not.
Add an Invoke-AmdSmiNoElevate helper (both scripts) that runs amd-smi via
Start-Process under __COMPAT_LAYER=RunAsInvoker so it cannot auto-elevate
(no prompt), with a 30s timeout (matching amd.py) so a flaky amd-smi
cannot stall the install for minutes. On failure/timeout the existing WMI
name -> gfx fallback still resolves the arch, so detection is unchanged on
working hosts.
Verified on a Strix Halo (Radeon 8060S / gfx1151) box: the prompt is gone
and the probe is bounded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add experimental ROCm-on-WSL setup helper for Strix Halo (gfx1151)
install.sh already routes gfx1151 (Radeon 8060S / Strix Halo) to the
repo.amd.com/rocm/whl/gfx1151 wheels once a ROCm runtime is present, but
it does not install AMD's driver/ROCm stack -- a large, admin-gated
prerequisite. scripts/install_rocm_wsl_strixhalo.sh automates the Linux
side on a dedicated Ubuntu 24.04 WSL2 distro: ROCm 7.2 (wsl usecase), the
rocr4wsl HSA runtime, a librocdxg build, env setup, and a PyTorch gfx1151
GPU smoke test. A hard preflight refuses to run until the Adrenalin
>=26.3.1 driver is actually present, so it cannot half-install.
Procedure adapted from AMD's ROCm-on-WSL docs and community gfx1151 notes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Detect AMD GPUs by name so native Windows gets a GPU llama.cpp
The gfx-arch inference from the WMI GPU name was gated behind $HasROCm,
which the hipinfo/amd-smi probe leaves false on the common Windows case
(Adrenalin driver only, no HIP SDK -- and amd-smi often cannot read the
arch without elevation). So an AMD GPU was detected by name but never
mapped to a gfx target, --rocm-gfx was not forwarded, and studio setup
fell back to a CPU llama.cpp build.
Un-gate the inference (install.ps1 + studio/setup.ps1) so it runs whenever
an AMD GPU name is available. The inferred gfx is forwarded as --rocm-gfx,
which makes install_llama_prebuilt.py download the matching lemonade-sdk
ROCm prebuilt (e.g. llama-bNNNN-windows-rocm-gfx1151-x64.zip) -- a
GPU-accelerated llama.cpp that bundles its own ROCm runtime, so it runs
with just the Adrenalin driver. PyTorch's ROCm wheels still require a
confirmed HIP SDK ($HasROCm), so this only affects llama.cpp / inference
and never pulls broken ROCm torch.
Also broaden the name->arch table to every family lemonade ships Windows
assets for: gfx120X (RDNA 4), gfx110X (RDNA 3), gfx1151/gfx1150
(RDNA 3.5), and gfx103X (RDNA 2). Unknown names still fall back to CPU.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Suppress amd-smi DiskPart UAC prompt in the Python install/runtime paths
The earlier PowerShell guard covered install.ps1 / setup.ps1, but the
Python installer (install_llama_prebuilt.py detect_host,
install_python_stack.py ROCm probes) and the Studio backend monitor
(amd.py) also shell out to amd-smi on Windows, where it auto-elevates and
pops the same DiskPart UAC prompt mid-install / at runtime.
Inject __COMPAT_LAYER=RunAsInvoker into the amd-smi subprocess env on
Windows so it runs un-elevated (no prompt). Callers already tolerate an
empty/failed result and fall back to WMI / name detection (installer) or
the existing circuit breaker (amd.py). Gated to Windows so Linux/macOS
amd-smi behaviour is unchanged.
- install_llama_prebuilt.py: handled centrally in run_capture (covers
detect_host's `amd-smi list` and the version probe).
- install_python_stack.py: new _amd_smi_env() helper on its 3 raw
subprocess.run amd-smi calls.
- amd.py: merge RunAsInvoker into the existing child env.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Tighten AMD GPU name->arch patterns to avoid mismatches
The W9[0-9]{3} and RX 90[0-9]{2} patterns added for RDNA 4 were
speculative and over-broad: W9xxx would also match old GCN FirePro
W9100/W9000 cards (wrong gfx1201 -> a lemonade gfx120X download that
fails validation), and RX 90[0-9]{2} was redundant with the explicit
9070/9060 entries. Drop both; keep only confirmed RDNA 4 SKUs. Unmatched
AMD names still fall back cleanly to CPU.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fetch the llama.cpp validation model via huggingface_hub
The prebuilt validation downloads a tiny GGUF test model from huggingface
via bare urllib. On Windows / proxy setups where the server sends an
incomplete TLS chain, urllib cannot complete the Amazon CA chain (it does
no AIA intermediate fetching) and fails with CERTIFICATE_VERIFY_FAILED, so
a perfectly good GPU prebuilt is rejected and the installer falls back to a
CPU source build.
Route the validation-model download through huggingface_hub
(hf_hub_download) -- the same mechanism Studio uses for model downloads,
which completes the chain where urllib cannot -- keeping the direct URL as
a fallback. This lets the lemonade ROCm prebuilt validate and install on
cert-restricted machines (verified: hf_hub_download succeeds where urllib
returns CERTIFICATE_VERIFY_FAILED).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Guard the remaining raw amd-smi version probe via run_capture
A ROCm-version detector in install_llama_prebuilt.py called amd-smi version through a raw subprocess.run that bypassed run_capture's Windows RunAsInvoker guard, so it still triggered the DiskPart UAC prompt during setup. Route it through run_capture like the other amd-smi calls.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Forward --rocm-gfx even when the ROCm runtime is unconfirmed
setup.ps1 forwarded --rocm-gfx (and picked the windows-hip llama.cpp
prebuilt) only inside `if ($HasROCm)`. On Adrenalin-only hosts (amd-smi
present but no HIP SDK, so $HasROCm stays false) the gfx arch was
name-inferred but never forwarded, so install_llama_prebuilt.py saw
has_rocm=False and installed the CPU build -- even though the lemonade
gfx1151 GPU prebuilt runs fine there (it bundles its own ROCm runtime;
verified: llama-cli --list-devices -> ROCm0: AMD Radeon 8060S, 69 GB).
Forward --rocm-gfx whenever a gfx arch is known (it is authoritative and
implies ROCm in install_llama_prebuilt.py), and treat a known gfx arch as
windows-hip in the existing-install mismatch check. --has-rocm stays gated
on the confirmed-runtime signal.
Verified on Radeon 8060S / gfx1151: the installer now selects, validates,
and installs llama-b1286-windows-rocm-gfx1151-x64.zip (ROCm DLLs present)
instead of the CPU build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Install AMD ROCm PyTorch on name-inferred gfx hosts (enables Train/Export)
setup.ps1 picked the AMD ROCm PyTorch wheels only inside `if ($HasROCm ...)`.
On Adrenalin-only hosts (amd-smi present but no HIP SDK, so $HasROCm is
false) the gfx arch was name-inferred but the ROCm-wheel branch never ran,
so the host got torch+cpu. With CPU torch, torch.cuda.is_available() is
False, so the Studio backend sets CHAT_ONLY=True and hides Train/Export.
Un-gate the ROCm PyTorch index resolution on a known gfx arch (mirrors the
llama.cpp --rocm-gfx fix). AMD's per-arch Windows wheels
(repo.amd.com/rocm/whl/<gfx>) bundle the ROCm runtime, so they work without
a HIP SDK; a failed install still falls back to CPU.
Verified on Radeon 8060S / gfx1151: torch 2.11.0+rocm7.13.0 installs and
torch.cuda.is_available() -> True, device "AMD Radeon(TM) 8060S Graphics",
GPU matmul OK -> CHAT_ONLY=False -> Train/Export enabled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Force amd-smi un-elevated process-wide in the Python installers
Guarding individual amd-smi call sites kept missing some (install_python_stack.py's probe loop and its Windows GPU re-check), so the DiskPart UAC prompt kept reappearing. Set __COMPAT_LAYER=RunAsInvoker process-wide at the top of install_python_stack.py and install_llama_prebuilt.py on Windows so every amd-smi subprocess (current and future) runs un-elevated with no per-call guard. Safe: these scripts only spawn amd-smi/rocminfo/hipinfo probes and pip/uv. setup.ps1 keeps per-call guards because it also spawns winget installers that need elevation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix Invoke-AmdSmiNoElevate exit code on PS 5.1 + RX 7700S arch match
Start-Process -PassThru leaves the returned process object's .ExitCode
$null after WaitForExit on Windows PowerShell 5.1, so the helper set
$LASTEXITCODE to $null and every caller's `if ($LASTEXITCODE -eq 0 ...)`
was always false -- the amd-smi GPU / gfx-token / ROCm-version detection
branch was effectively dead (masked only because the un-gated WMI
name->gfx inference still ran). Reproduced on PS 5.1.26100.
Rewrite the helper to use [System.Diagnostics.Process]::Start with a
ProcessStartInfo (UseShellExecute=false), whose .ExitCode is reliable,
with async stream reads (ReadToEndAsync) to avoid a pipe-buffer deadlock
and WaitForExit(timeout) to bound a flaky amd-smi. __COMPAT_LAYER=
RunAsInvoker (inherited via the process env) still suppresses the
auto-elevation / DiskPart prompt. Also drops the temp files and the
empty-ArgumentList edge case. Verified: exit code propagates
(7 -> $LASTEXITCODE=7), output captured, env restored.
Also fix the gfx1100 name pattern `RX 7700(?! S)` -> `RX 7700(?!S)` so the
spaceless retail name "RX 7700S" is correctly excluded (it belongs to the
gfx1102 row). Both found by PR review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address PR review follow-ups (install.sh table, update path, tests, WSL)
From the multi-agent PR review:
- install.sh: sync the AMD name->arch table with install.ps1 / setup.ps1
(the bash table had drifted to the old narrow patterns). Adds RDNA 2
(gfx103X), workstation PRO W SKUs, and more Strix Halo/Point names, and
orders gfx1102 before gfx1100 so the spaceless retail name "RX 7700S"
resolves correctly (bash case has no negative lookahead). AMD-ROCm-only:
the name inference stays gated behind _has_amd_rocm_gpu(), so NVIDIA /
CPU / macOS are unaffected.
- setup.ps1: the "dependencies up to date" fast path skipped the torch
reinstall, so an existing user who had CPU torch (installed before
ROCm-wheel support) stayed stuck in CHAT_ONLY. Now, when an AMD gfx arch
is known AND the installed torch is CPU-only, don't skip -- force the
dependency pass so the ROCm wheels install.
- scripts/install_rocm_wsl_strixhalo.sh: resolve the real /opt/rocm dir
instead of hardcoding ROCM_VER for LD_LIBRARY_PATH / the librocdxg
symlink (breaks if amdgpu-install lays ROCm under a patch-version dir);
add a LIBROCDXG_REF pin knob and a "verified against" freshness header.
- tests/studio/install/test_pr5940_followups.py: cover _hf_resolve_url_parts,
_fetch_validation_model_bytes (hf path + urllib fallback), run_capture's
Windows-only amd-smi RunAsInvoker injection, and install.ps1 vs setup.ps1
name-table parity (catches future drift). 14 tests, all passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix DiskPart UAC prompt: skip amd-smi on Windows without a HIP SDK
On Windows, amd-smi re-initialises the ROCm runtime on every invocation
(even `amd-smi version`) and, on hosts without a working HIP runtime
(consumer APUs/dGPUs with only the Adrenalin driver), elevates a child
process at runtime -- popping a UAC/DiskPart prompt. amd-smi's own
manifest is asInvoker, so __COMPAT_LAYER=RunAsInvoker cannot suppress
that runtime elevation (verified: even `amd-smi version` hangs and
times out with RunAsInvoker set).
Replace the ineffective RunAsInvoker-only approach with a real gate:
only spawn amd-smi on Windows when a HIP SDK is detectable (hipinfo
present, so amd-smi runs un-elevated) or the user opts in with
UNSLOTH_ENABLE_AMD_SMI=1. The gfx arch is already resolved from WMI
name inference (forwarded via --rocm-gfx), so ROCm wheel + lemonade
llama.cpp selection is unaffected. Linux/macOS amd-smi never elevates
and is untouched (no regression). RunAsInvoker is kept as harmless
belt-and-suspenders for tools that DO use manifest elevation.
Applied consistently across:
- studio/backend/utils/hardware/amd.py (runtime GPU polling)
- install.ps1, studio/setup.ps1 (install-time detection)
- studio/install_llama_prebuilt.py (prebuilt arch probe + version)
- studio/install_python_stack.py (ROCm version + arch probe)
Verified live on AMD Radeon 8060S (gfx1151), native Windows: fresh
install detects the GPU, installs ROCm torch (torch.cuda.is_available()
True), launches Studio with no DiskPart prompt, and inference, tool
calling, web search, LoRA finetuning, and GGUF export all run on the GPU.
Tests: add 6 _amd_smi_allowed() gating tests + PowerShell-installer gate
assertions; update the three amd-smi monitoring tests to opt in (they
mock amd-smi as available). Full suite: 267 passed, 2 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* install.sh: helpful WSL message when the GPU isn't exposed to ROCm
In WSL, an AMD GPU's ROCm-on-WSL runtime is only available with a recent
Adrenalin driver AND a distro AMD supports (currently Ubuntu 24.04). When
neither is in place, GPU detection (rocminfo/_has_amd_rocm_gpu) finds
nothing and we silently fall back to CPU.
Add an actionable hint in the CPU-fallback path, shown only on WSL and
only AFTER detection has already failed -- so it is forward-compatible:
the moment a driver/distro DOES expose the GPU (e.g. if AMD later adds
Ubuntu 26.04 support), detection succeeds and the hint never fires. The
message:
- notes a GPU is plumbed in (/dev/dxg) but no ROCm runtime is exposed,
- lists the two prerequisites (Adrenalin driver + Ubuntu 24.04),
- if the distro is not 24.04, says AMD may not support it yet,
- tells the user to `wsl --install Ubuntu-24.04` and re-run,
- links AMD's ROCm-on-WSL guide + the experimental Strix Halo helper.
Verified live: on Ubuntu-24.04 the hint shows (version-warning omitted)
and the CPU install completes; on Ubuntu-26.04 the extra "this distro may
not be supported" line appears and points to 24.04.
Also fix the experimental scripts/install_rocm_wsl_strixhalo.sh: AMD's
repo.radeon.com/amdgpu-install/ is indexed by unified installer version
(30.30, 31.30, ...), NOT ROCm version, so the hard-coded
amdgpu-install/7.2.0/ path 404'd. Scan the installer dirs newest-first
for a noble .deb matching the target ROCm major.minor (ROCm 7.2 ->
30.30.x/amdgpu-install_7.2.x), falling back to the newest available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* WSL: fix shortcut collision + pin ROCm-on-WSL driver reqs from AMD docs
Two WSL-related fixes informed by AMD's official ROCm-on-WSL docs and
field reports for Strix Halo / Ryzen AI Max+ (Radeon 8060S, gfx1151):
1. Shortcut collision (real bug). install.sh's WSL branch wrote
"Unsloth Studio.lnk" to the SAME Desktop / Start Menu folder as the
native-Windows installer (install.ps1 New-StudioShortcuts). Running
install.sh in WSL therefore silently retargeted the native shortcut at
the WSL launcher (wt.exe -> wsl.exe), so the desktop/start-menu icon
stopped launching native GPU Studio. Now the WSL shortcut uses a
DISTINCT name -- "Unsloth Studio (WSL - <distro>).lnk" -- and fetches
the Unsloth .ico to %LOCALAPPDATA%\Unsloth Studio so it shows the
proper icon. Native and WSL shortcuts now coexist.
2. Precise ROCm-on-WSL prerequisites. Research (AMD radeon-ryzen WSL
compatibility matrix, gianni.rosagallina.com Feb-2026 guide,
ROCm/ROCm#4952/#5509/#6022) confirms WSL GPU on Strix Halo requires
AMD Adrenalin Edition >= 26.1.1 (26.2.2+ is the first production
ROCDXG/WSL release) + ROCm 7.2.1 + Ubuntu 24.04; an older driver does
not inject the ROCm/DXG runtime into /usr/lib/wsl/lib, so rocminfo sees
only the CPU. install.sh's WSL hint and the experimental
install_rocm_wsl_strixhalo.sh header/preflight now state the exact
driver version (was a guessed ">=26.3.1"), bump ROCM_VER to 7.2.1, link
AMD's radeon-ryzen docs, and document the known librocdxg caveat that
usable VRAM is currently capped at the .wslconfig memory setting.
bash -n clean; install test suite 267 passed, 2 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer: hint when the AMD driver is too old for ROCm-on-WSL
Adds a detect-and-guide hook for the optional WSL-GPU path. An AMD GPU on
native Windows can also be used inside WSL2, but only with AMD Adrenalin
Edition >= 26.2.2 (the first production ROCDXG/WSL release). Native Windows
GPU works with any recent driver, so this is purely about enabling the WSL
path.
We intentionally do NOT auto-install the driver: AMD referrer-gates driver
downloads (scripted curl/Invoke-WebRequest are blocked) and does not publish
Adrenalin via winget, so no installer can reliably fetch it -- and silently
swapping a live display driver is risky. Instead we point the user at AMD's
official download page (one click), after which the existing WSL detection
lights up automatically.
- install.ps1: new Show-AmdWslDriverHint -- when an AMD GPU is present and the
installed driver predates the 26.2.2 release (DriverDate < 2026-02-01),
print a concise tip with the AMD download URL. Handles DriverDate as either
a CIM DateTime or a WMI string. Suppress with UNSLOTH_SKIP_AMD_DRIVER_HINT=1.
- install.sh (WSL hint): add the direct Adrenalin 26.2.2 download URL and note
that AMD downloads are referrer-gated (open in a browser).
Verified: hint fires on a Sept-2025 driver, auto-suppresses on >= 2026-02-01;
install.ps1 parses; install.sh bash -n clean; suite 267 passed, 2 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* install.ps1: refresh shell icon cache after creating the shortcut
After writing the Desktop / Start Menu .lnk, nudge Explorer to refresh
its icon (ie4uinit.exe -show). Without this, a stale icon cache can show
a blank shortcut icon until the next explorer restart -- most visible
when a shortcut of the same name was rewritten (e.g. a native install
followed by a WSL install, which previously shared the name; now they use
distinct names, but the cache nudge makes the icon appear immediately
regardless). Best-effort and wrapped in try/catch so it never fails the
install. The bundled unsloth.ico itself is valid (verified it renders).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* setup.ps1: don't silently CPU-build llama.cpp on an AMD GPU
For AMD, GPU acceleration comes from the lemonade ROCm prebuilt (it bundles
the ROCm runtime, no HIP SDK needed) and is the preferred/default path. The
source-build fallback is CPU-only -- a HIP/ROCm *source* build would need the
full HIP SDK + ROCm clang toolchain, which the prebuilt exists to avoid.
Previously, if an AMD-GPU host ever fell through to the source build (e.g. the
prebuilt could not be downloaded), it printed "building llama.cpp (CPU-only,
no NVIDIA GPU detected)" and quietly produced a CPU binary -- masking the lost
GPU acceleration. Now that case emits a loud [WARN] explaining the GPU prebuilt
is the AMD path and how to restore it (re-run / check network / set
UNSLOTH_LLAMA_RELEASE_TAG), so AMD never silently degrades to CPU.
No behavior change on the happy path: AMD still gets the GPU prebuilt (verified
on gfx1151: ggml-hip.dll bundled, ~80% GPU compute during inference). NVIDIA
(CUDA source build) and CPU-only hosts are unchanged.
setup.ps1 parses; install suite 267 passed, 2 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* uninstall: remove shared llama.cpp build, kill lock-holders, match WSL shortcut
Three gaps found by running a real uninstall on a native-Windows + WSL host;
all fixes are scoped to Unsloth-owned paths and no-op on the other pathways
(env/custom-root, NVIDIA/AMD/CPU, Mac) so nothing else regresses.
uninstall.ps1:
- Remove the default-mode SHARED llama.cpp build + cache. setup.ps1 installs
them at ~/.unsloth/llama.cpp and ~/.unsloth/.cache -- SIBLINGS of studio,
not under it -- so deleting <studio> left hundreds of MB behind. Now removed
explicitly, then ~/.unsloth is dropped ONLY if empty (never nukes unrelated
content). No-op in env/custom mode (llama.cpp nests under the custom root,
removed already) and when absent. UNSLOTH_LLAMA_CPP_PATH (user-owned) is kept.
- New _StopProcessesLockingRoots: _StopStudioProcesses only matched the venv
unsloth/python/studio exe, so it missed (a) llama-server.exe under llama.cpp
and (b) an orphaned multiprocessing python fork that ran from the SYSTEM
python but loaded a venv DLL (bitsandbytes) -- on Windows an open DLL handle
blocks the directory delete, leaving a half-removed install. The new helper
kills any process whose image path OR loaded module is under a target root
(module scan scoped to python/unsloth/llama-server names; vendor-agnostic).
- _RemovePath now retries (transient post-kill handle release).
uninstall.sh:
- Remove the default-mode ~/.unsloth/llama.cpp + ~/.unsloth/.cache; rmdir
~/.unsloth only if empty.
- WSL Windows-side shortcut cleanup now matches by TARGET (any
"Unsloth Studio*.lnk" whose target launches wsl.exe), covering both the
legacy "Unsloth Studio.lnk" and the new "Unsloth Studio (WSL - <distro>).lnk"
-- and never removes a native-Windows shortcut (which launches wscript.exe).
uninstall.ps1 parses; uninstall.sh passes sh -n and bash -n.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* install.ps1: invalidate Win11 Start Menu tile cache after creating shortcut
The Start Menu shortcut kept showing a blank/generic icon even after the
Explorer icon-cache rebuild, because Windows 11's StartMenuExperienceHost
keeps its OWN pre-rendered tile-icon cache
(%LOCALAPPDATA%\Packages\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\
TempState\TileCache_*.bin + StartUnifiedTileModelCache.dat), separate from
Explorer's iconcache_*.db. ie4uinit and an explorer.exe restart do not touch
it, and they don't recycle the host -- so a rewritten same-name shortcut keeps
showing the first-rendered (often the generic wscript ">") tile until the host
restarts on its own.
Fix: after creating the shortcut, drop only the Start Menu RENDER caches
(TileCache_* + StartUnifiedTileModelCache.dat) and stop StartMenuExperienceHost
(Windows auto-relaunches it), so the tile re-resolves the real icon via the
shell image factory. start2.bin (the user's pinned layout) is deliberately
preserved. Guarded by Test-Path (Windows 10 has no such host -> skipped) and
wrapped in try/catch so it can never fail the install. Windows-only
(install.ps1); no effect on Linux/macOS/Studio.
Verified live: rendering the shortcut via IShellItemImageFactory::GetImage (the
API StartMenuExperienceHost uses) returns the Unsloth sloth icon, color-matched,
after this invalidation -- previously it returned the generic script tile.
install.ps1 parses; install suite 267 passed, 2 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ROCm-on-WSL for AMD Strix Halo (gfx1151): auto-setup + runtime enablement
Make Unsloth Studio set up ROCm-on-WSL automatically for AMD Strix Halo
(Radeon 8060S / gfx1151) and use the GPU at runtime, validated end-to-end
on a Ryzen AI Max+ PRO 395 (ROCm 7.2.1 + librocdxg + Adrenalin Apr-2026):
rocminfo enumerates gfx1151, torch.cuda True, ~85.8 GB UMA pool.
Every change is a strict no-op for all other configs (NVIDIA/CUDA,
discrete + native-Linux AMD ROCm, macOS/MLX, Windows, CPU-only, non-Strix
WSL) and can never abort the installer.
- scripts/install_rocm_wsl_strixhalo.sh: rewrite to the validated recipe.
Fixes that would have broken a working box: drop the /usr/lib/wsl/lib
preflight (a working ROCDXG host has only d3d12/dxcore there); remove the
obsolete rocr4wsl step (gone from the 7.2.1 repo; would hard-fail and also
rips out the standard hsa-rocr ROCDXG needs); dynamic librocdxg soname
(was hardcoded 1.1.0; build is 1.2.0); direct apt-repo install; Windows
SDK auto-discovery; persist env to /etc/profile.d + ~/.bashrc; idempotent.
- install.sh: _maybe_bootstrap_rocm_wsl auto-offers/runs the helper when it
detects a Strix Halo APU in WSL (/dev/dxg) with no ROCm runtime, then
loads the env so detection routes to the gfx1151 wheels. Fast-path when
already configured. Fix an inaccurate WSL hint line.
- studio/backend/main.py + worker.py: set HSA_ENABLE_DXG_DETECTION=1
in-process before torch (gated on /dev/dxg AND librocdxg.so), so the
worker uses the GPU even when launched outside a login shell. Mirrors the
existing BNB_ROCM_VERSION injection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* uninstall: clean up ROCm-on-WSL artifacts + Start Menu tile cache
- uninstall.sh: remove the ROCm-on-WSL helper artifacts -- the librocdxg
build clone (~/.unsloth/librocdxg, which otherwise blocks the empty-dir
rmdir of ~/.unsloth), the throwaway smoke-test venv, the persisted env
(/etc/profile.d/unsloth-rocm-wsl.sh) and the ~/.bashrc block. The system
ROCm userspace is a shared prereq like CUDA and is kept by default;
UNSLOTH_UNINSTALL_ROCM=1 removes it too. No-ops on macOS / non-Strix Linux.
- uninstall.ps1: invalidate the Win11 Start Menu tile cache after removing
the shortcut so its tile disappears promptly (mirrors install.ps1),
preserving start2.bin.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer: accurate AMD ROCm messaging (HIP SDK optional, not required)
The Windows installer printed "HIP SDK not found - GPU-accelerated training
unavailable" / "ROCm wheels require the HIP SDK" whenever the HIP SDK was
absent. That is misleading: for a detected AMD GPU arch (gfx1151 etc.),
setup.ps1 installs AMD's bundled-runtime ROCm PyTorch wheels (repo.amd.com)
which ship their own ROCm runtime and do NOT need the HIP SDK -- verified
end-to-end (torch 2.11.0+rocm7.13.0, cuda True, QLoRA training on GPU) on a
Radeon 8060S with no HIP SDK installed.
Gate the GPU-detection + rocm-step messages on a detected gfx arch: when one
is known, state that GPU PyTorch uses bundled-runtime wheels and the HIP SDK
is optional; only when the arch is unknown fall back to the HIP-SDK hint.
Behavior (torch routing) is unchanged; this is messaging only. No-op for
NVIDIA/CUDA, HIP-SDK-present, and CPU paths (they hit earlier branches).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer: fix /opt/rocm data-loss + make WSL shortcut create/remove interop-robust
Two fixes from the 3-reviewer regression audit + live testing on a
systemd-enabled WSL distro (interop disabled):
F1 (data-loss, install_rocm_wsl_strixhalo.sh): the /opt/rocm symlink-repair
could force-delete a pre-existing REAL ROCm install. The guard only checked
that /opt/rocm is a real directory, not that it is the stray librocdxg stub.
Now it only touches /opt/rocm when it is NOT a real install (no bin/rocminfo,
bin/hipcc, or .info/version present), and MOVES it aside (rocm.unsloth-stub-bak)
instead of deleting it, so a wrong guess can never lose data.
WSL interop robustness (install.sh + uninstall.sh): both relied on
`command -v powershell.exe`, which is true even when WSL interop cannot EXECUTE
it (on systemd distros powershell.exe fails with "Exec format error"). Result:
the WSL shortcut silently failed to create (install) and to remove (uninstall).
- uninstall.sh: test that powershell.exe actually runs; if not, remove the
"Unsloth Studio (WSL...).lnk" files directly via drvfs (/mnt/<drive>), which
works without interop. The name is WSL-install-specific, so a native install's
"Unsloth Studio.lnk" is never touched.
- install.sh: when the shortcut cannot be created, warn with the manual launch
command + how to re-enable interop, instead of failing silently.
No behavior change on the interop-on path. The regression audit otherwise found
no regressions on Linux/Mac/Windows/CPU/NVIDIA install paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* install.sh: fast-path fully restores ROCm-on-WSL env when the drop-in is gone
Reinstall regression found by uninstall->reinstall testing: after a Studio
uninstall that removed /etc/profile.d/unsloth-rocm-wsl.sh but KEPT the shared
ROCm (the default), a non-login reinstall hit the bootstrap fast-path
(librocdxg present) and its else-branch only set HSA_ENABLE_DXG_DETECTION --
NOT PATH/LD_LIBRARY_PATH. So rocminfo was not on PATH, GPU detection failed,
and the installer fell back to CPU-only PyTorch.
Fix: when librocdxg is present but the env drop-in is missing, restore the
FULL env inline (HSA + TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL + PATH +
LD_LIBRARY_PATH) so rocminfo is found and detection routes to the GPU, and
recreate /etc/profile.d/unsloth-rocm-wsl.sh so future shells and the Studio
worker get it too. No change to the env-present fast-path or any other host.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer: clear Explorer icon cache so shortcut icons aren't blank
Root cause of the persistent blank Desktop + Start Menu icons: Explorer caches
each shortcut's icon in iconcache_*.db and does NOT re-read the .ico when a
same-name .lnk is recreated across reinstalls. The .ico and .lnk are correct
(the shell renders them non-blank via IShellItemImageFactory; the .ico has real
image data at 16/32/48/128 px), but the stale cache entry wins. The previous
fix only ran a weak `ie4uinit -show` + the Start Menu tile-cache clear -- it
never invalidated Explorer's icon cache, so the desktop icon stayed blank.
Fix (native install.ps1 New-StudioShortcuts AND the WSL shortcut path in
install.sh):
- ie4uinit -ClearIconCache (thorough; replaces -show as the primary refresh)
- SHChangeNotify(SHCNE_ASSOCCHANGED) to force a live desktop/taskbar refresh
WITHOUT restarting explorer
- keep the Win11 Start Menu tile-cache invalidation (and add it to the WSL
shortcut path too, preserving start2.bin)
Non-disruptive (no explorer restart). install.ps1 parses clean; install.sh
passes bash -n + dash -n; the heredoc-generated WSL PowerShell parses clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer: per-item SHChangeNotify(UPDATEITEM) reliably fixes blank icons
The blank Desktop/Start Menu shortcut icons are a stale Explorer PER-ITEM icon
cache: when a same-name .lnk is recreated across reinstalls, Explorer caches the
previously-resolved (often generic "white page") icon for that item and won't
re-extract the .ico on its own. The .ico and the .lnk's IconLocation are correct
(every icon API renders the sloth) -- only Explorer's cached display is stale.
The previous refresh (ie4uinit -ClearIconCache + a GLOBAL SHCNE_ASSOCCHANGED
broadcast) does NOT recover a stale item -- confirmed by reproduction. The
reliable, NON-disruptive fix (no explorer restart) is a PER-ITEM
SHChangeNotify(SHCNE_UPDATEITEM, SHCNF_PATHW, <lnk path>) for each created
shortcut, which forces Explorer to re-read that exact item's icon.
Verified end-to-end: deliberately staled a shortcut to the generic icon, ran the
installer's exact new refresh code, and the sloth icon recovered with NO explorer
restart (confirmed by capturing the live desktop via PrintWindow).
Applied to both native install.ps1 (New-StudioShortcuts) and the WSL shortcut
path in install.sh. Still clears the on-disk icon cache (ie4uinit) and the Win11
Start Menu tile cache (preserving start2.bin).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* uninstall: remove leftover llama.cpp .staging root so ~/.unsloth is cleaned
The llama.cpp atomic-install staging root (install_llama_prebuilt.py
INSTALL_STAGING_ROOT_NAME=.staging) is a sibling of the llama.cpp install
dir (~/.unsloth/.staging in default mode). It is normally pruned after a
successful activate, but an interrupted or retained build can leave a
<name>.staging-XXXX tree behind. The uninstallers removed llama.cpp and
.cache but not .staging, so the final empty-dir cleanup of ~/.unsloth failed
and the directory lingered. Reproduced on WSL (Ubuntu-24.04) where an empty
llama.cpp.staging-XXXX dir kept ~/.unsloth alive after uninstall.
Remove ~/.unsloth/.staging in both uninstall.sh and uninstall.ps1. No-op in
env/custom mode (staging nests under the custom root removed already) and
when absent. Cross-platform fix (the staging logic is platform-agnostic).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer: WSL-absent hint + fix here-string lint false positive
install.ps1: in the AMD WSL-ROCm driver hint, detect when wsl.exe is absent
and add a one-line "wsl --install -d Ubuntu-24.04" pointer so a Strix Halo
user with no WSL yet gets an actionable next step (the hint previously assumed
an Ubuntu-24.04 distro already existed). Best-effort, informational only.
test_rocm_support.py: test_no_here_strings did a crude substring check that
false-positived on the conda-style block marker
printf '# <<< Unsloth ROCm-on-WSL (gfx1151) <<<' -- a string literal written
into the /etc/profile.d drop-in, also used as a sed delimiter pair by
uninstall.sh, not a here-string. Strip quoted spans before the check so the
lint still catches a real here-string operator but ignores quoted literals.
install.sh remains POSIX-clean (sh -n / dash -n / bash -n all pass).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* installer: address PR review comments (gfx1150 mapping, amd-smi opt-out, WSL bootstrap, SDK path, make)
Apply the valid bot review findings on #5940; reject the ones that don't hold.
Fixed:
- AMD name->gfx table (setup.ps1 + install.ps1): Radeon 890M and Ryzen AI 9 HX
370/375 are Strix POINT (gfx1150), not Strix Halo (gfx1151). Move 890M / HX 37x
/ AI 9 HX to the gfx1150 row and drop the bogus HX 38x pattern (no such Strix
Halo SKU). Matches the runtime classifier in worker.py (890M/880M -> gfx1150;
8060S/8050S -> gfx1151). Prevents Strix Point hosts from getting the wrong ROCm
prebuilt/wheels.
- amd-smi opt-out (setup.ps1 + install.ps1): an explicit UNSLOTH_ENABLE_AMD_SMI=
0/false/no/off now wins over the HIP-SDK heuristic, so a host with a HIP SDK
binary but a broken runtime no longer gets the DiskPart/UAC prompt the opt-out
exists to avoid.
- amd-smi warning probes (install_python_stack.py): _has_rocm_gpu and
_detect_amd_gfx_codes now gate amd-smi behind _amd_smi_allowed() (and pass
_amd_smi_env()), closing the last unguarded amd-smi spawn on Windows.
- WSL ROCm bootstrap (install.sh): the "already-usable ROCm?" early return now
requires rocminfo to enumerate the real gfx1151 agent instead of the generic
_has_amd_rocm_gpu (whose broad gfx[1-9][0-9] match accepts a fallback
"gfx11-generic" ISA), so a Strix Halo box missing the ROCDXG bridge is no longer
skipped. The shared helper is untouched (no gfx90a regression).
- install_rocm_wsl_strixhalo.sh:
* Quote-safe Windows SDK discovery: the old for-in-$(ls -d "...Program Files
(x86)/...") word-split on the space and never matched; use find + read loop.
* Add `make` to apt prereqs (cmake only recommends it; minimal images lacked it
and the librocdxg `make -j` build failed).
* Verification requires gfx1151 exactly (not gfx1[0-9]) so a generic ISA or an
unrelated RDNA GPU can't pass while the real GPU is absent.
Reviewed but NOT changed:
- "Forward inferred ROCm arch without HasROCm" (setup.ps1): already correct --
--rocm-gfx is forwarded under `if ($script:ROCmGfxArch)`, not `if ($HasROCm)`.
- "Route inferred arch into install.ps1 torch path": not a bug -- install.ps1
installs CPU torch as a base by design and setup.ps1 swaps in the ROCm wheel for
the inferred arch (gate `($HasROCm -or $ROCmGfxArch) -and cpu`); verified live
the native install ends on torch 2.11.0+rocm7.13.0.
- "$p null guard after Start-Process" (install.ps1/setup.ps1): redundant -- the
amd-smi runner uses [Process]::Start wrapped in try/catch, so a null process
already returns "" with LASTEXITCODE=1 (no uncaught exception).
- "ls -> find for /usr/lib/wsl/lib" (gemini): stale -- that heuristic was removed;
only a comment about it remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer(rocm-wsl): auto-install the Windows 11 SDK via winget (fewer manual steps)
librocdxg's build needs the Windows SDK 'shared' headers on the Windows host.
Previously the helper just die()d with "install the Windows 11 SDK and re-run" if
they were missing -- a manual prerequisite that broke the otherwise-seamless
`curl ... install.sh | sh` one-liner on Strix Halo.
Now, when the headers aren't found, the helper installs the Windows 11 SDK on the
Windows host from inside WSL via winget (powershell.exe interop), then
re-discovers them. The SDK installer elevates -> ONE UAC prompt on the Windows
desktop; the headers appear under /mnt/c immediately (drvfs is live, no reboot).
The user already consented to the ROCm-on-WSL setup, so no extra prompt is added
beyond the OS UAC gate.
- New _find_win_sdk (space-safe find of the newest installed SDK 'shared' dir)
and _install_windows_sdk_via_winget helpers.
- winget IDs tried newest-stable first: Microsoft.WindowsSDK.10.0.26100, then
.22621. The presence of the headers (re-check) is the source of truth, not
winget's exit code. </dev/null so winget never consumes a piped `curl|sh` stdin.
- Best-effort + non-fatal: interop-off / no-winget / declined-UAC all fall
through to the existing clear manual-install die(). Opt out with
UNSLOTH_SKIP_WIN_SDK_INSTALL=1.
Removes the last avoidable manual step from the WSL Strix Halo path; only the AMD
Adrenalin driver (AMD referrer-gates the download) remains manual. Verified
_find_win_sdk resolves the spaced "Program Files (x86)" path; bash -n clean; all
winget flags validated against `winget install --help`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* installer(amd): gate install-time amd-smi probe to fix DiskPart UAC prompt
install_python_stack.py's Windows "AMD GPU detected but ROCm torch missing"
warning probe ran `amd-smi list` whenever amd-smi was on PATH -- and amd-smi
ships in C:\Windows\System32 with the AMD Adrenalin driver -- without the
_amd_smi_allowed() gate that every other amd-smi call site in the file uses.
On Adrenalin-only hosts (no HIP SDK) amd-smi elevates a child at runtime and
pops a UAC/DiskPart prompt that __COMPAT_LAYER=RunAsInvoker cannot suppress
(amd-smi's manifest is asInvoker). The probe also ran before the
ROCm-torch-installed check, so it fired on every Windows AMD install.
Gate it behind _amd_smi_allowed() and pass _amd_smi_env(), matching
_has_rocm_gpu()/_detect_amd_gfx_codes(). When skipped, the only loss is the
best-effort "AMD GPU detected" note on HIP-SDK-less hosts.
Adds a per-function AST regression test asserting every function in
install_python_stack.py that names the amd-smi command and spawns a subprocess
also references _amd_smi_allowed() (flags the pre-fix code; passes after).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* studio(cli): fix `unsloth studio stop` crashing on Windows
`stop` used the POSIX `os.kill(pid, 0)` liveness probe, but on Windows
CPython raises OSError (WinError 87, "The parameter is incorrect") for
*every* pid -- alive or dead. `stop` only catches ProcessLookupError /
PermissionError, so the OSError propagated and the command crashed with
a traceback before ever reaching its (correct) `taskkill /F` path.
Add a cross-platform `_pid_alive(pid)` helper (tasklist on Windows,
signal-0 elsewhere) and use it for both the pre-check and the post-kill
wait loop. The actual kill path is unchanged.
Verified on Windows (Python 3.13): os.kill(pid,0) raises WinError 87 for
both a live and a dead pid; `_pid_alive` returns True/False correctly and
the full stop() flow (alive -> taskkill -> dead -> "stopped") passes
end-to-end against a throwaway process.
Adds tests/studio/test_cli_studio_stop_windows.py (AST guard against a
bare os.kill(pid,0) liveness probe + mock-only _pid_alive behaviour for
the win32 tasklist branch and the POSIX signal-0 branch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* installer(amd): fix install.sh name->arch table misrouting Strix Point to gfx1151
The bash name->arch inference table in install.sh placed Strix Point
identifiers (Radeon 890M, "Ryzen AI 9 HX 370/375", "AI 9 HX") in the
gfx1151 (Strix Halo) row, diverging from the install.ps1 / setup.ps1
PowerShell tables which correctly map them to gfx1150. It also carried a
stray "HX 38" token absent from the PowerShell source-of-truth.
Align install.sh with the PowerShell tables:
gfx1151 row: 8060S|8050S|8040S|Strix Halo|Ryzen AI Max|AI Max
gfx1150 row: 890M|880M|860M|840M|Strix Point|Krackan|HX 37|AI 9 HX|...
Impact is low (the bash table only feeds the display label _gpu_disp_gfx
and the "set UNSLOTH_ROCM_GFX_ARCH=..." hint; wheel selection is driven
by the detected ROCm version, not this name string) but a Strix Point
user would otherwise see/copy the wrong gfx arch.
Add a parity test (test_install_sh_name_arch_agrees_with_ps_for_strix_and_non_amd)
that parses install.sh's case table and asserts Strix Halo->gfx1151,
Strix Point->gfx1150, RX 7700S->gfx1102, and NVIDIA/Intel->no match,
cross-checking against install.ps1 (the previous parity test only
compared install.ps1 <-> setup.ps1, missing install.sh).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* setup.ps1: keep prebuilt-llama ownership guard within the test's block window
The AMD additions to the prebuilt-llama.cpp block (the windows-hip vs
windows-cpu existing-install kind validation) pushed the
install_llama_prebuilt.py invocation to ~1999 chars after the
"installing prebuilt llama.cpp bundle (preferred path)" anchor, right at
the edge of the 2000-char window that
test_setup_ps1_prebuilt_llama_cpp_has_ownership_guard slices -- so the
helper string was truncated and the test failed with "substring not
found" (CI: Repo tests (CPU)).
The ownership-guard invariant (Assert-StudioOwnedOrAbsent precedes the
install_llama_prebuilt.py call) was already satisfied; only the proximity
to the anchor regressed. Move the "installing prebuilt..." substep to
immediately before the install (after the existing-install pre-cleanup),
which also reads better (validate/clean existing -> then "installing"),
shrinking anchor->helper from 1999 to 413 chars. Behaviour is unchanged
(console message ordering only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* install.sh: auto-run Strix Halo ROCm-on-WSL setup by default
`curl -fsSL https://unsloth.ai/install.sh | sh` should make a Strix Halo
(gfx1151) GPU usable inside WSL with no extra commands. Previously the
ROCm-on-WSL bootstrap was opt-in: it required UNSLOTH_ROCM_WSL_AUTO=1 or an
interactive [Y/n] at a TTY, and silently skipped under a pipe (no /dev/tty),
so the piped one-liner never set the GPU up automatically.
Flip it to auto-by-default for the single narrow case the existing guards
allow (WSL + Strix Halo + /dev/dxg + no usable ROCm yet) -- exactly the GPU
setup the user ran the installer for. Opt out with
UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The Tauri desktop app keeps its own consent UI
(only auto-runs when it passes UNSLOTH_ROCM_WSL_AUTO=1). All hardware/OS
guards are unchanged, so non-Strix / non-WSL / NVIDIA / native-Linux / macOS /
CPU paths are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* PR comments: condense to be succinct (comments/docstrings only)
Shorten the verbose explanatory comments and docstrings this PR added across
the installer, scripts, backend shims, CLI, and tests -- tighter, fewer lines,
while preserving every non-obvious "why" (os.kill WinError 87, amd-smi
RunAsInvoker/UAC, /dev/dxg + librocdxg gating, the ROCm-on-WSL bootstrap guard
chain, ownership guards, etc.). No executable code, string literals, messages,
or behavior changed.
Verified comments-only: docstring-normalized AST equality (Python, 9 files),
non-comment token equality (PowerShell, 3 files), comment-stripped diff +
sh -n / bash -n (shell, 3 files). Behavior re-confirmed: get_torch_index_url +
gfx name->arch table 44/44 under dash & bash; rocm_support / pr5940_followups /
cli_studio_stop tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Installer: address PR review (amd-smi opt-out, pipefail, multi-distro, non-root)
Fixes valid findings from the Codex/Gemini PR review:
- install.ps1 / setup.ps1: gate the `amd-smi version` ROCm-version fallback with
$amdSmiAllowed so UNSLOTH_ENABLE_AMD_SMI=0 opt-out is honored (the device
probe was gated but this fallback wasn't), avoiding the DiskPart/UAC prompt.
- install_rocm_wsl_strixhalo.sh: make the post-verification rocminfo summary
best-effort (|| true) so head's early pipe-close under `set -o pipefail` can't
fail the bootstrap after gfx1151 was already enumerated; pin the Windows SDK
`winget install` to --source winget (matches the msstore-cert fix rationale).
- install.ps1: python.org fallback installs the py launcher per-user
(InstallLauncherAllUsers=0, avoids admin), and derives the fallback full
version from the requested minor so a non-default UNSLOTH_PYTHON (e.g. 3.12)
isn't silently replaced with 3.13 when the listing is unreachable.
- install.sh: recreate /etc/profile.d/unsloth-rocm-wsl.sh via `sudo tee` for a
non-root reinstall (a plain redirect failed silently, dropping the ROCm env).
- uninstall.sh: scope WSL Windows-side shortcut removal to the current
WSL_DISTRO_NAME (per-distro name or -d "<distro>" arg) so uninstalling one
distro no longer deletes other distros' launchers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Studio ROCm Windows: fix field-reported issues from Strix Halo testers
Four fixes from PR #5940 field reports (Win11 native, gfx1151):
1. bitsandbytes arch-probe spam: bnb's get_rocm_gpu_arch() runs
hipinfo.exe via subprocess PATH at import; the AMD torch wheel ships
hipInfo.exe in the venv Scripts dir, which is only on PATH for
activated venvs. Every bnb import logged "Could not detect ROCm GPU
architecture: [WinError 2]" ERROR + WARNING (even with the HIP SDK
installed, whose bin dir is not on PATH either). Prepend the Scripts
dir to PATH before bnb imports in main.py, worker.py, and
install_python_stack.py, gated on the file existing (only AMD wheels
ship it). Verified on gfx1151: ROCM_GPU_ARCH now resolves to gfx1151
with zero errors.
2. OOM-guard double-tax on native Windows unified APUs: mem_get_info's
total is the WDDM budget the driver grants HIP (BIOS carve + ~half
of remaining RAM) -- the OS share is already outside it. The 0.80
unified cap on top denied loads that fit (field report: 48.49 GiB
budget -> "38.79 GiB allowed" OOM for a 47.29 GiB load with 48.08
free). Use 1.0 on win32 unified; Linux keeps 0.80, discrete 0.90.
3. "Missing VRAM" confusion: log the WDDM budget vs physical RAM with
the fix (BIOS UMA frame buffer / AMD Software Variable Graphics
Memory) when the grant is under 75% of RAM, so a 48 GiB cap on a
96 GiB box reads as policy, not a Studio bug.
4. llama-server fit-step crash (Qwen3.6-27B-MTP + mmproj, lemonade
gfx1151): --fit defaults to 'on' upstream, so the fit step runs even
when Studio already placed the model via -ngl -1, and aborts in
ggml-cuda.cu on some ROCm hosts. Retry the spawn once with --fit off
when the server crashes during startup and Studio's own VRAM math
had placed the model (never when use_fit or an explicit fit flag was
passed). Also keep the TAIL of crash output in the error log (the
diagnostic line prints last; head-truncation cut exactly that) and
reference the full on-disk log.
Verified live on Radeon 8060S: bnb import clean, Qwen3.5-4B-MTP loads
and generates through the new spawn loop, stub-crash retry appends
--fit off and recovers, fraction probes confirm WDDM overcommit and
sub-1.0-only enforcement on current AMD wheels.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio ROCm Windows: GPU-name fallbacks so nothing depends on amd-smi
amd-smi does not reliably exist on Windows: the HIP SDK never ships a
CLI, inbox Windows Update drivers do not, and only some full Adrenalin
packages drop amd-smi.exe into System32 (field report: fresh Win11 +
Adrenalin + HIP SDK, still no amd-smi anywhere). Make every consumer
work without it:
- install_python_stack._detect_windows_gfx_arch: two new probes after
hipinfo/amd-smi -- (2b) the venv Scripts hipInfo.exe shipped by AMD
torch wheels (drives `studio update` on driver-only hosts), and (4) a
last-resort GPU marketing-name -> gfx table via WMI
(Win32_VideoController), mirroring setup.ps1's $nameArchTable so a
standalone repair resolves the arch with zero AMD tooling installed.
- install_llama_prebuilt._resolve_exe: also probe the venv Scripts dir
so a standalone rerun finds hipInfo.exe without HIP_PATH.
- hardware/amd.py _run_amd_smi: which() guard before spawning --
absence now disables the poller in one step instead of burning the
3-strike circuit breaker on FileNotFoundError; corrected the stale
comment claiming Adrenalin ships amd-smi.
Simulated against the real detection functions on gfx1151: amd-smi
absent, present-but-crashing (exit 1), present-but-hanging (60s sleep
vs 5-10s probe timeouts), and hard opt-out -- all resolve gfx1151, no
exceptions, bounded time. Full adversarial install (broken amd-smi
stub first on PATH + UNSLOTH_ENABLE_AMD_SMI=1, fresh uninstall first):
exit 0, name-table arch inference, lemonade gfx1151 b1292 prebuilt,
torch 2.11.0+rocm7.13.0 cuda_avail=True on the 8060S, Studio boots
healthy and stops cleanly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-attempt llama-server log names + amd-smi test portability
Found by cross-platform simulation of the --fit off retry (Windows +
Linux sandboxes, real load_model with stub servers):
- llama-server log filename now carries the spawn-attempt index. The
retry can respawn within the same epoch second; reusing the name
opened the same file with "w" and truncated the crash log the retry
warning had just pointed the user at (proven with a frozen
time.time: one file, crash evidence gone; with the suffix both
attempts keep their logs). Regression-pinned in
test_llama_cpp_wait_for_health.py.
- test_amd_primary_gpu_with_mock now mocks shutil.which alongside
subprocess.run: the amd-smi absence guard which()-checks before
spawning, so on hosts without a real amd-smi (Linux CI, driver-only
Windows) the subprocess mock was never reached and the test failed.
Surfaced by running the suite in a clean Linux sandbox.
Simulation coverage on both OSes: 67-case platform/edge matrix
(real shipped code blocks under win32/linux/darwin spoofs: OOM-guard
fractions + VGM-hint boundary, bnb PATH-prepend gates, retry
eligibility incl. equals-forms and decoy tokens, GPU-name table
adversarial set, WMI fallback without powershell, monitor absence
semantics), 6-scenario live retry matrix (crash-once/crash-always/
exit-zero/explicit-fit/hang/log-collision) against real llama-server
spawns on Windows and WSL (GPU success legs on the 8060S), and a
3-engine browser matrix (chromium/firefox/webkit) driving the live
backend's health + authed /v1 chat completion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: classify unified-memory via props.is_integrated first
Align the ROCm OOM-guard classifier with PR #5988's UMA gate: consult
hipDeviceProp_t.integrated (props.is_integrated) before the hardcoded
arch set. Strictly additive -- truthy upgrades to unified; 0/absent
falls through to the existing gfx1150/gfx1151 + device-name logic, so
wheels that omit or zero the field cannot downgrade the known APU set.
Extends correct unified-cap treatment to APUs outside that set (e.g.
gfx1103 Phoenix iGPUs) and keeps Studio's two unified-memory consumers
on one driver signal. Verified live on gfx1151 (is_integrated == 1 on
the AMD Windows wheel -> ('gfx1151', True) via the new path).
* AMD detection: probe rocminfo with HSA_ENABLE_DXG_DETECTION and sync setup.sh gfx table
Fleet validation on a Strix Halo WSL2 box showed the system rocminfo
(HSA 1.18, ROCm 7.2.1) only enumerates the GPU over /dev/dxg when
HSA_ENABLE_DXG_DETECTION=1, and that rocminfo can sit at /opt/rocm/bin
off PATH outside login shells. Detection probes that miss either of
these report no GPU on a working ROCDXG host and select the CPU build
even though the lemonade bundle offloads fine (95.7 tok/s measured vs
64.5 CPU on the same laptop). Seed the env (a no-op on bare metal) and
the PATH fallback in install.sh, studio/setup.sh, and the installer's
Linux rocm probe, mirroring what main.py/worker.py already do for the
runtime.
Also sync studio/setup.sh's name->gfx table with install.sh: 890M and
the HX 37/AI 9 HX SKUs are Strix Point (gfx1150, not gfx1151), RX 7700S
must match gfx1102 before the gfx1100 row, and the RDNA2/workstation
rows were missing. New parity test pins the two bash tables together so
they cannot drift again.
* Studio: persist server session logs + native-crash stacks to disk
Field report (Strix Halo, 96 GB UMA carve, WSL and native Windows):
"the studio just terminates without a warning". A native crash in the
GPU runtime kills the process with no Python traceback, and a desktop-
shortcut console closes before anything can be read. The server only
ever logged to the console, so there was nothing to send back.
run_server now tees stdout/stderr to
~/.unsloth/studio/logs/server/server-<ts>-pid<n>.log (console behavior
unchanged; file copy is best-effort), arms faulthandler at the same
file so access violations / SIGSEGV leave a stack trace on disk, and
exports PYTHONFAULTHANDLER=1 so training workers inherit crash dumps
on their captured stderr. Armed before `from main import app` so even
import-time failures leave evidence. Keeps the newest 20 session logs;
opt out with UNSLOTH_STUDIO_NO_FILE_LOG=1. Prints "Session log: <path>"
at startup so users know what to attach.
Verified on this box: a forced real segfault (faulthandler._sigsegv)
leaves the full session output plus "Fatal Python error: Segmentation
fault" and the thread stack in the file while the console shows
nothing; a normal server boot captures the startup banner and serves
health as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* AMD probe: honor a pre-set HSA_ENABLE_DXG_DETECTION value
Match the shell helpers, which use the parameter-default form: a user
who exports HSA_ENABLE_DXG_DETECTION=0 to deliberately hide the GPU
from DXG detection should not have the probe override it.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController.
* Studio: fix OpenAI- and Anthropic-compatible API spec compliance
* Studio: fix API spec-compliance gaps on passthrough and streaming paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry context_length_exceeded through the OpenAI passthrough error path
* Studio: count tool-schema tokens in the Anthropic server-tool stream, and small stream-handling guards
* Studio: guard message_delta usage against None and normalize developer role before proxying
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor max_completion_tokens on the external-provider proxy path
* Studio: forward llama-server cached_tokens into OpenAI prompt_tokens_details
* Studio: sanitize messages in count_tokens to match the /v1/messages prompt
* Studio: report max_tokens for truncated tool calls and guard null usage in metadata events
* Studio: drop the request-id middleware (headers aren't declared in either spec)
* Studio: include the required request_id field in Anthropic error bodies
* Studio: honor max_completion_tokens on the audio (TTS / audio-input) paths
* Studio: add the _effective_max_tokens helper and route all max-token sites through it
* Studio: align API compatibility edge cases
* Studio: clarify multi-choice chat support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: clarify logprobs chat support
* Studio: opt the local chat UI into the streaming usage chunk so the context bar and tok/s repopulate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward seed to llama-server, and fix Anthropic server-tool stop_reason, tool_result id correlation, and parallel-tool execution cap
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align OpenAI chat completion spec edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align backend API compatibility tests
* Studio: honor tool caps and internal stream usage
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: coerce nullable stream usage counts
* Studio: preserve system prompts with developer messages
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Unify the Studio local tool-call loop (GGUF + safetensors) behind a shared ToolLoopController: ordered preface-then-tool-card rendering, duplicate-call de-looping with a forced final answer, XML-leak containment, and a parser fix that accepts closed <function=...> calls followed by trailing prose. Includes backend tests for the controller, strict parser, and GGUF route cursor reset.
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Daniel Han