Detect a DiffusionGemma GGUF (general.architecture=diffusion-gemma) and serve
it through the diffusion runner instead of llama-server: launch the OpenAI-compat
shim (unsloth_zoo.diffusion_studio.shim, or UNSLOTH_DG_SHIM) driving the on-device
visual decoder, resolving the visual-server binary from DG_VISUAL_BIN or next to
llama-server. Surface is_diffusion to the UI and auto-render the per-step denoising
canvas artifact for DiffusionGemma (no toggle). install_llama_prebuilt + setup.sh/ps1
place the visual-server binary in the install tree best-effort. Other models unchanged.
transformers >= 5.x makes _is_package_available always return a
(exists, version) tuple, which is truthy even when the package is
absent. The flash_attn and vLLM availability checks treated the result
as a bool, so they always entered the "package present" branch:
- flash_attn (CUDA + HIP): the inner import raises when flash-attn is
not installed, printing a false "Flash Attention 2 installation seems
to be broken" warning before falling back to xformers.
- is_vLLM_available(): always reported vLLM as installed.
Add a small _package_available() helper that normalises the result to a
bool (handling both the new tuple and the legacy bool return), and route
the three call sites through it.
Fixes#6155
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(rocm): stop overwriting ROCR_VISIBLE_DEVICES in apply_gpu_ids
ROCR_VISIBLE_DEVICES uses HSA agent-level indexing, not physical GPU
indices. Setting it to a bare integer breaks multi-GPU ROCm systems
where the parent already set ROCR_VISIBLE_DEVICES=0,1: narrowing to
1 causes torch.cuda.is_available() to return False in the training
worker, producing a misleading 'no HIP accelerator' error even on a
correctly configured ROCm host.
HIP_VISIBLE_DEVICES is sufficient for GPU selection on ROCm.
Leave ROCR_VISIBLE_DEVICES inherited from the parent environment.
* test(rocm): update apply_gpu_ids test to assert ROCR_VISIBLE_DEVICES is not overwritten
• fix: handle empty responses tool output
Normalize empty Responses `function_call_output.output` values before converting them into Chat Completions `role="tool"` messages. Empty strings, whitespace-only strings, and empty arrays now use the existing no-output sentinel, while non-empty text and content arrays are preserved.
Add regression coverage for empty tool outputs, image payloads outside `output`, content-array serialization, validator round trips, and preserving non-empty text.
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
When pinning GPUs for the llama-server child, the ROCm path set both
HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES to the same physical
indices. These masks filter at different layers and stack:
ROCR_VISIBLE_DEVICES reduces the visible set at the HSA/ROCr layer and
re-indexes from 0, then HIP_VISIBLE_DEVICES indexes into that reduced
set. _select_gpus ranks by free VRAM and picks the most-free card, so a
single non-zero pin (e.g. "1") becomes out of range at the HIP layer,
HIP enumerates 0 devices, and the model silently runs on CPU
("ggml_cuda_init: failed to initialize ROCm: no ROCm-capable device is
detected").
Set only HIP_VISIBLE_DEVICES (which narrows correctly on its own) and
clear any inherited ROCR mask so it can't double up.
Verified on a 2x Radeon AI PRO R9700 (gfx1201) host, ROCm 7.1.1: the
same selected=[1] load that fell back to CPU (~7.7 tok/s) now runs on
the GPU (~78 tok/s).
Fixes#6175
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* studio: import MCP servers from a config file
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* import config' on the add-server form
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: defensively handle MCP config imports
* fix: address MCP import review follow-ups
* fix: preserve apostrophes in Windows MCP commands
* fix: preserve apostrophe-wrapped Windows MCP args
* fix: align Windows MCP parsing with list2cmdline
* fix: preserve explicit MCP remote transport intent
* fix: trim MCP remote URLs before transport checks
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: add `unsloth chat` CLI command
Interactive chat REPL on the shared Studio backend: trained-model picker
when no model is given, /think and /compare toggles (adapter toggle on
CUDA, side-by-side base-model load on MLX), markdown streaming, and
connect-if-running Studio server mode so models stay warm across
sessions and are shared with the UI.
* fix settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix error handling and compare base precision
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix chat CLI backend imports and GGUF drafter loading
* Hide split thinking tags in chat CLI streams
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* fix(studio): surface live step with null loss through the SSE progress stream
The metric histories skip non-finite steps, so during a NaN stretch the
SSE live loop and final complete event replayed the last finite
step/loss pair. Follow the live progress step when it is ahead of the
history tail and report its loss honestly (null until recovery).
Completes the NaN honesty fix for the SSE consumer flagged in review.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply live-step handling to inactive streams and clear the UI loss on null for PR #6206
Fresh /progress connections after a finished run took the inactive branch
which still replayed the last finite step and loss pair; apply the same
live-step correction there. On the frontend, applyProgress kept the stale
currentLoss when a payload advanced the step with a null loss; clear it so
the display shows -- until the loss recovers. Widen the runtime state type
to number | null, which the view layer already handles.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving
A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.
Changes:
- After llama-server health, read default_generation_settings.n_ctx from
/props and adopt it whenever it is below Studio's computed context, with
a warning. The load response, status route, UI value, and the passthrough
max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
the passthrough drops whole middle turn-groups (system prompt, first turn,
and recent turns kept; tool calls stay paired with their results), clips
oversized contents middle-out when group-dropping is not enough, clamps
max_tokens to the generation headroom, and retries. Default stays 'error'
with code=context_length_exceeded so clients running their own compaction
keep full control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)
Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
always passes --parallel N, which silently splits -c into per-slot windows
of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
--fit-ctx <requested> for explicit requests so fit offloads or fails into
the existing --fit off retry instead of silently shrinking the window.
Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix config.rope_scaling being dropped by the replaced rotary embedding (#2405)
On modern transformers, LlamaModel builds its rotary embedding from config
using unsloth's replacement LlamaRotaryEmbedding class, whose config path
computed vanilla inv_freq and ignored config.rope_scaling entirely. The
llama3/linear/longrope dispatch in patch_llama_rope_scaling rewrites
LlamaAttention.__init__, which no longer constructs rotary embeddings, so it
never fires; the model-level rotary is then copied onto every attention
layer. Result: Llama-3.1/3.2/3.3 ran with unscaled RoPE on the
FastLanguageModel path and collapsed into repetition loops past roughly 29K
tokens (PASS at 28867, FAIL at 31767 in needle retrieval). FastModel was
unaffected because vision.py keeps transformers' own rotary. qwen2, qwen3,
qwen3_moe, mistral and cohere assign the same base class, so any rope-scaled
config of those families was equally exposed.
The fix makes the base class config path compute inv_freq and
attention_scaling via transformers' ROPE_INIT_FUNCTIONS (covers llama3,
linear, dynamic, yarn, longrope), with an inline llama3 fallback reading
factors from config for older transformers, degrading to prior behavior on
any failure. attention_scaling is applied in _set_cos_sin_cache (1.0 default,
exact no-op for unscaled paths) and persists across extend_rope_embedding.
A type(self) guard prevents double-scaling via the legacy scaled subclasses.
Adds tests/utils/test_rope_scaling_drift.py (AST tripwire + behavioral
inv_freq/cos-cache/extension checks, validated to fail 4 of 5 on the unfixed
code) and wires it into the existing consolidated CI HARD GATE step.
Verified on GPU: 48K-token needle retrieval flips FAIL to PASS for
FastLanguageModel in bf16 and 4bit, 20K stays PASS, scaled inv_freq matches
transformers exactly, and the left-padded batch generation guard still gets
exact solo-vs-batched token matches.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: normalize object-style rope_scaling, vectorize llama3 fallback
config.rope_scaling can be a config object rather than a dict on newer
transformers; _rope_scaling_as_dict normalizes it (to_dict/dict/vars
fallbacks) before any .get() access, with a regression test using a
dataclass stand-in. The inline llama3 fallback now uses torch.where instead
of a per-frequency Python loop; verified bit-for-bit equal to transformers
ROPE_INIT_FUNCTIONS for factor 8 (Llama-3.1) and factor 32 (Llama-3.2).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: CPU-safe rope guard tests, normalized config for delegation
The rotary constructor builds per-device CUDA caches, so the behavioral tests
that instantiate it cannot run on GPU-less CI. Restructured into three layers:
the AST tripwire now also asserts the constructor stays wired to
_compute_config_rope_inv_freq; the CPU layer tests that pure helper directly
(llama3 dict, llama3 object, linear object, default type) with no
instantiation; the instantiation and cache tests are gated behind a real CUDA
probe (actual tensor allocation, so import-time CUDA spoofs cannot fool the
gate). Verified: 9 passed with GPU; 5 passed 4 skipped with CUDA hidden; 5
failed 4 skipped on the unfixed code in CPU mode.
Delegation to ROPE_INIT_FUNCTIONS now retries with a shallow config copy
carrying the normalized rope_scaling dict when the original was an object the
installed transformers cannot read; covered by a linear-object test, which has
no inline fallback and passes only through that retry path.
* Tighten comments in rope scaling fix and guard test
Comment and docstring reduction only; verified code-identical with
scripts/comment_tools.py check --strip-docstrings (AST signature match on
both Python files). All guard tests unchanged: 20 passed with GPU, 5 passed
4 skipped with CUDA hidden.
* Apply repo kwarg-spacing format
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Studio's frontend exposes a Resume action and submits requests with
resume_from_checkpoint set to a previous run's output_dir. The CUDA
training paths in worker.py read this field from config and pass it to
trainer.train() (see lines 2729-2787 and 3108-3229). The MLX path
_run_mlx_training did neither: it never read config['resume_from_checkpoint']
and called trainer.train() with no args. The MLX trainer also did not
accept the kwarg, so even threading it through would have been a no-op.
With this PR + the unsloth-zoo companion PR adding the trainer-side
support (saves optimizer_state + trainer_state, accepts and applies
resume_from_checkpoint in MLXTrainer.train()), MLX Resume now works
end-to-end. Verified on M2 16GB with Qwen3-0.6B + unsloth/LaTeX_OCR:
loss at every post-resume step matches a fresh run bit for bit
(2.168627977371216 == 2.168627977371216 at step 6, etc).
Two lines: read the field near the other config.get() extractions in
_run_mlx_training, pass it as a kwarg at the trainer.train() call site.
Companion PR: unslothai/unsloth-zoo#751
When training produced a NaN or Inf loss event, the handler filtered the
value to None but never updated progress.loss — clients kept seeing the
last finite value as if everything were fine.
Now: on non-finite loss, clear progress.loss to None and log a one-shot
warning. Training continues (no phase=error, no _should_stop), matching
the expected behavior for a non-fatal numerical event.
Test: tests/test_training_nan_loss_handling.py with 6 cases covering
finite, NaN, +/-Inf, idempotency of the one-shot warning, and recovery
when a finite step follows a non-finite one.
* Studio: auto Cloudflare tunnel for 0.0.0.0 launches
Binding Studio to 0.0.0.0 for remote access often leaves the raw
http://<ip>:<port> URL unreachable (https-vs-http, blocked high ports,
closed cloud security groups). On a wildcard bind, auto-start a free
cloudflared quick tunnel and show its https://*.trycloudflare.com URL in
the startup banner:
Secure link access via Cloudflare: https://<random>.trycloudflare.com
- new studio/backend/cloudflare_tunnel.py: find or download+cache the
cloudflared binary (per-OS/arch GitHub release, safe .tgz extract),
start the tunnel, parse the URL, tear it down. Stdlib only; best-effort
and non-fatal throughout (a missing binary or offline box never blocks
or slows startup).
- run_server starts the tunnel for 0.0.0.0 only (skips loopback, api-only
and Colab), prints the line in the banner, and _graceful_shutdown stops
the child so it never orphans.
- --cloudflare/--no-cloudflare flag (default on) on `unsloth studio` and
`unsloth studio run`, forwarded through the re-exec into run_server.
- tests for the helper, the CLI flag forwarding, and the run.py defaults.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio cloudflare: send a User-Agent on the cloudflared download
GitHub's CDN can 403 the default Python-urllib User-Agent on release asset
downloads. Set an explicit UA and pin it with a test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio cloudflare: address review (opt-out for subcommands, tunnel teardown)
- reject --no-cloudflare placed before a subcommand (it would not reach the
subcommand), mirroring the --parallel guard
- register the tunnel before waiting for its URL so a shutdown during the wait
stops cloudflared instead of orphaning it
- tear the server + children down if `unsloth studio run` startup aborts
(health timeout, model-load error, Ctrl+C) before the wait loop
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Strip trailing whitespace in construct_chat_template (fixes#992)
construct_chat_template() only lstrip()s the template, so a template ending
in a newline (e.g. the default Llama-3 template + a trailing '\n') keeps the
trailing whitespace. That breaks the rfind-based repeated-example detection
and, on current main, also the regex fallback, which then raises
"Could not recover a two-example structure from chat_template".
Surrounding whitespace is not significant to the template here (the left side
is already stripped), so strip() both ends. Verified that the valid no-newline
template still parses via the primary path and that the meaningful trailing
token (e.g. <|eot_id|>) is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix trailing whitespace handling via parse retry for PR #6008
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix(studio): reuse venv Python in setup instead of re-probing system
* Reuse venv Python for studio setup
Pass the venv interpreter from install.ps1 to studio/setup.ps1 via UNSLOTH_SETUP_PYTHON and prefer it over probing the system. Added Resolve-ReusedSetupPython to accept the handed-off path (or derive the venv python when setup runs standalone), validate it (Python 3.11–3.13 and non-conda), and inject its Scripts dir onto PATH. When a reused interpreter is accepted, py.exe enumeration and further system probing are skipped. install.ps1 also sets the env var before running setup and removes it on cleanup to avoid leaving state behind. This prevents setup from being tripped by unsupported Python 3.14 or Windows Store stubs on PATH.
* Harden setup Python detection for PR #6033: py -All, shared conda check, bare ~ guard
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* center the search dialog and change the wrong borders
* fix the mistake of 1 to l
* Fix/adjust search dialog radius for PR #6184
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Stop HF 429 rate limits from sinking the llama.cpp prebuilt path in Studio CI
The Windows Studio API smoke job failed when anonymous huggingface.co
fetches of the tiny GGUF validation model (stories260K.gguf) hit HTTP 429
on the shared runner IP. The installer correctly refused the unvalidated
prebuilt and fell back to a source build, which the prebuilt assert then
flags. Three layers fix this:
1. Installer: auth_headers sends HF_TOKEN (or HUGGING_FACE_HUB_TOKEN) to
huggingface.co hosts, mirroring the existing GH_TOKEN handling for the
GitHub API rate limit. A redirect handler strips Authorization when a
download is redirected off-host (CDN signed URLs reject foreign auth;
urllib forwards headers on redirect, unlike requests/huggingface_hub).
2. Workflows: the HF_HOME prime steps also prefetch the validation model
so the install's hf_hub_download resolves from the local cache even
when the Hub is rate limiting; cache keys bumped v1 to v2 to repopulate.
This also covers fork PRs, which cannot see secrets.
3. Workflows: every Install Studio / update step that already passes
GH_TOKEN now also passes HF_TOKEN, so both the huggingface_hub path and
the direct URL fallback are authenticated.
Tests: tests/studio/install/test_hf_auth.py covers token-to-host routing,
the cross-host redirect strip, and the download_bytes wiring (offline).
Verified live: authenticated download of the validation model through the
new opener (CDN redirect exercised, pinned sha matches) and an offline
hf_hub_download cache hit against an HF_HOME primed by the new step.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Require a found ROCm DLL before forcing BNB_ROCM_VERSION in Studio paths
main.py previously set BNB_ROCM_VERSION=72 whenever HIP_PATH or ROCM_PATH
was set, and the training worker fell back to a blind 72 when DLL
detection found nothing. On a Windows machine with the AMD HIP SDK
installed but CUDA or CPU torch, that forces a ROCm backend onto a
non-ROCm bitsandbytes wheel, which raises at import. Both paths now only
write the override when a libbitsandbytes_rocm DLL actually exists (or a
seeded value is already present), matching the strict gates in
unsloth/import_fixes.py.
Also removes four redundant local import shutil statements in
unsloth/save.py that shadow the module-level import, the same pattern
that caused the UnboundLocalError fixed in #6149.
* Worker: gate the BNB override on a found ROCm DLL, preserving seeded marker
Review follow-ups: track _found_rocm_bnb in the worker like main.py so a
ROCm DLL with an unparsable name still gets the seeded or 72 fallback,
and skip the env write entirely when no DLL exists so a seeded value
keeps its sitecustomize marker and stays redetectable by later import
fixes.
* Fix bitsandbytes ROCm GPU arch and warp size detection on Windows
bitsandbytes resolves the ROCm GPU architecture (and warp size on
0.49.x) by shelling out to rocminfo / hipinfo.exe via PATH at import
time. On Windows neither tool is normally on PATH (AMD torch wheels
ship hipInfo.exe into the venv Scripts dir, only on PATH while
activated), so every `import bitsandbytes` logs an ERROR and WARNING,
ROCM_GPU_ARCH degrades to unknown, and the 0.49.x warp size defaults
to 64, which is wrong on RDNA (wave 32) and silently disables
pre-quantized 4-bit models via ALLOW_PREQUANTIZED_MODELS.
Install a one-shot MetaPathFinder before unsloth_zoo is imported (the
first bitsandbytes import on ROCm) that swaps get_rocm_gpu_arch and
get_rocm_warpsize for torch-device-properties-first implementations
right after bitsandbytes.cuda_specs executes, before cextension reads
them. Falls back to running hipInfo.exe by absolute path (venv
Scripts, conda Scripts, HIP SDK / AMD installer dirs). Repairs the
constants in place when bitsandbytes was imported first. Strict no-op
on non-Windows, non-ROCm builds, missing bitsandbytes, and versions
that fix this upstream. Opt out with UNSLOTH_DISABLE_BNB_ROCM_FIX=1.
Proposed upstream in bitsandbytes-foundation/bitsandbytes#1969;
shipped here so all bitsandbytes versions are covered. Verified on
gfx1151 Strix Halo, Windows 11, torch 2.11.0+rocm7.13.0 against
bitsandbytes main, 0.49.2, and a torch-props-fixed variant.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the bitsandbytes ROCm detection fix
Comment and docstring pass only. AST comparison with docstrings
stripped confirms every definition is identical to the version the
12 scenario suite ran against, and the suite plus the drift test
pass unchanged on the edited files.
* Keep the bitsandbytes cuda_specs finder installed for reload support
Simulation testing caught a regression in the one-shot design:
importlib.reload(bitsandbytes.cuda_specs) re-resolves the spec through
sys.meta_path, so with the finder already removed the reload reinstalled
the unpatched upstream detector and the Windows ROCm noise returned.
Keep the finder on sys.meta_path permanently, matching the lifecycle of
the existing causal_conv1d and vllm import blockers. The finder matches
a single module name and patching stays idempotent via the sentinel
flags, so repeat hits are no-ops.
Validated on gfx1151 Windows 11: 22 simulation scenarios (conda and
embedded layouts, Program Files scan ordering, paths with spaces and
unicode, hanging probe timeout, lru-wrapped and C-function helper
shapes, reload, failed-import retry, threads, spawn, dormant finder,
Studio PATH coexistence, early fix-block ordering, bnb 0.45.5 / 0.47.0
/ 0.49.2 / main / upstream-fixed) plus the original 12 scenario suite,
CPU-torch and stale-HIP_PATH sandboxes, Python 3.10 to 3.13 gates, and
a WSL Linux leg proving byte-identical Linux behavior with and without
the fix, with and without rocminfo on PATH.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
* fix(studio): prevent UI freeze when switching tabs from heavy pages
Change AnimatePresence mode from "wait" to "popLayout" to fix issue
where switching tabs from Export (or other heavy pages) would cause
the URL to update but the UI to freeze.
With mode="wait", the exit animation must complete before the new
component mounts. If the exiting page has expensive computations,
this blocks the UI. mode="popLayout" allows the new route to mount
immediately while the old one animates out.
Fixes#5850
* fix: add relative positioning for popLayout mode
AnimatePresence mode='popLayout' applies position: absolute to the
exiting element, so the parent container needs position: relative
to prevent layout jumps during transitions.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
The export route's load_checkpoint waits for the training subprocess to
exit by calling time.sleep(0.5) in a loop (up to 30s) inside an async
function. time.sleep blocks the whole event loop, so every other request
to the server stalls for that duration. Use await asyncio.sleep(0.5),
matching the async pattern already used elsewhere in this file
(asyncio.to_thread, await asyncio.sleep).
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
The kwarg-dispatch in unsloth_base_fast_generate recognized input_ids,
input, input_features and the misspelled input_embeds, but not HF's
canonical inputs_embeds. So generate(inputs_embeds=...) fell through to the
'first kwarg' fallback, which picks whatever kwarg happens to come first
(e.g. attention_mask) and uses it as input_ids -- giving the wrong tensor /
batch size, or the KeyError reported in #3082 on older versions.
Add an explicit inputs_embeds branch so embedding inputs (e.g. multimodal
audio+text) are routed correctly regardless of kwarg order.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
* fix(studio): fall back to copy when os.replace is blocked during install activation
On Windows ARM64 the antivirus scanner can transiently hold a freshly
extracted DLL open while MoveFileEx runs, so activating the staged
llama.cpp prebuilt fails with [WinError 5] Access is denied. Attempt
os.replace first, then fall back to a file-by-file copytree which
bypasses the rename.
* address review: keep os.replace for rollback, scope copy-fallback to staging
The copy + rmtree fallback could silently corrupt a live install if the
existing directory is busy. Restrict it to freshly extracted staging
trees (renamed activate_staged_dir) and keep strict os.replace for the
rollback move so a busy active install raises immediately.
* fix(studio): scope copy-fallback to busy-lock errors, log it, and add tests
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
* Studio: enable MTP for sub-3B Gemma separate-drafter GGUFs
The sub-3B auto-drop to ngram-mod was tuned for an embedded draft head
(Qwen), whose per-token cost regresses below 3B. Gemma ships the head as a
separate root mtp-*.gguf drafter, a tiny standalone model that is cheap
enough to win below 3B: B200 Q4_K_XL bench, draft-mtp n=2 vs spec-off,
gemma-4-E2B (2B) = 1.21x (accept ~0.65) while ngram-mod is 1.00x.
Exempt a separate drafter from the sub-3B gate everywhere the threshold is
applied: the resolver (_mtp_too_small), the auto-fit VRAM reserve, the
drafter auto-download decision, and the reload-skip mirror via a
has_separate_drafter flag on _auto_mode_drops_mtp. Embedded sub-3B heads
(Qwen) still drop to ngram-mod. A drafter the binary cannot build (older
prebuilt, or a CUDA kernel limit) still aborts the spawn and the load
retries once without speculative decoding.
Adds the full Qwen3.5 + Gemma-4 (regular and QAT) auto/off/forced resolver
matrix, plus explicit sub-3B exemption tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Always compare the separate drafter in the reload-skip mirror
The sub-3B wrapper around the drafter compare could skip it when the drafter
was deleted out from under a running sub-3B server (detected None, stored set),
leaving a stale launch. The resolved-path compare is cheap and already handles
every case, so drop the _auto_mode_drops_mtp guard (and its now-unused imports)
and always compare when the mode can use a drafter and the user does not own
--spec-type. Addresses review feedback on #6191.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* -Added catch for unsloth running inside of system32 (for windows)
- changed import statement to import os as _os instead of only os.path as _osp
- replaced __osp mention with __os.path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adressed geminis codereview. Did not change Error message as "System32" is precise enough and the _system32 variable is lowercase only
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
* Studio: fetch the release source asset for exact (mix) source builds
The source-build fallback rebuilt the codeload/archive URL from the
source repo and commit. A mix build's merged tree is never pushed to any
repo (it ships only as the release's llama.cpp-source-commit-<sha>.tar.gz
asset), so codeload 404s on the merge commit and an uncovered host could
not build from source. When an exact-source asset exists, fetch it
directly from the release and keep codeload as the fallback for vanilla
builds whose commit is real.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep audio feature extractors right padded when loading processors
FastBaseModel.from_pretrained passes padding_side=left to
AutoProcessor.from_pretrained for generation, and ProcessorMixin forwards
the kwarg to every sub-component, including audio feature extractors.
Stock transformers right-pads audio: frame-validity masks assume trailing
padding. The leaked left padding shifts mel content to the end of the 30s
window for Whisper and gives Gemma 4 one extra valid mel frame on clip
lengths off the hop boundary, desyncing audio features from placeholder
tokens and crashing training on transformers 5.5.0 to 5.9.x with 'Audio
features and audio tokens do not match'.
Reset the feature extractor to right padding at the single processor
finalization point. Text tokenizer padding stays left. Verified on
transformers 5.5.0: the loaded processor now matches a fresh stock
AutoProcessor exactly, and a 9-combination Gemma 4 audio forward probe
goes from 4 failures to none.
* Add explicit None check before reading feature extractor attributes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: ignore unsupported env proxy during Studio startup
* fix: handle missing socksio env proxy at startup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Match printf logging style and inline the proxy predicate for PR #6102
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Installer: harden GPU detection follow-ups after #6174
Ports the NVIDIA-priority and /proc/driver/nvidia/gpus hardening from #6174
to the remaining pathways and adds recovery for already-poisoned venvs:
- install_python_stack.py: add _ensure_cuda_torch so 'unsloth studio update'
force-reinstalls CUDA torch when the venv carries a ROCm build on an NVIDIA
Linux host (the pre-#6174 poisoning signature). Honors UNSLOTH_TORCH_BACKEND,
UNSLOTH_ROCM_TORCH_INSTALLED, and CUDA_VISIBLE_DEVICES=-1/'' opt-outs; never
touches healthy CUDA, deliberate CPU wheels, macOS, or Windows.
- install_llama_prebuilt.py: detect_host gains the /proc NVIDIA fallback and
skips ROCm probes when NVIDIA is usable; forwarded --rocm-gfx/--has-rocm
overrides still win.
- setup.sh: GPU summary classifies NVIDIA first through a timeout-bounded
probe with the /proc fallback; AMD probes are bounded and gain a KFD
vendor_id 4098 fallback; the llama.cpp source build only selects
GGML_CUDA/GGML_HIP when the matching GPU is actually detected.
- install.sh: bound both nvidia-smi calls with a 10s timeout (no behavior
change when healthy or when the timeout binary is absent); classify the
exported UNSLOTH_TORCH_BACKEND on the final index path segment so custom
mirrors containing 'rocm'/'gfx' in their base path are not mislabeled.
- install.ps1 + setup.ps1: NVIDIA probes now require a real 'GPU N:' row from
nvidia-smi -L under a 10s bound instead of bare exit code 0; later CUDA
version and compute_cap queries are bounded too.
Tests: 3 new test files (50+ tests), suite at 788 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Resolve-CudaToolkit driver probe for extracted-function unit test
tests/studio/test_resolve_cuda_toolkit.ps1 extracts Resolve-CudaToolkit alone
into a child pwsh and stubs nvidia-smi with a .ps1 script. The bounded runner
is not in scope there (and ProcessStartInfo cannot dispatch .ps1 stubs), so
the DriverMaxCuda parse silently returned nothing and the major-mismatch
scenarios failed. Fall back to direct invocation when Invoke-NvidiaSmiBounded
is unavailable; production setup.ps1 always has it defined and keeps the
10s bound.
* Treat CUDA_VISIBLE_DEVICES empty or -1 as hidden in NVIDIA-first guards
The NVIDIA-first guards added in this branch only special-cased
CUDA_VISIBLE_DEVICES=-1 at two setup.sh gates and ignored the empty-string
form entirely, while the Python detector (install_llama_prebuilt.py)
already treats both as hidden. On a mixed AMD+NVIDIA host steered to the
AMD card via CUDA_VISIBLE_DEVICES, the guards suppressed the AMD probes,
so setup.sh fell to a CPU llama.cpp build and install.sh picked CUDA
wheels instead of ROCm.
Move the policy into the helpers so every consumer agrees:
- install.sh: new _cvd_hides_nvidia checked first in _has_usable_nvidia_gpu
- studio/setup.sh: same via _setup_cvd_hides_nvidia; the two ad-hoc
CUDA_VISIBLE_DEVICES=-1 gate conditions are now redundant and removed
- studio/install_python_stack.py: _has_usable_nvidia_gpu returns False
when CUDA_VISIBLE_DEVICES is set to or -1 (whitespace tolerated)
Tests: 5 new sh scenarios (hidden via , -1, padded -1, visible device,
and mixed host with hidden NVIDIA restoring the ROCm route) plus a pytest
class covering all three implementations behaviourally.
Addresses the review comment on the NVIDIA-first setup.sh block.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Retrigger CI after PyPI 503 outage during the previous run
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The import-hoist safety step built its changed-file list with a two-dot
diff against the base branch tip on a shallow clone. Once the base
branch moves past a PR's branch point, that diff includes every file
the base changed since, and the verifier compares newer base code
(BEFORE) against the PR's older snapshot (AFTER). This time-reversed
comparison reports the base branch's own refactors as blockers on PRs
that never touched those files, forcing branch updates or admin merges.
Resolve the true merge-base through the compare API, fetch that single
commit by SHA (clone stays shallow), and use it for both the file list
and --before. Verified against the PR 6137 failure: with the base tip
the step flags studio files from a later main commit; with the
merge-base the file list contains only the PR's own files and the
verifier passes.
* Add FastDiffusionModel slow path for text-diffusion models (DiffusionGemma)
Text-diffusion models (DiffusionGemma) use a block-diffusion generate loop and a novel
backbone, so Unsloth's autoregressive kernel/compile patching does not apply. FastDiffusionModel
loads the unmodified HuggingFace model (outputs stay bit-identical to transformers) and adds only
the safe conveniences: 4bit/8bit loading, PEFT LoRA (attention + dense MLP; the fused 3D MoE
experts are noted as a follow-up), the (model, tokenizer) API, and for_inference/for_training.
FastModel.from_pretrained auto-routes diffusion model_types to this path, including a fallback that
aliases the legacy "diffusion_gemma" config to the "diffusion_gemma4" classes current transformers
ships. New file unsloth/models/diffusion.py; loader.py adds the dispatch and the slow-path-aware
get_peft_model / for_inference / for_training.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* FastDiffusionModel: use unsloth's standard dtype resolution
Match the FastModel loader instead of the one-line bf16/fp16 pick:
SUPPORTS_BFLOAT16 = is_bfloat16_supported(), default to bf16 when supported
else fp16, downgrade an explicit bf16 on unsupported hardware with a warning,
and assert a valid dtype.
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: offer the in-app llama.cpp update for source-build (markerless) installs
Source-build installs have no UNSLOTH_PREBUILT_INFO.json marker, so freshness
reported supported=False and the Update button never showed (notably on macOS,
where the fork shipped no prebuilt before b9585 and setup fell back to a source
build). When an install has no marker but an official prebuilt now exists for
the host, surface the update and let one click swap it in place.
- install_llama_prebuilt.py: published_repo_for_host() (the setup.sh host->repo
rule in Python) and a --resolve-prebuilt mode that reports whether a prebuilt
exists for this host without downloading.
- llama_cpp_update.py: markerless branch in get_update_status/start_update,
version-suppressed so source builds already newer than latest are not nagged;
fail-open throughout.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: run llama update detection off the event loop, expose source_build
The markerless source-build check probes the host and reads GitHub, so run
get_update_status and start_update in a worker thread to keep the API
responsive. Expose source_build in the status response so the banner can label
the source-build switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the llama-route auth stub out of sys.modules for the rest of the suite
test_llama_route.py replaced sys.modules['auth.authentication'] with a
bare stub at collection time and never restored it, so every later test
importing create_access_token got the stub: 17 failures across
test_desktop_auth, test_middleware, test_openai_tool_passthrough and
test_rag_preview on all four Backend CI Python versions. Import the
real module when its deps are available and only stub in minimal envs,
popping the stubs after the standalone route load either way.
* Studio: address review on the source-build update path
- published_repo_for_host: route CPU-only Windows to ggml-org too (mirrors
setup.ps1; the fork ships no win-cpu bundle), macOS always the fork.
- markerless detection compares/display the upstream llama_tag, not a possible
fork wrapper release_tag, so a source build is not wrongly judged newer.
- do not offer when there is no resolvable install root (a pinned
LLAMA_SERVER_PATH outside a managed dir): an apply would not take effect.
* Ignore version probes in the update tests' subprocess capture
The status polls in these tests trigger the new source-build detection,
which shells out to llama-server --version through the same patched
subprocess.run. On slow runners that probe lands after the installer
call and clobbers the single captured argv, failing the flag
assertions (seen on the 3.10/3.11 Backend CI jobs). Skip probe calls
in all three fakes so only the installer invocation is captured.
* Skip markerless re-detection while the update job is swapping the tree
On a source-build install the frontend polls update-status every 3s
during an apply, and each poll ran _source_build_status, which execs
the very llama-server binary the job is concurrently replacing. On
Windows that exec can hold the exe long enough to fail the installer's
os.replace; everywhere it is a per-poll subprocess spawn for a status
the poller does not read (it only consumes job progress). Gate the
markerless branch on the job not running; the marked path is probe-free
and still returns the live job state.
* Studio: tighten source-build update root, repo routing, and downgrade guard
Only manage a markerless install when the active binary lives under a
resolvable llama.cpp root (marker dir, UNSLOTH_LLAMA_CPP_PATH it sits in,
or a llama.cpp ancestor); a pinned LLAMA_SERVER_PATH or a PATH/system
binary is left alone so an apply cannot install where it would not take
effect. Gate start_update on the same suppression as detection so a
direct POST cannot downgrade a source build newer than the latest
prebuilt. Route Linux hosts with AMD tooling (rocminfo/amd-smi/hipconfig/
hipinfo) to the fork in --resolve-prebuilt, matching setup.sh, so a HIP
source build is not offered an upstream CPU prebuilt.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover inactive env root and pinned llama.cpp checkout in update root tests
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
#5963 folded the manifest resolver into the simple-path resolver,
removed the installer's --simple-policy flag, and moved macOS prebuilts
to the unslothai/llama.cpp fork. The MLX CI prebuilt step still passed
the deleted flag, so argparse exits 2 and the workflow has been red on
every main push since. Point the step at the fork with no extra flags,
exactly like studio/setup.sh on macOS.
* Studio: forward preserve_thinking + reasoning_effort on the OpenAI passthrough
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Provide _request_reasoning_kwargs on the responses passthrough test backend mock
The OpenAI passthrough body builder now asks the active backend for
capability-gated reasoning kwargs. The responses stream adapter test fakes
the backend with a bare SimpleNamespace, so give it the same method a
non-reasoning template would expose (returns None, keeping
chat_template_kwargs out of the captured body).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten the backend mock comment
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
PR #6153 taught the chat UI driver to find Compare chat inside the
decluttered plus menu (#6140) but the extra UI driver kept looking for
the old top-level item, so all three Studio UI CI workflows have been
red on main since (Compare nav not found). Mirror the same
hover-then-click More fallback here.
* fix: prevent ROCm torch from installing on NVIDIA Linux hosts
NVIDIA's open kernel module (driver 560+) registers GPU topology nodes in
the KFD sysfs hierarchy with non-zero gpu_id values. The _has_amd_rocm_gpu
(install.sh) and _has_rocm_gpu (install_python_stack.py) sysfs fallbacks
previously treated any non-zero gpu_id as proof of an AMD GPU, so an
NVIDIA-only host with the open kernel driver was misrouted to the ROCm
install path, replacing the correctly-installed CUDA torch with ROCm wheels.
Fixes:
1. install.sh _has_amd_rocm_gpu sysfs fallback: require vendor_id 4098
(AMD 0x1002) in the KFD node properties file before declaring an AMD
GPU present. NVIDIA KFD nodes carry vendor_id 4318 (0x10DE) and are
now skipped.
2. install_python_stack.py _has_rocm_gpu sysfs fallback: same vendor_id
guard. Also preserves the existing fallback for older kernels that
don't ship a properties file (trusts gpu_id alone there).
3. install.sh now exports UNSLOTH_TORCH_BACKEND ("cuda"/"rocm"/"cpu")
immediately after get_torch_index_url() resolves the wheel family.
install_python_stack.py reads this as _TORCH_BACKEND and short-circuits
_ensure_rocm_torch() entirely on cuda/cpu hosts, providing a second
layer of defense that is independent of subprocess GPU detection.
Tests: 9 new cases in TestHasRocmGpuKfdVendorGuard,
TestEnsureRocmTorch, and TestInstallShStructure cover all three changes.
Full test_rocm_support.py suite: 289 passed, 2 skipped, 0 failed.
Closes#6172
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: show actual torch backend in progress step labels
The 'ROCm torch check' and 'ROCm torch (final)' step labels were
hardcoded regardless of whether the installer was targeting CUDA, ROCm,
or CPU. On NVIDIA hosts they showed 'ROCm' even though no ROCm wheels
were being installed, which was misleading.
Add _torch_step_label(suffix) which reads UNSLOTH_TORCH_BACKEND (set by
install.sh) and formats the label as e.g. 'torch check (cuda)' or
'torch final (rocm)'. Falls back to live GPU detection for standalone
studio update runs that bypass install.sh.
* fix: make KFD sysfs vendor check conservative -- skip if no properties file
The previous implementation fell through to `return True` when the KFD
node's properties file was missing (OSError), intending to support older
kernels. But NVIDIA open driver KFD nodes can also lack a properties file
on some kernel versions, so the fallback still produced a false positive.
Change the `except OSError: pass` to `continue` so any node without a
readable properties file is skipped rather than trusted. KFD properties
files exist on every kernel version that actually exposes /sys/class/kfd,
so this does not regress real AMD GPU detection -- if the directory exists
at all, properties files will be present for genuine GPU nodes.
* fix: bulletproof NVIDIA vs AMD GPU detection
Four changes that together ensure ROCm torch can never be installed on an
NVIDIA host regardless of which detection path fires:
1. _has_rocm_gpu() (Python): NVIDIA guard at the top -- returns False
immediately when _has_usable_nvidia_gpu() is True, blocking rocminfo,
amd-smi, and KFD sysfs from producing a false positive even when ROCm
tools are co-installed alongside the NVIDIA driver.
2. _has_amd_rocm_gpu() (install.sh): same NVIDIA guard -- calls
_has_usable_nvidia_gpu first and returns 1 if it succeeds.
3. _has_usable_nvidia_gpu() (Python): adds /proc/driver/nvidia/gpus/
sysfs fallback. The NVIDIA driver populates this directory on Linux
regardless of nvidia-smi state, so a subprocess PATH gap, timeout, or
driver initialisation race can no longer silence NVIDIA detection.
4. _has_usable_nvidia_gpu() (install.sh): same /proc/driver/nvidia/gpus
fallback, tried after nvidia-smi -L rather than instead of it.
Together: NVIDIA wins at every decision point. If nvidia-smi works, it
confirms NVIDIA. If it fails, /proc/driver/nvidia confirms NVIDIA. If
somehow both fail, _has_rocm_gpu still checks NVIDIA first before any AMD
path runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: two KFD/proc-only corner cases from Codex review
1. KFD awk state not reset per node file (Ryzen+NVIDIA false positive):
The awk glob processes all topology node properties files in one pass.
Without FNR==1 reset, a Ryzen+NVIDIA host where an AMD CPU-agent node
sets amd=1 (vendor_id 4098, gpu_id 0) can combine with a later NVIDIA
node setting gpu=1 (gpu_id > 0), triggering found=1 before vendor_id
4318 is seen. Added FNR==1{ gpu=0; amd=0 } to reset per file.
2. proc-only NVIDIA not reaching CUDA wheel selection:
_has_usable_nvidia_gpu returning true via /proc/driver/nvidia fallback
left _smi empty, so get_torch_index_url entered the AMD/CPU branch and
selected CPU wheels despite NVIDIA being confirmed. Introduced
_nvidia_detected flag (separate from _smi) so the AMD branch is skipped
whenever NVIDIA is confirmed by any path, while _cuda_ver reads from
_smi when available (with the existing cu126 fallback when _smi is absent).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Follow-up to #6097. The update banner appeared 8s after load and auto-hid after
about 10s. Show it about 1s after a newer prebuilt is detected and keep it up
until the user dismisses it (click outside or the X) or runs the update; it
stays during an in-progress update so the progress is visible.
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: gracefully disable MTP when the model has no head or drafter
Selecting MTP or MTP+Ngram in Speculative Decoding on a GGUF with no nextn
head and no separate drafter aborted the whole load. llama-server does not
no-op an empty draft-mtp request: it exits with 'failed to measure MTP
context memory: failed to create llama_context', surfaced to the user as a
generic 'llama-server failed to start. Check that the GGUF file is valid
and you have enough memory.'
Build-time fix in _build_speculative_flags: when a forced mtp / mtp+ngram
mode targets a model with no MTP head and no drafter (is_mtp_model is
False), default back instead of emitting draft-mtp. mtp falls back to
--spec-default; mtp+ngram keeps the ngram-mod half, which needs no head.
Real MTP models (embedded head or separate drafter), sub-3B MTP overrides,
and the auto path are unchanged.
Runtime hardening: the existing post-launch MTP retry only fired for
separate-file drafters (--model-draft in spec_flags), so an embedded-head
model that the binary cannot build still hard-failed. Gate the retry on the
spec block requesting MTP, recognise the embedded-head abort strings
('failed to measure MTP context memory', 'failed to create llama_context'),
and make the drafter name None-safe in the warning.
Tests: extend the resolver matrix (forced mtp / mtp+ngram on a non-MTP
model) and add two cases asserting the default-back emission.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Adds an in-app "Update llama.cpp" banner and button to Unsloth Studio. When the installed prebuilt is behind the latest published release, a non-invasive banner appears; clicking Update downloads the latest prebuilt for this host and swaps it in place in the background, with no restart.
Detection reuses the freshness check from #5529. The update re-runs install_llama_prebuilt.py the same way setup.sh and setup.ps1 do after #5963: it forwards the published repo and the AMD gfx target derived from the install marker, and does not pass the removed --simple-policy or the arm64-only --cpu-fallback.
While the installer swaps binaries the backend enters a maintenance state (flag set under the serial load lock, active server unloaded) so a concurrent load cannot start a server from a half-swapped binary; the next load uses the new build. The banner also handles refused responses and jobs started in another tab so it never sticks on "Updating...".
Verified end to end on an NVIDIA B200: installed b9493, detected the update, applied it, and confirmed the binary at the same path advanced to b9585 in the same process. Hermetic backend tests and the frontend type-check pass.
* feat(hub): enable Run/New Chat for downloaded GGUF models, fix README bottom spacing
- enable the Run / New Chat action for GGUF models that are already downloaded, across the download card, on-device card, and hub page
- remove the extra bottom spacing under the model inspector README
* fix(hub): sync active GGUF variant on mount
* feat(hub): open a new chat immediately when Run is clicked
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
