entrypoint.sh: a container started without a GPU request has no
nvidia-smi at all (the toolkit injects it), so the old check 1 reported
'CUDA runtime in this image is broken, re-pull' for the most common user
error. Fold the missing-binary case into the actionable 'No GPU visible'
message and document the CPU-only option (UNSLOTH_ALLOW_CPU=1).
run.sh / test_locally.sh: guard empty-array expansions with the
${arr[@]+...} form; bash 3.2 (macOS /bin/bash) treats "${empty[@]}"
as unbound under set -u, which broke the documented macOS CPU path.
studio_launch.sh: exclude *_TOKEN, *_API_KEY, *_PASSWORD, *_SECRET,
*_LICENSE from the env snapshot written for SSH sessions; secrets stay
in process env only, never on disk.
supervisord.conf / Dockerfile.studio: pin HOME=/root for the studio and
jupyter programs (jupyter would silently fall back to token auth if HOME
were unset), default JUPYTER_PORT and UNSLOTH_ENABLE_SSHD at the image
level so a direct supervisord invocation cannot hit a bad %(ENV_*)s
expansion, and document the root-services decision (non-root parity with
the previous production image is a tracked follow-up).
docker_confirm.ps1: mirror the bash script's GPU selector translation so
GPUS=0 / 0,1 select devices instead of silently using all GPUs.
docker-publish.yml: studio cache scope moves to mode=min; a mode=max
cache of a ~24GB image would evict everything else in the 10GB GHA
quota for no hit-rate gain.
Studio's setup.sh provisioning runs install_llama_prebuilt.py, whose
host-probing cannot succeed inside an image build, so it fell back to a
CPU-only llama.cpp source build layered over the baked CUDA bundle.
setup.sh skips that fallback when build/bin/llama-server and
build/bin/llama-quantize are executable, so hardlink the installed bundle
into build/bin: zero extra bytes, $ORIGIN rpath still resolves, and no
symlink cycle when setup.sh later relinks the root quantizer to
build/bin/llama-quantize.
Two failures from the first in-image Studio install, both rooted in
install.sh probing the build host:
1. setup.sh aborted on the pre-linked llama.cpp dir: 'already exists and
is not marked as a Studio-owned llama.cpp install'. The dir is the
image's baked prebuilt, provisioned exclusively for Studio, so write
the .unsloth-studio-owned marker next to the binaries.
2. With no GPU and no nvidia-smi in the build container, install.sh fell
back to cu126 torch wheels for the Studio venv (and would pick cpu
wheels on a CI runner without /proc/driver/nvidia), so the published
image's Studio venv would depend on which host built it and could not
train on Blackwell. get_torch_index_url now honours an explicit
UNSLOTH_TORCH_INDEX_FAMILY override naming the index leaf (cu128,
cu130, rocm7.2, cpu, ...). The resolved family flows into
UNSLOTH_TORCH_BACKEND, which install_python_stack.py already consumes,
so the whole downstream chain follows the pin. Dockerfile.studio sets
cu128 on amd64 and cu130 on arm64 (DGX Spark / Grace).
The dockerignore uses an everything-out whitelist; fetch_llama_prebuilt.py
(base bake) and supervisord.conf + studio_launch.sh (Dockerfile.studio)
need explicit entries. docker_confirm.ps1 is the Windows Docker Desktop
counterpart of docker_confirm.sh.
The first bake attempt reused studio/install_llama_prebuilt.py, but that
resolver selects a bundle for the CURRENT host: on a GPU build host
/proc/driver/nvidia leaks into docker build and the resolver goes down the
CUDA path with no readable driver runtime (chosen_asset=none, exit 2),
while on a GPU-less CI runner it would resolve a CPU bundle instead. Both
violate the image's build-host-independence rule.
fetch_llama_prebuilt.py pins by build target only: amd64 takes the
linux-x64-cuda12-portable bundle, arm64 the linux-arm64-cuda13-portable
bundle (DGX Spark / Grace), both sha256-verified against the release's
llama-prebuilt-sha256.json. convert_hf_to_gguf.py plus gguf-py/ are
hydrated from the same release's source tarball so the converter's tensor
mappings match the binaries, mirroring unsloth_zoo's
_hydrate_converter_sources layout. LLAMA_PREBUILT_TAG build-arg overrides
the pinned release.
docker_confirm.sh: one-command confirmation script for any machine
(Linux / WSL2 / macOS) following the staging confirm-script conventions:
host + docker + GPU detection with CPU-mode auto-fallback, image pulls,
in-container torch.cuda check, 5-step LoRA training smoke, baked llama.cpp
verification, full-image boot probing Studio /api/health and JupyterLab
/api, PASS/WARN/FAIL summary with RESULT line.
Base image (docker/Dockerfile):
- Install JupyterLab + notebook + ipywidgets in a separate pure-Python uv
pass so the cu128 pin set cannot move; EXPOSE 8888.
- Bake the prebuilt llama.cpp bundle into /opt/unsloth/llama.cpp at the
runtime stage using studio/install_llama_prebuilt.py from the same
UNSLOTH_REF (sha256-verified, portable CUDA bundle since the build host
has no GPU; arm64 resolves the linux-arm64-cuda13 bundle). Export
UNSLOTH_LLAMA_CPP_PATH so unsloth_zoo's save_pretrained_gguf finds it
and never reaches the interactive install prompt or a source build.
- Optional github_token BuildKit secret for the resolver's API calls on
shared CI runner IPs.
Entrypoint: UNSLOTH_ALLOW_CPU=1 degrades a missing GPU to a warning so
Docker Desktop on macOS / Windows-without-WSL2-GPU and plain CPU hosts can
run Jupyter, GGUF tooling and Studio chat; with a GPU visible the normal
pre-flight still runs.
Full image (docker/Dockerfile.studio): now mirrors the production service
set under supervisord - Studio on 8000, JupyterLab on 8888, key-only sshd
on 22 (enabled only when PUBLIC_KEY/SSH_KEY is set). Points Studio's
llama.cpp dir at the baked bundle to skip a duplicate download, accepts
any git ref via fetch+checkout (CI passes commit SHAs), and FROMs a
digest-pinned BASE_IMAGE.
Publish workflow: base image moves to the base-* tag namespace; new
build-studio/merge-studio jobs publish the full image as :latest (hub
parity with the previous production image, which shipped Studio + Jupyter
+ SSH). Studio builds FROM the exact base manifest digest published by the
same run. GPU smoke job now also boots the full image and probes Studio
/api/health and Jupyter /api.
run.sh: UNSLOTH_GPUS=none, UNSLOTH_ALLOW_CPU forwarding, UNSLOTH_PORTS
publish flags, CPU-mode and Jupyter usage examples.
* Handle rope_type 'default' on transformers 5 to stop false RoPE warning
transformers 5 reports rope_type="default" for every plain (unscaled) config
and dropped "default" from ROPE_INIT_FUNCTIONS. _compute_config_rope_inv_freq
then did ROPE_INIT_FUNCTIONS["default"], hit KeyError, returned None and logged
"Could not apply RoPE scaling 'default'; long-context generation may degrade"
on every model load. The inv_freq was still correct (the constructor recomputes
vanilla on None), but the warning is a false alarm for unscaled models.
Compute the unscaled inv_freq directly for rope_type "default"/None instead of
going through ROPE_INIT_FUNCTIONS, so plain configs return the right value with
no warning. Scaled types (llama3/linear/yarn/...) are unchanged.
Also skip test_object_style_rope_scaling_on_config_delegates_correctly when
transformers strict-validates rope_scaling (5.x): it rejects a non-dict object
on config.rope_scaling, so the object-style delegation path cannot be set up
there. The test still runs and asserts on transformers <5.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: deduplicate lemonade ROCm prebuilt selection log
resolve_lemonade_rocm_choice() is called twice per install (direct
planner + resolve_upstream_asset_choice). The API fetch is already
memoised via _fetch_lemonade_release_cached but the selection log
lines were still emitted on both calls, printing the 'trying
lemonade-sdk ROCm prebuilt' banner and hash-manifest NOTE twice.
Add _lemonade_selection_logged set keyed on (gfx_target, asset_name)
and guard the two log() calls behind a membership check so they print
exactly once per process regardless of call count.
Also extend the _clear_lemonade_release_cache test fixture to clear
the new set between tests to prevent cross-test state bleed.
Fixes#6020
* fix: write log() output to stdout to avoid PowerShell NativeCommandError
On Windows, PowerShell treats any stderr output from a native process as
an error record and prefixes it with 'python.exe :' and sets the
ErrorId to NativeCommandError. Since log() wrote to sys.stderr, every
[llama-prebuilt] status line triggered this, making normal progress
output look like errors in the installer console.
Switch log() to sys.stdout. The download progress bar (DownloadProgress)
retains its stderr/tty logic unchanged -- that path is for interactive
terminal rendering, not status logging.
* fix: remove redundant 'or ""' in lemonade log_key
host.rocm_gfx_target is already guaranteed truthy by the early
return at the top of resolve_lemonade_rocm_choice. The fallback
was dead code.
* Keep resolver stdout machine-readable, route install logs to stdout
log() sending everything to stdout breaks the resolver modes: setup.sh
json.load()s the whole stdout, so one helper log line (network retry,
release-tag scan) corrupts the parse and silently drops back to building
"latest". Default log() to stderr and flip to stdout only on the install
path, where PowerShell otherwise renders stderr as NativeCommandError
noise. Also tighten the lemonade dedup comments.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
On a 0.0.0.0 bind whose public ip:port is not reachable (cloud firewall),
the banner still printed "Secure link access via Cloudflare: <url>" right
after "is NOT reachable from the public internet", which reads as if the
tunnel might also be blocked. The Cloudflare quick-tunnel works regardless.
Thread the reachability probe result through a module-level _public_reachable
tri-state and, when the public probe definitively failed but the tunnel is
up, print "Also, the secure link access via Cloudflare works: <url>".
Reachable or undecided cases keep the existing wording.
PyPI release unsloth 2026.6.3 is now live. Bump the pinned floor in
install.sh and install.ps1 from unsloth>=2026.6.2 to unsloth>=2026.6.3
so fresh installs resolve to the new wheel.
* Studio: llama.cpp update banner redesign, About tab license info, inline system prompt editing, naming cleanup
- Redesign the llama.cpp update banner to match the chat composer surface
(borderless rounded card, composer shadow, Hellix Medium title), rename
actions to Update and add a 15 minute Remind me later snooze
- Keep the banner up until the user explicitly acts on it; drop the
outside click dismissal
- Add a Settings > General > Notifications toggle to disable the banner
for training-only setups (on by default)
- Rename the Help settings tab to About and add a License section
(Unsloth Studio AGPL-3.0, Unsloth Core Apache-2.0) linking to the
license files in this repo
- Make the run settings system prompt box an inline editable textarea;
the popup editor opens when the prompt overflows the box
- Pointer cursor on the preset dropdown chevron
- Dark mode toasts use the chat composer surface color
- Replace standalone Studio with Unsloth in user facing strings; keep
Unsloth Studio, LM Studio, Fine-tuning Studio, Recipe Studio and CLI
commands unchanged
* Studio: open the system prompt popup on box click, balance banner padding
- The system prompt box opens the Edit System Prompt dialog on click,
matching the pencil action
- Slightly more bottom padding on the llama.cpp update banner so the
spacing reads even next to the action pills
* Studio: replace unsloth studio update with the installer commands in update guidance
- The unsloth studio update command no longer works, so the About tab
update section now shows the one-line installer (curl or irm) for
PyPI and unknown installs, and git pull plus the local installer for
checkouts
- Add a short note that unsloth studio update is no longer supported
- Link the Installation, Updating and Windows install docs pages
- The package update banner now copies the platform installer command
instead of unsloth studio update
* Studio: rounder account menu, inline system prompt box with popup from the label
- Account menu corners go from 14px to 18px via a specific override,
since list menus pin border-radius globally
- llama.cpp banner bottom padding 22px
- System prompt is an inline editable textarea again; clicking the
System Prompt label opens the popup editor, and an overflowing
prompt opens it on box click
* Studio: show the standard install commands in the About update section
- Both one-line install commands (MacOS/Linux/WSL and Windows
PowerShell) are always shown, labeled like the docs, since running
them again updates an existing install
- Drop the unsloth studio update deprecation note
- Add the Mac install guide to the docs links
* Studio: clearer platform toggle and layout in the About update section
- Section heading is Update
- Platform picker is a pair of pill buttons, MacOS / Linux and Windows,
and only the selected platform's install command is shown
- Intro reads: To install or update Unsloth
- Local update heading separates checkout guidance from the standard
install command
* Studio: report GitHub branch instead of dev for source checkouts
A source checkout not on an exact release tag now shows
GitHub <branch> (e.g. GitHub main) as the Studio version in About.
Detached or unusual HEADs still fall back to dev.
* Studio: tighten the About update section copy and toggle styling
- Platform toggle buttons are borderless pills
- Shorter local update wording and restart note
- Docs links read Mac and Windows
* Studio: tighten line spacing in the sidebar account button
* Studio: fix vanishing compact MCP icon on hover, single line pill tooltips
- Compact caret pills (MCP, RAG) keep their icon on hover for inactive
pills too; the off switch hover rules hid the icon while compact mode
hid the X, leaving an empty slot
- Compact icon tooltips and single line compact tooltips render as full
pills; wrapped tooltips keep the 9px corners. TooltipContent measures
line count in a ref callback since Radix mounts portal content
without re-rendering the wrapper
- 1px gap between the name and Unsloth lines in the sidebar account
button
* Studio: Projects hover plus button, align recents with the label
- Hovering the Projects nav item reveals a plus button that opens the
New project dialog, with the same circular hover treatment as the
chat row actions
- Recent chat titles start at the same x as the Recents label
- The system prompt overflow lock only engages for a non-empty prompt
with a laid-out box, so a mis-measure cannot turn clicks into the
popup
* Clip system prompt overflow inside the rounded box
Wrap the inline system prompt textarea in a rounded overflow-hidden
surface so scrolled text and the scrollbar stay inside the box. The
focus ring moves to the wrapper via focus-within.
* Add updating progress bar to llama banner and shorten settings copy
While an update is applying, the banner action row becomes an
indeterminate progress bar that keeps animating under reduced motion,
matching the other loading indicators. Settings descriptions across
General, Profile, Appearance, Chat, Connections, API, and About are
trimmed without losing meaning.
* Address review: desktop update note, server platform detection, zh-CN keys
The About tab no longer shows terminal install commands in the desktop
app, where the bundled backend updates through the built-in updater;
it shows a short note and the docs links instead.
fetchDeviceType now sends the auth token to /api/health, which only
reports the server platform to authed callers, and caches only a
server-reported value. Copied install commands then match the host
platform rather than the browser when they differ (WSL, SSH).
zh-CN gains translations for the new notification and license keys,
the renamed About tab title, and the desktop update note.
* Real download progress for llama.cpp updates, prompt and sidebar polish
The update worker now streams the installer output and parses its
download percent lines into job progress, exposed via the update-status
API. The installer emits finer non-tty milestones when
UNSLOTH_PROGRESS_PERCENT_STEP is set; the worker requests 5 percent
steps. The banner renders a determinate bar from the reported fraction
and falls back to the sweep until the first percent arrives.
Also removes the focus ring on the inline system prompt box and
slightly shrinks the Projects hover plus icon.
transformers >= 5.x makes _is_package_available always return a
(exists, version) tuple, which is truthy even when the package is
absent. The flash_attn and vLLM availability checks treated the result
as a bool, so they always entered the "package present" branch:
- flash_attn (CUDA + HIP): the inner import raises when flash-attn is
not installed, printing a false "Flash Attention 2 installation seems
to be broken" warning before falling back to xformers.
- is_vLLM_available(): always reported vLLM as installed.
Add a small _package_available() helper that normalises the result to a
bool (handling both the new tuple and the legacy bool return), and route
the three call sites through it.
Fixes#6155
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(rocm): stop overwriting ROCR_VISIBLE_DEVICES in apply_gpu_ids
ROCR_VISIBLE_DEVICES uses HSA agent-level indexing, not physical GPU
indices. Setting it to a bare integer breaks multi-GPU ROCm systems
where the parent already set ROCR_VISIBLE_DEVICES=0,1: narrowing to
1 causes torch.cuda.is_available() to return False in the training
worker, producing a misleading 'no HIP accelerator' error even on a
correctly configured ROCm host.
HIP_VISIBLE_DEVICES is sufficient for GPU selection on ROCm.
Leave ROCR_VISIBLE_DEVICES inherited from the parent environment.
* test(rocm): update apply_gpu_ids test to assert ROCR_VISIBLE_DEVICES is not overwritten
• fix: handle empty responses tool output
Normalize empty Responses `function_call_output.output` values before converting them into Chat Completions `role="tool"` messages. Empty strings, whitespace-only strings, and empty arrays now use the existing no-output sentinel, while non-empty text and content arrays are preserved.
Add regression coverage for empty tool outputs, image payloads outside `output`, content-array serialization, validator round trips, and preserving non-empty text.
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
When pinning GPUs for the llama-server child, the ROCm path set both
HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES to the same physical
indices. These masks filter at different layers and stack:
ROCR_VISIBLE_DEVICES reduces the visible set at the HSA/ROCr layer and
re-indexes from 0, then HIP_VISIBLE_DEVICES indexes into that reduced
set. _select_gpus ranks by free VRAM and picks the most-free card, so a
single non-zero pin (e.g. "1") becomes out of range at the HIP layer,
HIP enumerates 0 devices, and the model silently runs on CPU
("ggml_cuda_init: failed to initialize ROCm: no ROCm-capable device is
detected").
Set only HIP_VISIBLE_DEVICES (which narrows correctly on its own) and
clear any inherited ROCR mask so it can't double up.
Verified on a 2x Radeon AI PRO R9700 (gfx1201) host, ROCm 7.1.1: the
same selected=[1] load that fell back to CPU (~7.7 tok/s) now runs on
the GPU (~78 tok/s).
Fixes#6175
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* studio: import MCP servers from a config file
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* import config' on the add-server form
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: defensively handle MCP config imports
* fix: address MCP import review follow-ups
* fix: preserve apostrophes in Windows MCP commands
* fix: preserve apostrophe-wrapped Windows MCP args
* fix: align Windows MCP parsing with list2cmdline
* fix: preserve explicit MCP remote transport intent
* fix: trim MCP remote URLs before transport checks
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: add `unsloth chat` CLI command
Interactive chat REPL on the shared Studio backend: trained-model picker
when no model is given, /think and /compare toggles (adapter toggle on
CUDA, side-by-side base-model load on MLX), markdown streaming, and
connect-if-running Studio server mode so models stay warm across
sessions and are shared with the UI.
* fix settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix error handling and compare base precision
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix chat CLI backend imports and GGUF drafter loading
* Hide split thinking tags in chat CLI streams
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* fix(studio): surface live step with null loss through the SSE progress stream
The metric histories skip non-finite steps, so during a NaN stretch the
SSE live loop and final complete event replayed the last finite
step/loss pair. Follow the live progress step when it is ahead of the
history tail and report its loss honestly (null until recovery).
Completes the NaN honesty fix for the SSE consumer flagged in review.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply live-step handling to inactive streams and clear the UI loss on null for PR #6206
Fresh /progress connections after a finished run took the inactive branch
which still replayed the last finite step and loss pair; apply the same
live-step correction there. On the frontend, applyProgress kept the stale
currentLoss when a payload advanced the step with a null loss; clear it so
the display shows -- until the loss recovers. Widen the runtime state type
to number | null, which the view layer already handles.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving
A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.
Changes:
- After llama-server health, read default_generation_settings.n_ctx from
/props and adopt it whenever it is below Studio's computed context, with
a warning. The load response, status route, UI value, and the passthrough
max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
the passthrough drops whole middle turn-groups (system prompt, first turn,
and recent turns kept; tool calls stay paired with their results), clips
oversized contents middle-out when group-dropping is not enough, clamps
max_tokens to the generation headroom, and retries. Default stays 'error'
with code=context_length_exceeded so clients running their own compaction
keep full control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)
Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
always passes --parallel N, which silently splits -c into per-slot windows
of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
--fit-ctx <requested> for explicit requests so fit offloads or fails into
the existing --fit off retry instead of silently shrinking the window.
Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix config.rope_scaling being dropped by the replaced rotary embedding (#2405)
On modern transformers, LlamaModel builds its rotary embedding from config
using unsloth's replacement LlamaRotaryEmbedding class, whose config path
computed vanilla inv_freq and ignored config.rope_scaling entirely. The
llama3/linear/longrope dispatch in patch_llama_rope_scaling rewrites
LlamaAttention.__init__, which no longer constructs rotary embeddings, so it
never fires; the model-level rotary is then copied onto every attention
layer. Result: Llama-3.1/3.2/3.3 ran with unscaled RoPE on the
FastLanguageModel path and collapsed into repetition loops past roughly 29K
tokens (PASS at 28867, FAIL at 31767 in needle retrieval). FastModel was
unaffected because vision.py keeps transformers' own rotary. qwen2, qwen3,
qwen3_moe, mistral and cohere assign the same base class, so any rope-scaled
config of those families was equally exposed.
The fix makes the base class config path compute inv_freq and
attention_scaling via transformers' ROPE_INIT_FUNCTIONS (covers llama3,
linear, dynamic, yarn, longrope), with an inline llama3 fallback reading
factors from config for older transformers, degrading to prior behavior on
any failure. attention_scaling is applied in _set_cos_sin_cache (1.0 default,
exact no-op for unscaled paths) and persists across extend_rope_embedding.
A type(self) guard prevents double-scaling via the legacy scaled subclasses.
Adds tests/utils/test_rope_scaling_drift.py (AST tripwire + behavioral
inv_freq/cos-cache/extension checks, validated to fail 4 of 5 on the unfixed
code) and wires it into the existing consolidated CI HARD GATE step.
Verified on GPU: 48K-token needle retrieval flips FAIL to PASS for
FastLanguageModel in bf16 and 4bit, 20K stays PASS, scaled inv_freq matches
transformers exactly, and the left-padded batch generation guard still gets
exact solo-vs-batched token matches.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: normalize object-style rope_scaling, vectorize llama3 fallback
config.rope_scaling can be a config object rather than a dict on newer
transformers; _rope_scaling_as_dict normalizes it (to_dict/dict/vars
fallbacks) before any .get() access, with a regression test using a
dataclass stand-in. The inline llama3 fallback now uses torch.where instead
of a per-frequency Python loop; verified bit-for-bit equal to transformers
ROPE_INIT_FUNCTIONS for factor 8 (Llama-3.1) and factor 32 (Llama-3.2).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: CPU-safe rope guard tests, normalized config for delegation
The rotary constructor builds per-device CUDA caches, so the behavioral tests
that instantiate it cannot run on GPU-less CI. Restructured into three layers:
the AST tripwire now also asserts the constructor stays wired to
_compute_config_rope_inv_freq; the CPU layer tests that pure helper directly
(llama3 dict, llama3 object, linear object, default type) with no
instantiation; the instantiation and cache tests are gated behind a real CUDA
probe (actual tensor allocation, so import-time CUDA spoofs cannot fool the
gate). Verified: 9 passed with GPU; 5 passed 4 skipped with CUDA hidden; 5
failed 4 skipped on the unfixed code in CPU mode.
Delegation to ROPE_INIT_FUNCTIONS now retries with a shallow config copy
carrying the normalized rope_scaling dict when the original was an object the
installed transformers cannot read; covered by a linear-object test, which has
no inline fallback and passes only through that retry path.
* Tighten comments in rope scaling fix and guard test
Comment and docstring reduction only; verified code-identical with
scripts/comment_tools.py check --strip-docstrings (AST signature match on
both Python files). All guard tests unchanged: 20 passed with GPU, 5 passed
4 skipped with CUDA hidden.
* Apply repo kwarg-spacing format
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Studio's frontend exposes a Resume action and submits requests with
resume_from_checkpoint set to a previous run's output_dir. The CUDA
training paths in worker.py read this field from config and pass it to
trainer.train() (see lines 2729-2787 and 3108-3229). The MLX path
_run_mlx_training did neither: it never read config['resume_from_checkpoint']
and called trainer.train() with no args. The MLX trainer also did not
accept the kwarg, so even threading it through would have been a no-op.
With this PR + the unsloth-zoo companion PR adding the trainer-side
support (saves optimizer_state + trainer_state, accepts and applies
resume_from_checkpoint in MLXTrainer.train()), MLX Resume now works
end-to-end. Verified on M2 16GB with Qwen3-0.6B + unsloth/LaTeX_OCR:
loss at every post-resume step matches a fresh run bit for bit
(2.168627977371216 == 2.168627977371216 at step 6, etc).
Two lines: read the field near the other config.get() extractions in
_run_mlx_training, pass it as a kwarg at the trainer.train() call site.
Companion PR: unslothai/unsloth-zoo#751
When training produced a NaN or Inf loss event, the handler filtered the
value to None but never updated progress.loss — clients kept seeing the
last finite value as if everything were fine.
Now: on non-finite loss, clear progress.loss to None and log a one-shot
warning. Training continues (no phase=error, no _should_stop), matching
the expected behavior for a non-fatal numerical event.
Test: tests/test_training_nan_loss_handling.py with 6 cases covering
finite, NaN, +/-Inf, idempotency of the one-shot warning, and recovery
when a finite step follows a non-finite one.
* Studio: auto Cloudflare tunnel for 0.0.0.0 launches
Binding Studio to 0.0.0.0 for remote access often leaves the raw
http://<ip>:<port> URL unreachable (https-vs-http, blocked high ports,
closed cloud security groups). On a wildcard bind, auto-start a free
cloudflared quick tunnel and show its https://*.trycloudflare.com URL in
the startup banner:
Secure link access via Cloudflare: https://<random>.trycloudflare.com
- new studio/backend/cloudflare_tunnel.py: find or download+cache the
cloudflared binary (per-OS/arch GitHub release, safe .tgz extract),
start the tunnel, parse the URL, tear it down. Stdlib only; best-effort
and non-fatal throughout (a missing binary or offline box never blocks
or slows startup).
- run_server starts the tunnel for 0.0.0.0 only (skips loopback, api-only
and Colab), prints the line in the banner, and _graceful_shutdown stops
the child so it never orphans.
- --cloudflare/--no-cloudflare flag (default on) on `unsloth studio` and
`unsloth studio run`, forwarded through the re-exec into run_server.
- tests for the helper, the CLI flag forwarding, and the run.py defaults.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio cloudflare: send a User-Agent on the cloudflared download
GitHub's CDN can 403 the default Python-urllib User-Agent on release asset
downloads. Set an explicit UA and pin it with a test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio cloudflare: address review (opt-out for subcommands, tunnel teardown)
- reject --no-cloudflare placed before a subcommand (it would not reach the
subcommand), mirroring the --parallel guard
- register the tunnel before waiting for its URL so a shutdown during the wait
stops cloudflared instead of orphaning it
- tear the server + children down if `unsloth studio run` startup aborts
(health timeout, model-load error, Ctrl+C) before the wait loop
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Strip trailing whitespace in construct_chat_template (fixes#992)
construct_chat_template() only lstrip()s the template, so a template ending
in a newline (e.g. the default Llama-3 template + a trailing '\n') keeps the
trailing whitespace. That breaks the rfind-based repeated-example detection
and, on current main, also the regex fallback, which then raises
"Could not recover a two-example structure from chat_template".
Surrounding whitespace is not significant to the template here (the left side
is already stripped), so strip() both ends. Verified that the valid no-newline
template still parses via the primary path and that the meaningful trailing
token (e.g. <|eot_id|>) is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix trailing whitespace handling via parse retry for PR #6008
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix(studio): reuse venv Python in setup instead of re-probing system
* Reuse venv Python for studio setup
Pass the venv interpreter from install.ps1 to studio/setup.ps1 via UNSLOTH_SETUP_PYTHON and prefer it over probing the system. Added Resolve-ReusedSetupPython to accept the handed-off path (or derive the venv python when setup runs standalone), validate it (Python 3.11–3.13 and non-conda), and inject its Scripts dir onto PATH. When a reused interpreter is accepted, py.exe enumeration and further system probing are skipped. install.ps1 also sets the env var before running setup and removes it on cleanup to avoid leaving state behind. This prevents setup from being tripped by unsupported Python 3.14 or Windows Store stubs on PATH.
* Harden setup Python detection for PR #6033: py -All, shared conda check, bare ~ guard
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* center the search dialog and change the wrong borders
* fix the mistake of 1 to l
* Fix/adjust search dialog radius for PR #6184
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Stop HF 429 rate limits from sinking the llama.cpp prebuilt path in Studio CI
The Windows Studio API smoke job failed when anonymous huggingface.co
fetches of the tiny GGUF validation model (stories260K.gguf) hit HTTP 429
on the shared runner IP. The installer correctly refused the unvalidated
prebuilt and fell back to a source build, which the prebuilt assert then
flags. Three layers fix this:
1. Installer: auth_headers sends HF_TOKEN (or HUGGING_FACE_HUB_TOKEN) to
huggingface.co hosts, mirroring the existing GH_TOKEN handling for the
GitHub API rate limit. A redirect handler strips Authorization when a
download is redirected off-host (CDN signed URLs reject foreign auth;
urllib forwards headers on redirect, unlike requests/huggingface_hub).
2. Workflows: the HF_HOME prime steps also prefetch the validation model
so the install's hf_hub_download resolves from the local cache even
when the Hub is rate limiting; cache keys bumped v1 to v2 to repopulate.
This also covers fork PRs, which cannot see secrets.
3. Workflows: every Install Studio / update step that already passes
GH_TOKEN now also passes HF_TOKEN, so both the huggingface_hub path and
the direct URL fallback are authenticated.
Tests: tests/studio/install/test_hf_auth.py covers token-to-host routing,
the cross-host redirect strip, and the download_bytes wiring (offline).
Verified live: authenticated download of the validation model through the
new opener (CDN redirect exercised, pinned sha matches) and an offline
hf_hub_download cache hit against an HF_HOME primed by the new step.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Require a found ROCm DLL before forcing BNB_ROCM_VERSION in Studio paths
main.py previously set BNB_ROCM_VERSION=72 whenever HIP_PATH or ROCM_PATH
was set, and the training worker fell back to a blind 72 when DLL
detection found nothing. On a Windows machine with the AMD HIP SDK
installed but CUDA or CPU torch, that forces a ROCm backend onto a
non-ROCm bitsandbytes wheel, which raises at import. Both paths now only
write the override when a libbitsandbytes_rocm DLL actually exists (or a
seeded value is already present), matching the strict gates in
unsloth/import_fixes.py.
Also removes four redundant local import shutil statements in
unsloth/save.py that shadow the module-level import, the same pattern
that caused the UnboundLocalError fixed in #6149.
* Worker: gate the BNB override on a found ROCm DLL, preserving seeded marker
Review follow-ups: track _found_rocm_bnb in the worker like main.py so a
ROCm DLL with an unparsable name still gets the seeded or 72 fallback,
and skip the env write entirely when no DLL exists so a seeded value
keeps its sitecustomize marker and stays redetectable by later import
fixes.
* Fix bitsandbytes ROCm GPU arch and warp size detection on Windows
bitsandbytes resolves the ROCm GPU architecture (and warp size on
0.49.x) by shelling out to rocminfo / hipinfo.exe via PATH at import
time. On Windows neither tool is normally on PATH (AMD torch wheels
ship hipInfo.exe into the venv Scripts dir, only on PATH while
activated), so every `import bitsandbytes` logs an ERROR and WARNING,
ROCM_GPU_ARCH degrades to unknown, and the 0.49.x warp size defaults
to 64, which is wrong on RDNA (wave 32) and silently disables
pre-quantized 4-bit models via ALLOW_PREQUANTIZED_MODELS.
Install a one-shot MetaPathFinder before unsloth_zoo is imported (the
first bitsandbytes import on ROCm) that swaps get_rocm_gpu_arch and
get_rocm_warpsize for torch-device-properties-first implementations
right after bitsandbytes.cuda_specs executes, before cextension reads
them. Falls back to running hipInfo.exe by absolute path (venv
Scripts, conda Scripts, HIP SDK / AMD installer dirs). Repairs the
constants in place when bitsandbytes was imported first. Strict no-op
on non-Windows, non-ROCm builds, missing bitsandbytes, and versions
that fix this upstream. Opt out with UNSLOTH_DISABLE_BNB_ROCM_FIX=1.
Proposed upstream in bitsandbytes-foundation/bitsandbytes#1969;
shipped here so all bitsandbytes versions are covered. Verified on
gfx1151 Strix Halo, Windows 11, torch 2.11.0+rocm7.13.0 against
bitsandbytes main, 0.49.2, and a torch-props-fixed variant.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the bitsandbytes ROCm detection fix
Comment and docstring pass only. AST comparison with docstrings
stripped confirms every definition is identical to the version the
12 scenario suite ran against, and the suite plus the drift test
pass unchanged on the edited files.
* Keep the bitsandbytes cuda_specs finder installed for reload support
Simulation testing caught a regression in the one-shot design:
importlib.reload(bitsandbytes.cuda_specs) re-resolves the spec through
sys.meta_path, so with the finder already removed the reload reinstalled
the unpatched upstream detector and the Windows ROCm noise returned.
Keep the finder on sys.meta_path permanently, matching the lifecycle of
the existing causal_conv1d and vllm import blockers. The finder matches
a single module name and patching stays idempotent via the sentinel
flags, so repeat hits are no-ops.
Validated on gfx1151 Windows 11: 22 simulation scenarios (conda and
embedded layouts, Program Files scan ordering, paths with spaces and
unicode, hanging probe timeout, lru-wrapped and C-function helper
shapes, reload, failed-import retry, threads, spawn, dormant finder,
Studio PATH coexistence, early fix-block ordering, bnb 0.45.5 / 0.47.0
/ 0.49.2 / main / upstream-fixed) plus the original 12 scenario suite,
CPU-torch and stale-HIP_PATH sandboxes, Python 3.10 to 3.13 gates, and
a WSL Linux leg proving byte-identical Linux behavior with and without
the fix, with and without rocminfo on PATH.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
* fix(studio): prevent UI freeze when switching tabs from heavy pages
Change AnimatePresence mode from "wait" to "popLayout" to fix issue
where switching tabs from Export (or other heavy pages) would cause
the URL to update but the UI to freeze.
With mode="wait", the exit animation must complete before the new
component mounts. If the exiting page has expensive computations,
this blocks the UI. mode="popLayout" allows the new route to mount
immediately while the old one animates out.
Fixes#5850
* fix: add relative positioning for popLayout mode
AnimatePresence mode='popLayout' applies position: absolute to the
exiting element, so the parent container needs position: relative
to prevent layout jumps during transitions.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
The export route's load_checkpoint waits for the training subprocess to
exit by calling time.sleep(0.5) in a loop (up to 30s) inside an async
function. time.sleep blocks the whole event loop, so every other request
to the server stalls for that duration. Use await asyncio.sleep(0.5),
matching the async pattern already used elsewhere in this file
(asyncio.to_thread, await asyncio.sleep).
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
The kwarg-dispatch in unsloth_base_fast_generate recognized input_ids,
input, input_features and the misspelled input_embeds, but not HF's
canonical inputs_embeds. So generate(inputs_embeds=...) fell through to the
'first kwarg' fallback, which picks whatever kwarg happens to come first
(e.g. attention_mask) and uses it as input_ids -- giving the wrong tensor /
batch size, or the KeyError reported in #3082 on older versions.
Add an explicit inputs_embeds branch so embedding inputs (e.g. multimodal
audio+text) are routed correctly regardless of kwarg order.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
* fix(studio): fall back to copy when os.replace is blocked during install activation
On Windows ARM64 the antivirus scanner can transiently hold a freshly
extracted DLL open while MoveFileEx runs, so activating the staged
llama.cpp prebuilt fails with [WinError 5] Access is denied. Attempt
os.replace first, then fall back to a file-by-file copytree which
bypasses the rename.
* address review: keep os.replace for rollback, scope copy-fallback to staging
The copy + rmtree fallback could silently corrupt a live install if the
existing directory is busy. Restrict it to freshly extracted staging
trees (renamed activate_staged_dir) and keep strict os.replace for the
rollback move so a busy active install raises immediately.
* fix(studio): scope copy-fallback to busy-lock errors, log it, and add tests
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
* Studio: enable MTP for sub-3B Gemma separate-drafter GGUFs
The sub-3B auto-drop to ngram-mod was tuned for an embedded draft head
(Qwen), whose per-token cost regresses below 3B. Gemma ships the head as a
separate root mtp-*.gguf drafter, a tiny standalone model that is cheap
enough to win below 3B: B200 Q4_K_XL bench, draft-mtp n=2 vs spec-off,
gemma-4-E2B (2B) = 1.21x (accept ~0.65) while ngram-mod is 1.00x.
Exempt a separate drafter from the sub-3B gate everywhere the threshold is
applied: the resolver (_mtp_too_small), the auto-fit VRAM reserve, the
drafter auto-download decision, and the reload-skip mirror via a
has_separate_drafter flag on _auto_mode_drops_mtp. Embedded sub-3B heads
(Qwen) still drop to ngram-mod. A drafter the binary cannot build (older
prebuilt, or a CUDA kernel limit) still aborts the spawn and the load
retries once without speculative decoding.
Adds the full Qwen3.5 + Gemma-4 (regular and QAT) auto/off/forced resolver
matrix, plus explicit sub-3B exemption tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Always compare the separate drafter in the reload-skip mirror
The sub-3B wrapper around the drafter compare could skip it when the drafter
was deleted out from under a running sub-3B server (detected None, stored set),
leaving a stale launch. The resolved-path compare is cheap and already handles
every case, so drop the _auto_mode_drops_mtp guard (and its now-unused imports)
and always compare when the mode can use a drafter and the user does not own
--spec-type. Addresses review feedback on #6191.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* -Added catch for unsloth running inside of system32 (for windows)
- changed import statement to import os as _os instead of only os.path as _osp
- replaced __osp mention with __os.path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adressed geminis codereview. Did not change Error message as "System32" is precise enough and the _system32 variable is lowercase only
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
* Studio: fetch the release source asset for exact (mix) source builds
The source-build fallback rebuilt the codeload/archive URL from the
source repo and commit. A mix build's merged tree is never pushed to any
repo (it ships only as the release's llama.cpp-source-commit-<sha>.tar.gz
asset), so codeload 404s on the merge commit and an uncovered host could
not build from source. When an exact-source asset exists, fetch it
directly from the release and keep codeload as the fallback for vanilla
builds whose commit is real.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep audio feature extractors right padded when loading processors
FastBaseModel.from_pretrained passes padding_side=left to
AutoProcessor.from_pretrained for generation, and ProcessorMixin forwards
the kwarg to every sub-component, including audio feature extractors.
Stock transformers right-pads audio: frame-validity masks assume trailing
padding. The leaked left padding shifts mel content to the end of the 30s
window for Whisper and gives Gemma 4 one extra valid mel frame on clip
lengths off the hop boundary, desyncing audio features from placeholder
tokens and crashing training on transformers 5.5.0 to 5.9.x with 'Audio
features and audio tokens do not match'.
Reset the feature extractor to right padding at the single processor
finalization point. Text tokenizer padding stays left. Verified on
transformers 5.5.0: the loaded processor now matches a fresh stock
AutoProcessor exactly, and a 9-combination Gemma 4 audio forward probe
goes from 4 failures to none.
* Add explicit None check before reading feature extractor attributes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: ignore unsupported env proxy during Studio startup
* fix: handle missing socksio env proxy at startup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Match printf logging style and inline the proxy predicate for PR #6102
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Installer: harden GPU detection follow-ups after #6174
Ports the NVIDIA-priority and /proc/driver/nvidia/gpus hardening from #6174
to the remaining pathways and adds recovery for already-poisoned venvs:
- install_python_stack.py: add _ensure_cuda_torch so 'unsloth studio update'
force-reinstalls CUDA torch when the venv carries a ROCm build on an NVIDIA
Linux host (the pre-#6174 poisoning signature). Honors UNSLOTH_TORCH_BACKEND,
UNSLOTH_ROCM_TORCH_INSTALLED, and CUDA_VISIBLE_DEVICES=-1/'' opt-outs; never
touches healthy CUDA, deliberate CPU wheels, macOS, or Windows.
- install_llama_prebuilt.py: detect_host gains the /proc NVIDIA fallback and
skips ROCm probes when NVIDIA is usable; forwarded --rocm-gfx/--has-rocm
overrides still win.
- setup.sh: GPU summary classifies NVIDIA first through a timeout-bounded
probe with the /proc fallback; AMD probes are bounded and gain a KFD
vendor_id 4098 fallback; the llama.cpp source build only selects
GGML_CUDA/GGML_HIP when the matching GPU is actually detected.
- install.sh: bound both nvidia-smi calls with a 10s timeout (no behavior
change when healthy or when the timeout binary is absent); classify the
exported UNSLOTH_TORCH_BACKEND on the final index path segment so custom
mirrors containing 'rocm'/'gfx' in their base path are not mislabeled.
- install.ps1 + setup.ps1: NVIDIA probes now require a real 'GPU N:' row from
nvidia-smi -L under a 10s bound instead of bare exit code 0; later CUDA
version and compute_cap queries are bounded too.
Tests: 3 new test files (50+ tests), suite at 788 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Resolve-CudaToolkit driver probe for extracted-function unit test
tests/studio/test_resolve_cuda_toolkit.ps1 extracts Resolve-CudaToolkit alone
into a child pwsh and stubs nvidia-smi with a .ps1 script. The bounded runner
is not in scope there (and ProcessStartInfo cannot dispatch .ps1 stubs), so
the DriverMaxCuda parse silently returned nothing and the major-mismatch
scenarios failed. Fall back to direct invocation when Invoke-NvidiaSmiBounded
is unavailable; production setup.ps1 always has it defined and keeps the
10s bound.
* Treat CUDA_VISIBLE_DEVICES empty or -1 as hidden in NVIDIA-first guards
The NVIDIA-first guards added in this branch only special-cased
CUDA_VISIBLE_DEVICES=-1 at two setup.sh gates and ignored the empty-string
form entirely, while the Python detector (install_llama_prebuilt.py)
already treats both as hidden. On a mixed AMD+NVIDIA host steered to the
AMD card via CUDA_VISIBLE_DEVICES, the guards suppressed the AMD probes,
so setup.sh fell to a CPU llama.cpp build and install.sh picked CUDA
wheels instead of ROCm.
Move the policy into the helpers so every consumer agrees:
- install.sh: new _cvd_hides_nvidia checked first in _has_usable_nvidia_gpu
- studio/setup.sh: same via _setup_cvd_hides_nvidia; the two ad-hoc
CUDA_VISIBLE_DEVICES=-1 gate conditions are now redundant and removed
- studio/install_python_stack.py: _has_usable_nvidia_gpu returns False
when CUDA_VISIBLE_DEVICES is set to or -1 (whitespace tolerated)
Tests: 5 new sh scenarios (hidden via , -1, padded -1, visible device,
and mixed host with hidden NVIDIA restoring the ROCm route) plus a pytest
class covering all three implementations behaviourally.
Addresses the review comment on the NVIDIA-first setup.sh block.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Retrigger CI after PyPI 503 outage during the previous run
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
