Daniel's normalisation commit updated the cap from rocm7.1 to rocm7.2
since PyTorch now publishes that index and rocm7.2 ships torch 2.11.0.
Test expectations were stale.
OOM guard (worker.py):
- Scope to _hw.IS_ROCM only -- NVIDIA CUDA has a graceful OOM path and
does not need the allocator cap
- Detect unified memory by comparing torch VRAM against psutil system RAM;
use 0.80 on unified-memory APUs (gfx1151 Strix Halo) where the GPU pool
is carved from host RAM, 0.90 on discrete cards
Multi-GPU arch selection:
- install.ps1 / setup.ps1: replace -match (first hit only) with
[regex]::Matches() to collect all gcnArchName entries, then index by
HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES
- install_python_stack.py: index into full token list before dedup so
HIP_VISIBLE_DEVICES=2 on [gfx1100, gfx1100, gfx1151] resolves gfx1151
- install.sh: remove awk dedup from gfx token collection for same reason
GCC multiarch (setup.sh):
- Only append -linux-gnu when gcc -print-multiarch does not already return
the full triple, fixing double-suffix on Ubuntu 24.04
* feat: add custom model v1/model loading
* fix: require base URL for local model catalog loading
* ux/studio-provider-model-loading-controls
* fix: normalize local provider base URLs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Move uninstall scripts into scripts/ and fix all references
Relocates `uninstall.sh` and `uninstall.ps1` from the repo root into
the existing `scripts/` directory, alongside the other helper scripts.
Reference fixes:
* `README.md`: Studio uninstall instructions now point at the raw
GitHub URLs under `scripts/`. The previous `unsloth.ai/uninstall.*`
short URLs currently 404 (unlike `unsloth.ai/install.sh`, which
301s to the raw github URL), so the raw URL is the working entry
point until that redirect is configured.
* `scripts/uninstall.sh` header `Usage:` example updated to the new
raw GitHub path.
* `scripts/uninstall.ps1` header `Usage:` example updated to the new
raw GitHub path.
* `.github/workflows/studio-update-smoke.yml`: `paths:` trigger and
round-trip exec/exists checks now use `scripts/uninstall.sh`.
* `.github/workflows/studio-mac-update-smoke.yml`: same.
* `.github/workflows/studio-windows-update-smoke.yml`: `paths:`
trigger and round-trip exec/exists checks now use
`scripts/uninstall.ps1`.
The in-script help hints (e.g. `sh uninstall.sh`, `.\uninstall.ps1`)
are left unchanged because they are user-facing examples shown after
the user already has the file locally, and the basename form works
regardless of which directory the user downloaded the script into.
Follow-up note for unsloth.ai: once this lands, please add the
`unsloth.ai/uninstall.sh` and `unsloth.ai/uninstall.ps1` short-URL
redirects to `raw.githubusercontent.com/unslothai/unsloth/main/scripts/...`
(matching the existing `unsloth.ai/install.sh` redirect pattern).
* Update remaining uninstall script help hints for new scripts/ path
Three user-facing strings inside the uninstall scripts still showed
the old basename form, which became misleading after the move:
* `scripts/uninstall.ps1` header `# Local:` example: now references
`.\scripts\uninstall.ps1` (the actual path from the cloned repo
root).
* `scripts/uninstall.sh` env-var re-run hint: now shows the canonical
curl-pipe form documented in README, since callers who came via
`curl -fsSL ... | sh` never had a local `uninstall.sh` to invoke.
* `scripts/uninstall.ps1` env-var re-run hint: same, switched to the
`irm ... | iex` form documented in README.
Pure string changes, no behavior change.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* feat: add custom model v1/model loading
* fix: require base URL for local model catalog loading
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* studio/ci: harden three pre-existing CI flakes
Three independent fixes to flakes that have been failing on main for
multiple PRs in a row and obscuring real signal.
1. tests/studio/playwright_chat_ui.py:
The theme-toggle x3 block called acct.click() then waited 3s for
[role="menu"] to appear. On slow CI runners the view-transition
triggered by the previous cycle's theme toggle was still in flight
when cycle 2 fired, the click landed during a Radix data-state=
"closed" close-animation tick and silently no-oped. Symptom:
"theme cycle 2: account menu didn't open" at line 963.
Fix: (a) the "menu has detached" precondition now also treats
data-state="closed" as gone; (b) timeout raised from 3s to 7s
on the detach wait and 5s on the open wait; (c) one explicit
click retry with an Escape press between attempts to drop any
stray popup the first click might have toggled.
2. .github/workflows/consolidated-tests-ci.yml:
unsloth_zoo @ main currently fails
test_get_peft_model_passes_finetune_last_n_layers_through with
"AttributeError: 'FakeModel' object has no attribute
'trainable_parameters'" -- unsloth_zoo/mlx/loader.py:2972 added a
model.trainable_parameters() call that the test's fake model
never stubbed. This blocks every unsloth PR's Core CI. Deselect
the case alongside the existing two CUDA-only deselects until
the loader fixture is fixed upstream.
3. .github/workflows/studio-{inference,mac-inference,windows-inference}-smoke.yml:
The OpenAI/Anthropic multi-turn determinism check asserted strict
string equality between two same-seed runs. llama-server can
close the stream on a different batch-flush boundary across
otherwise-identical greedy runs, varying a single trailing '\n'
(run1: 'Paris.\n' vs run2: 'Paris.'). Generated tokens are the
same; only trailing whitespace differs. Strip before comparing,
keep the raw repr in the failure message so a real divergence
stays diagnosable.
* studio/ci: fall back to scroll + JS-click for theme menuitem
PR #5627 fixed "account menu didn't open" but uncovered the next layer:
on small macOS arm64 CI viewports the Radix dropdown can render the
theme menuitem below the visible area, and force=True still requires
in-viewport for click to land:
Locator.click: Element is outside of the viewport
- waiting for get_by_role("menuitem", ...).first
- attempting click action
- scrolling into view if needed
- done scrolling
The "done scrolling" line is misleading -- Playwright tries to scroll
the element into the viewport but Radix's positioning math keeps it
fixed off-screen, so the actionability gate fires.
Three-tier click fallback:
1. force=True click with a 3s budget (current path).
2. scroll_into_view_if_needed() then click.
3. evaluate("el => el.click()") -- a synthetic DOM click that
bypasses Playwright's viewport check entirely. Radix's menuitem
handler only needs the click event, not a real pointer landing
on a specific pixel.
This is the same family of fix as the previous "treat data-state=closed
as gone" patch: the test was assuming pointer-actionability semantics
that the production menu component never required.
On RDNA 4 (gfx1200/gfx1201) and other ROCm GPUs, exhausting VRAM can
cause a HIP driver hang that freezes the entire system rather than
raising a recoverable Python exception.
Two-part fix:
- set_per_process_memory_fraction(0.90) caps the HIP/CUDA allocator at
90% of VRAM so PyTorch raises OutOfMemoryError before hitting the
hardware limit, keeping the driver alive and the system responsive
- top-level exception handler detects OOM errors by type and message
and surfaces a clear actionable message to the UI (reduce
max_seq_length, enable gradient_checkpointing, lower batch size)
instead of the raw CUDA/HIP error string
- main.py: numeric DLL sort (string sort picked rocm72 over rocm713);
add basename() to regex; log warning on detection failure; log info
when BNB_ROCM_VERSION is set (mirrors worker.py)
- worker.py: explicit len-guard in _hip_ver_at_least() with warning
logs instead of silent IndexError/ValueError swallow
- hardware.py: isinstance(result, dict) guard before result.get() in
_smi_query() to prevent AttributeError on non-dict backend returns
- amd.py: round() before int() on parsed GPU IDs; log warning when
truncation occurs (defensive against malformed amd-smi output)
- setup.sh: quote --gcc-install-dir value in CMAKE_HIP_FLAGS so paths
with spaces do not break the CMake argument
- install.ps1, setup.ps1: apply colon-split + ToLower() to hipinfo
gcnArchName match (consistent with each other and with setup.sh)
- install.sh: tighten ROCm tag case patterns to explicit
rocmX.Y|rocmX.Y.* to avoid unintended prefix matches
Reverts PR #5615 to give the safetensors + MLX healing parity work more time to bake before re-merging. The reverted feature branch `studio-tools-multi-format` remains untouched, and the follow-up PR will layer the healing-parity commits on top.
Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe.
* studio/frontend: show Generation stopped placeholder when cancelled mid-thinking
Closes#5563.
When the user clicks Stop before any visible content has streamed in,
the running indicator disappears but no Parts have rendered yet, leaving
just the AssistantActionBar floating below the user prompt. That looks
broken (and is the exact failure mode behind the 'tools work, but I
don't see anything happening' bucket of reports).
Add a sibling CancelledIndicator next to GeneratingIndicator that fires
when content is empty AND status is incomplete with reason cancelled,
rendering a muted 'Generation stopped.' italic. The terminal-state
label is consistent with tool-fallback's existing 'Cancelled tool'
treatment and with reasoning's 'Thought for N seconds' summary.
* studio/frontend: shorten CancelledIndicator comment
Trim the 3-line explanation to a single line describing what the
placeholder is for.
* studio/frontend: use 'Cancelled.' to match tool-fallback wording
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: settings dialog fits viewport at tablet widths
The dialog used a fixed w-[820px] with sm:w-[820px] override, so any
viewport between 640px and 820px (iPad portrait at 768px is the
canonical case) saw the dialog overflow horizontally by 26px on each
side -- the right-edge scroll arrow and the active-tab chevron got
clipped against the viewport.
Replace the hard 820 with min(820px, calc(100vw-2rem)) on both max-w
and w so the dialog caps at the original 820px on desktop and shrinks
to fit (with a 1rem gutter) on narrower screens. max-sm: still drives
the full-bleed h-dvh/w-dvw layout under 640px.
* studio/frontend: keep mobile full-bleed override !important
Bot review: base !max-w-[min(...)] is !important so the regular
max-sm:max-w-none never wins, leaving a 1rem gutter on phones where
the previous code rendered a true full-bleed dialog. Bump the mobile
override to !important too.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
The composer's mic icon buttons used tooltip="Dictate" /
"Stop dictation" but no aria-label, so screen-reader users heard
only the empty SVG-only button. Every other composer icon button
(Send, Add Attachment, audio buttons, composer pills) carries an
explicit aria-label; the shared-composer.tsx implementation already
does too. Mirror that here for parity.
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
The settings dialog opens via a global Ctrl+, keydown handler in
__root.tsx, not via a <DialogTrigger>. Radix's FocusScope tries to
capture document.activeElement at mount as the focus-restore target,
but settings-dialog.tsx schedules a requestAnimationFrame that focuses
the active tab button right after mount, racing FocusScope's previous-
focus capture. On Escape or close-button click, focus then lands on
<body> instead of the textarea (or button, or wherever the user was).
A Playwright focus-management probe confirmed: open dialog, press Tab
15 times (trap holds), press Escape, document.activeElement === BODY.
This is a WCAG 2.4.3 (Focus Order) violation: keyboard-only users
have to re-Tab from the start of the page after every settings visit.
Fix: capture document.activeElement in the Zustand store at the moment
openDialog() runs, then restore via onCloseAutoFocus on DialogContent.
Use opener.isConnected so a stale node from a re-rendered tree falls
back to Radix's default. closeDialog deliberately does NOT clear the
opener slot - onCloseAutoFocus reads it on the render after open=false,
so clearing in the same set() would null it before restoration.
Probe re-run confirms focus restored to the TEXTAREA opener after
Escape, after close-button click, on both repeats. Tab + Shift+Tab
trap still holds (unchanged Radix behaviour).
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: compare composer blocks send when no model picked
Closes the racing-handle half of #5569. In Compare mode (GeneralCompare
shell with model1/model2 props), if the user sends a prompt before
picking models in either pane, the SharedComposer used to fall through
to the per-handle append branch. Both panes then raced
createOpenAIStreamAdapter -> autoLoadSmallestModel, one won, the other
dispatched into an unloaded slot and produced an empty bubble with a
1000000.0 tok/s readout. The per-pane picker state never observed the
global checkpoint change either, so both pickers stayed at
"Select model".
Add a guard before the content build: when handlesRef has model1/model2
keys but both selections are empty, surface a toast asking the user to
pick models first, leave the text in the composer for retry, and never
enter the racing dispatch path. Keeps the per-pane picker state as the
source of truth for which model is on each side.
The unphysical tok/s readout that the same path produced is separately
covered by PR #5570 (display guard).
* studio/frontend: tighten compare-mode guard to require both panes
Review feedback on #5574:
- Gemini: the redundant `model1 !== undefined && model2 !== undefined`
checks let the racing-handle dispatch slip through whenever the
Compare props arrive as undefined, which is the exact case the
guard is trying to block.
- Codex: with `isGeneralizedCompare` keyed on `model1?.id || model2?.id`,
a half-selected Compare (one model picked, one empty) still falls
into the generalized branch. The composer clears, the empty pane
gets the user message appended, and `startRun` only fires for the
side with an id, leaving the empty pane with a dangling prompt
and no response.
Switch `isGeneralizedCompare` to require BOTH panes (`&&`), drop the
undefined gate, and surface the "Pick a model in each pane" toast for
either the fully-empty or half-selected case. `hasCompareHandles` is
true only inside GeneralCompareContent, so LoraCompare and the
single-pane path stay unchanged.
* studio/frontend: shorten compare-mode no-model-guard comment
* studio/frontend: clarify compare-pane toast wording
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: include filename in attachment aria-label and img alt
When a chat has multiple attachments of the same kind, the rendered
tiles all share the generic accessible name "Image attachment" or
"Document attachment". Sighted users get the filename from the Radix
tooltip that pops on hover, but:
- screen-reader users hear "Image attachment, Image attachment,
Image attachment" with no way to distinguish three PNGs;
- touch-device users (no hover) lose the filename entirely;
- keyboard-only users would have to focus and read a tooltip that
isn't always announced.
Fold the filename into both the button's aria-label and the thumbnail
<img alt>, falling back to the existing labels when the attachment has
no filename. Sighted UX is unchanged: the Radix tooltip already shows
the same name on hover, and the visible aria-label has no rendered
counterpart.
Found while running a multi-image attach probe in the autonomous Studio
UX loop (cycle 8). Repro:
await page.evaluate(`Array.from(document.querySelectorAll(
'button[aria-label*="attachment" i]'
)).map(b => b.getAttribute('aria-label'))`)
Before: ["Image attachment", "Document attachment", "Add Attachment"]
After: ["Image attachment: test_red_circle.png",
"Document attachment: notes.txt",
"Add Attachment"]
* studio/frontend: shorten attachment a11y comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: show Loading fallback instead of blank pane on lazy route navigation
Closes#5567.
Train, Recipes and Export pages are imported via React.lazy() in their
respective createRoute calls, and the Suspense boundary around <Outlet />
in __root.tsx passes fallback={null}. The result is a 1-3 second
completely white pane between sidebar click and content paint, which is
the exact failure mode behind reports that those pages look broken or
stuck. /chat does not suffer from this because chat.tsx imports its
ChatPage synchronously.
Replace fallback={null} on both Suspense boundaries (hideNavbar and
sidebar layouts) with a small centered 'Loading...' label using the
same muted-foreground style as elsewhere in the app. Synchronous routes
(/chat) never suspend so they are unaffected; lazy routes now have a
visible terminal-state placeholder while their chunk loads.
* studio/frontend: also apply RouteFallback to the sidebar Suspense
The first revision only replaced the fallback={null} inside the
hideNavbar branch (used for onboarding / login). The primary lazy
boundary that wraps Train / Recipes / Export is inside the SidebarInset
branch at the other Suspense site, which kept rendering null and made
the page look stuck for the same window the original bug describes
(per bot review feedback on #5568).
Replace both Suspense fallbacks with RouteFallback so the "Loading..."
placeholder fires on every lazy route, not just on the auth flows.
* studio/frontend: shorten RouteFallback comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: keep theme classes mutually exclusive on <html>
The Sonner Toaster reads next-themes (mounted at provider.tsx with
attribute="class" defaultTheme="light"), so on first mount next-themes
adds a "light" class to <html>. Studio's own setTheme path
(features/settings/stores/theme-store.ts) only toggled "dark", so
after the user picked Dark in settings the document ended up with
html.className = "light dark". Harmless in CSS cascade because the
dark variables override, but reads as a UI defect in devtools and trips
CSS-aware tooling that branches on class lists.
Toggle "light" alongside "dark" in applyToDocument so the two classes
stay mutually exclusive regardless of how next-themes seeded the
initial class.
* studio/frontend: shorten theme-toggle comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/web: distinguish "offline" from "studio crashed" in error toast
When the user's browser loses network mid-request, authFetch caught the
fetch TypeError and surfaced "Studio isn't running -- please relaunch it."
That is a correct diagnosis in the Tauri desktop app (the supervisor died
in-process), but it is a misleading diagnosis in the web build where the
backend lives elsewhere: the user will start hunting for a dead process
when the actual problem is connectivity.
Branch on navigator.onLine === false (web build only) and surface
"You appear to be offline. Check your network connection and try again."
instead. Tauri keeps the original wording so it stays accurate there.
Found while running a slow-network UX probe and toggling
Network.emulateNetworkConditions {offline: true} mid-stream.
* studio/frontend: shorten offline-error wording comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: guard message-timing badge against unphysical tok/s
llama.cpp can report `predicted_ms == 0` and `predicted_n == 0` on turns
that effectively produced no generation (most reliably reproduced today
on a Compare-mode pane that loses the auto-load race and dispatches a
generate against an unloaded slot, see issue #5569). The current display
trusts `predicted_per_second` verbatim, which turns into `Infinity` /
`1000000.0 tok/s` on the action toolbar of an otherwise empty bubble
and reads like a UI defect even when the underlying request did happen.
Require at least one predicted token, at least one millisecond of
generation time, and a finite rate before rendering. Falls back to the
total stream time formatter, which already handles the zero case
gracefully.
* studio/frontend: shorten predictedRate guard comment
* studio/frontend: tighten timing guard threshold and hide Generation row when suppressed
Raise the decode-window floor from 1ms to 10ms so race-lost panes that
emit a stray token in 1-2ms (still giving 1000-5000 tok/s) drop out
alongside the predicted_ms=0 case. Gate the tooltip's Generation row
on the same hasPredicted predicate as Speed so the tooltip never shows
'Generation: 0ms' with no Speed underneath.
* studio/frontend: accept sub-10ms decode windows in timing guard
Cycle-15 codex P2 flagged that the >= 10ms threshold hid legitimate
fast generation (cached single-token, small models). The original
Infinity-blocker was predicted_ms=0, so use >0 instead. predicted_n
>= 1 and Number.isFinite() still keep the no-op race-lost cases out.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio: respect prefers-reduced-motion across animations
Tailwind animate-in/out, Radix dialog/popover zoom-in/slide-in transforms,
and the infinite shine / shiny-text / icon-pop keyframes all run at their
full duration regardless of the user's OS-level reduced-motion preference.
A Playwright probe that emulated the media query confirmed every measured
transition was identical between no-preference and reduce, so users with
vestibular triggers see the same scaling overlays and continuous shimmers.
Add the canonical universal-selector override so animation-duration,
animation-iteration-count, and transition-duration collapse to ~0ms when
the preference is set, leaving end states intact. Probe re-run shows
settings-dialog animationDuration drop from 0.1s to 1e-05s and the 50ms
mid-open screenshot is byte-identical to the settled one.
* studio: exempt .animate-spin from reduced-motion collapse
The universal-selector rule from the previous commit froze every
animation including .animate-spin, which is used as the canonical
in-progress indicator across Studio: tool execution loaders
(tool-ui-python/terminal/web-search/code-execution/fallback/group),
sonner toast spinners, Tauri startup + update screens, and the
generic <Spinner /> primitive in components/ui/spinner.tsx.
Freezing those leaves reduced-motion users with no visual signal
that work is in flight, which trades one accessibility win for
another. WCAG treats progress indicators as "essential motion"
that should keep moving.
Restore .animate-spin with a 1.5s cadence (instead of the default
1s) so the rotation is still perceptible but less aggressive than
the no-preference path. animation-iteration-count goes back to
`infinite` so the spinner doesn't halt after one rotation.
Verified via a focused probe that injects a .animate-spin element
and a .animate-in fade element side by side:
no-preference spin=1s infinite fade=0.15s
reduce spin=1.5s infinite fade=1e-05s
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
The settings dialog sidebar was fixed at w-[200px], which left only
~92px of horizontal space for tab labels after icon, gap, and the
'New' badge for Connections/API. 'Connections' (11 chars at the
14.5px font weight medium) overflowed and rendered as 'Connectio...',
matching the paper-cut reported in issue #5572.
Bump the sidebar to w-[216px] -- 16 more pixels of label space, fully
within the existing dialog width and unchanged on mobile
(max-sm:w-full still drives the responsive layout).
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Adds tools, thinking blocks, code execution, and web search support to the safetensors / transformers and MLX inference backends in Studio, bringing them to parity with the GGUF path.
What ships
- safetensors / transformers agentic tool loop with cumulative-text state machine, tool-call XML parser, and template kwarg forwarding (tools / enable_thinking / reasoning_effort / preserve_thinking).
- MLX backend: same kwargs accepted on Apple Silicon; chat_template_info shipped through worker IPC; pills enable for Qwen / Qwen3 / Qwen3.5 / Gemma reasoning.
- Capability classifier (_detect_safetensors_features) gates supports_tools on actual parser-compatible emission markers (<tool_call> / <function=) so Llama-3 / Mistral / Gemma 4 do not advertise toggles the parser cannot honour.
- gpt-oss override stays: reasoning on, tools off (Harmony channel, not <tool_call> XML).
- CWE-209 hygiene: safetensors SSE error path emits a constant message and logs the trace server-side.
Validation
- 256 unit tests green (43 tool-loop, 11 capability advertise, 7 MLX backend, 5 main-added, 190 adjacent inference / anthropic / openai regression).
- Cross-OS staging CI green on ubuntu-latest / macos-14 / windows-latest plus a dedicated MLX cartesian probe against real unsloth/Qwen3.5-0.8B on macos-14 (CI 26098107440).
- Capability parity verified across Qwen3 / Qwen3.5 / Llama-3 / Mistral / Gemma / DeepSeek-R1 / gpt-oss (incl. BF16).
- Manual confirmation from Imagineer99 on Qwen3.5-2B: think + search + code exec working.
Closes the safetensors / MLX gap with the GGUF backend.
* studio: reserve VRAM headroom for the MTP draft cache in auto-fit
When MTP is going to engage on this load, _fit_context_to_vram now
budgets 0.85 of available VRAM instead of 0.90, leaving room for
llama.cpp's secondary MTP draft KV cache + compute graph buffers.
Motivation: a user report on RTX 5090 (32 GB) showed Qwen3.6-27B-MTP-GGUF
UD-Q4_K_XL at native auto-context running roughly half the speed of
the same model with a slightly smaller context. The most parsimonious
explanation is a VRAM cliff: at native context the target's KV
already eats the 90% budget, then llama-server allocates the draft
cache + draft graph on top and spills into a slower partial-offload
path. Reducing the budget by 5% on MTP loads avoids the spill without
penalising non-MTP loads. On hardware with abundant VRAM (B200, etc.)
the fit is unchanged because the requested context already fits in
the tighter budget too.
MTP detection mirrors the auto-promotion logic in load_model: the
GGUF advertises nextn_predict_layers, or the model identifier /
local path matches the -MTP marker, and the user has not explicitly
opted out via speculative_type="off" or --spec-type extra args.
Tests: two new cases in test_kv_cache_estimation.py verify that
mtp_engaged=True yields a context less-than-or-equal-to the
non-MTP path on a tight budget, and that kv_on_gpu=False still
short-circuits regardless of mtp_engaged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: gate _mtp_will_engage on canonical-mode resolver
After PR #5582 introduced the 5-mode Speculative Decoding dropdown plus
_canonicalize_spec_mode, the auto-fit MTP-engaged predicate becomes:
* forced mtp / mtp+ngram -> always engage MTP (extra VRAM needed)
* auto + MTP GGUF (>= 3B) -> engages MTP via auto-promotion
* auto + MTP GGUF (sub-3B) -> falls back to ngram-mod (no extra VRAM)
* ngram / ngram-simple / off -> never engage MTP
* user --spec-type in extra_args -> resolver suppressed; no headroom
The old gate triggered on "anything but off", so it over-reserved the
0.85 budget when the user explicitly picked Ngram (no MTP) or when
Auto fell back to ngram-mod on a sub-3B MTP model. The 5% headroom
cost was minor but unnecessary.
Mirrors the same logic already encoded in _build_speculative_flags so
the auto-fit budget and the actual emission agree on whether MTP is
running.
All 361 backend tests pass.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: add --spec-draft-n-max toggle for MTP speculative decoding
Surface llama-server's --spec-draft-n-max as a first-class
LoadRequest field so users can tune the MTP draft tree size from
the chat settings panel. Default behaviour is unchanged: when the
caller omits spec_draft_n_max, the existing platform defaults still
apply (6 on GPU, 3 on CPU/Mac).
Why this matters: on context-constrained loads the draft KV cache
competes with the target model's KV cache for VRAM. Lowering
spec_draft_n_max reduces that pressure, lets a larger user context
fit, and recovers throughput; raising it pays off when draft
acceptance is high enough to amortise the extra cache.
Backend
- LoadRequest gains an optional spec_draft_n_max: int (1..16).
- LlamaCppBackend.load_model accepts and persists the override on
self._spec_draft_n_max, used in place of the hardcoded 6/3 in the
MTP emit branch.
- LoadResponse and InferenceStatusResponse echo the active value
(None when the platform default is in effect) so the UI can
hydrate the input on refresh.
- _already_in_target_state and _request_matches_loaded_settings
compare spec_draft_n_max alongside speculative_type so a value
change triggers a reload rather than no-op'ing.
- strip_shadowing_flags now strips inherited --spec-* extras when
either speculative_type or spec_draft_n_max is in fields_set, so
an inherited --spec-draft-n-max cannot last-wins-override a fresh
request's first-class field.
Frontend
- LoadModelRequest, LoadModelResponse, InferenceStatusResponse
TypeScript shapes get spec_draft_n_max.
- chat-runtime-store gains specDraftNMax / loadedSpecDraftNMax and
a setter, hydrated from /v1/status and /v1/load.
- chat-settings-sheet renders a "Draft Tokens" numeric input
directly under the Speculative Decoding switch when that switch
is on. Toggling the switch off clears the override; the Reset
button restores the loaded value.
Tests
- Four new regression tests cover _already_in_target_state with
matching / mismatching / non-MTP / unset spec_draft_n_max.
- Existing test_llama_server_args.py and test_llama_cpp_mtp_detection.py
green: 141 passed locally.
* studio: add --spec-draft-p-min and --spec-draft-p-split to spec strip set
llama.cpp server documents --spec-draft-p-min (default 0.75, min draft
acceptance probability) and --spec-draft-p-split (default 0.10). Both
are first-class spec-decoding knobs that should travel with the rest
of the --spec-* family when an Apply re-sets speculative_type, so an
inherited override doesn't leak across a fresh load.
* studio/tests: skip MTP capability-probe tests on Windows
The four probe_server_capabilities tests use a bash stub written to
tmp_path/llama-server, which Windows' subprocess can't execute
directly (no shebang resolution, .bat / .cmd would be needed). Mark
them skipif sys.platform == 'win32' so the rest of the MTP plumbing
suite stays green on Windows CI. Unix coverage is unchanged.
* studio: lower MTP GPU default --spec-draft-n-max from 6 to 2
Bench on B200 / Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL across five prompt
types (essay, code, story, math, science) with greedy temp=0:
prompt OFF n=1 n=2 n=3 n=6
essay 79.1 93.4 93.8 84.7 64.6
code 79.1 104.4 116.6 113.5 103.0
story 79.1 99.2 105.7 101.8 88.9
math 79.1 100.8 110.8 111.8 98.2
science 79.1 100.1 110.8 110.8 102.9
The previous hardcoded GPU default of 6 was 17% SLOWER than spec-off
on the essay prompt (64.6 vs 79.1 t/s) and 11-50% slower than n=2 on
the rest. n=2 wins on 4/5 prompts with a 1.18x-1.47x speedup vs OFF;
n=3 wins on the math prompt by a hair. n=6 collapses once acceptance
rate drops past n=3 -- wasted draft decode dominates the per-step
budget.
Matches the dataset README ("n_max=2 is the sweet spot for 36 of 42
quants"). Keeps CPU/Mac default at 3, which empirically tracks the
narrower ngram+MTP chained budget on those platforms.
Users who want the old behaviour can pass spec_draft_n_max in
LoadRequest (the toggle this PR also adds) or --spec-draft-n-max via
llama_extra_args.
* studio: skip MTP auto-promote on sub-2B models, backfill chat usage
Two MTP-visibility fixes uncovered while bisecting llama.cpp post-#22673
on Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL on B200.
Size gate. Direct llama-server bench (no Studio measurement loop) at
n_predict=192 across 9 prompts shows MTP regresses vs spec-off on
sub-2B dense models because draft cost exceeds savings:
Qwen3.5-0.8B Q4_K_XL GPU: 452.0 OFF -> 283.4 t/s n=2 (0.63x)
CPU: 84.5 OFF -> 64.9 t/s n=3 (0.77x)
Qwen3.5-4B Q4_K_XL GPU: 241.0 OFF -> 258.2 t/s n=2 (1.07x)
Qwen3.5-9B Q4_K_XL GPU: 201.6 OFF -> 228.9 t/s n=2 (1.14x)
Qwen3.5-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-35B-A3B Q4 GPU: 192.3 OFF -> 223.2 t/s n=2 (1.16x)
The 2B inflection is sharp. Skip auto-promote to draft-mtp when the
identifier reports <2.0B params; users can still force via --spec-type
or the Speculative Decoding toggle. Mirror the gate in the
reload-skip check so a sub-2B reload-with-default does not bounce a
spec-off backend.
Chat-completions usage. llama-server's final SSE chunk emits both an
OpenAI-style usage block and a custom timings block. timings.predicted_n
is always populated, but usage.completion_tokens is zero on some
server builds. The Studio chat UI computes generation t/s from
meta.usage.completion_tokens / totalStreamTime, so a zero
completion_tokens makes the UI fall back to wall-clock time
(including SSE / proxy / template overhead) which dilutes MTP gains and
makes ON look the same as OFF.
Add _backfill_usage_from_timings: if usage.completion_tokens is missing
or zero AND timings has predicted_n/prompt_n, synthesize a complete
usage dict. Apply at the streaming metadata yield in
generate_chat_completion and at the three accumulator/yield sites in
generate_chat_completion_with_tools so per-iteration counts are not
silently lost across tool calls.
Tests cover both the gate (sub-2B skips, 2B+ promotes) and the
backfill (zero usage filled, real usage preserved, empty timings
passthrough).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe + emit legacy ngram-mod flags for pre-rename llama-server
llama.cpp upstream renamed the ngram-mod tuning knobs:
--draft-max -> --spec-ngram-mod-n-max (and --spec-draft-n-max)
--draft-min -> --spec-ngram-mod-n-min (and --spec-draft-n-min)
--spec-ngram-size-n -> --spec-ngram-mod-n-match
The new names are real flags on post-rename builds and stub removal
entries on the same builds (with description "argument has been
removed"). Pre-rename builds only carry the legacy names as real
flags. Studio was emitting the new names unconditionally, so a user
running a pre-rename llama-server (e.g. an older prebuilt or a
hand-installed binary) would see "unknown argument" errors when the
ngram-mod path engages, or silent drop of the ngram knobs.
Extend `probe_server_capabilities` to parse the help text into
per-flag description blocks and tell real flags apart from removal
stubs by the "argument has been removed" marker. Add three new probe
fields: `ngram_mod_flavor` ("new" / "legacy" / None),
`supports_ngram_mod`, and `spec_draft_n_max_flag` (the actual n_max
flag the binary accepts). Cached by (path, mtime) the same way as
`mtp_token`.
Add `_build_ngram_mod_flags(caps, ...)` that picks the right flag
set, returning [] when neither is usable so callers can drop ngram
chaining entirely on minimal binaries.
Wire both call sites to use the probe-driven flag set:
- CPU/Mac MTP comma-chain (--spec-type ngram-mod,draft-mtp) emits
legacy or new knobs as appropriate. If neither set is available,
degrade to MTP-only (warn but still engage spec).
- Standalone --spec-type ngram-mod branch uses the same helper.
Tests cover post-rename detection, legacy detection, removal-stub
discrimination, minimal-binary case, and all three branches of
`_build_ngram_mod_flags` plus custom n_match/n_min/n_max values.
Verified against three real binaries (Studio bundled 726704a, my
build of 45b455e HEAD, and the MTP merge baseline 2555826) all
correctly reporting ngram_mod_flavor=new.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: sub-3B MTP falls back to ngram-mod, not off
Earlier sub-2B gate disabled speculative decoding entirely for tiny
dense MTP models because the MTP draft head's per-token cost exceeds
the acceptance savings at that scale. The "fully off" fallback was
conservative -- ngram-mod has near-zero idle cost on diverse content
and consistently outperforms both off and draft-mtp at sub-3B.
Clean-methodology bench (each of 9 distinct prompts run once after
two unrelated warmup prompts so the ngram-mod hash pool is
realistically populated but never holds the exact deterministic
output we're about to measure):
Q4_K_XL on B200:
0.8B OFF=451 draft-mtp n=2=263 (0.58x) ngram-only=498 (1.10x)
2B OFF=377 draft-mtp n=2=308 (0.82x) ngram-only=369 (1.00x)
4B OFF=240 draft-mtp n=2=260 (1.08x) -- 4B+ wins with MTP
Q4_K_XL on x86 48 cores:
0.8B OFF= 80 chained n=2= 69 (0.86x) ngram-only= 95 (1.19x)
2B OFF= 62 chained n=2= 51 (0.83x) ngram-only= 63 (1.01x)
4B OFF= 31 chained n=2= 41 (1.33x)
Change:
- Raise the MTP-skip threshold from 2.0B to 3.0B (2B falls below it).
- When skipping the MTP head, fall back to --spec-type ngram-mod via
the probe-driven _build_ngram_mod_flags helper. Works on both
post-rename and pre-rename llama-server builds.
- If the binary advertises neither ngram-mod flavor, fall back to
spec-off (older binaries that don't support ngram-mod at all).
- Mirror the same fallback in _already_in_target_state so a sub-3B
reload-with-default does not bounce a ngram-mod backend.
Tests updated: monkeypatch probe_server_capabilities so the gate
behavior is deterministic regardless of which llama-server happens
to be on the host. +1 new test for the "binary has no ngram-mod
support" branch; renamed prior 2B/0.8B tests to reflect new semantics.
This generalizes the size gate to be probe-driven instead of a hard
"disable spec" branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: 5-mode Speculative Decoding dropdown (Auto / MTP / Ngram / MTP+Ngram / Off)
Replace the Chat Settings Speculative Decoding on/off Switch with a 5-option
Select. Auto preserves today's platform-aware resolver (MTP on MTP GGUFs,
ngram-mod fallback for sub-3B, --spec-default for non-MTP). The other 3 modes
force the user's choice on BOTH GPU and CPU: MTP emits draft-mtp only (no
ngram chain on CPU), Ngram emits ngram-mod only, MTP+Ngram emits the
ngram-mod,draft-mtp chain on both platforms. Off is the existing fully-off
state, kept so the Switch's "disable" capability isn't lost.
Backend
- New module-level _canonicalize_spec_mode(value) maps any accepted input
(canonical, legacy "default" / "draft-mtp" / "ngram-mod" / "ngram-simple",
or comma-chained "ngram-mod,draft-mtp") onto one of auto / mtp / ngram /
mtp+ngram / off / ngram-simple / None. Lets external callers and old
persisted UI state round-trip without breaking.
- LlamaCppBackend grows a _requested_spec_mode field + requested_spec_mode
property storing the canonical UI mode the user requested. Status
responses round-trip this instead of the resolved internal flag, so the
dropdown restores the picked value after reload / refresh (Auto on a 27B
MTP GGUF resolves to draft-mtp internally but the dropdown stays on
"Auto").
- The resolver block in load_model is extracted into a unit-testable
_build_speculative_flags method. Forced MTP / MTP+Ngram on a sub-3B or
non-MTP GGUF logs a warning and engages anyway (user override > the
Auto-path sub-3B fallback).
- _already_in_target_state and routes/inference._request_matches_loaded_settings
now compare canonical-requested mode, dropping the old auto-promotion
mirror. spec_draft_n_max still gates on the resolved spec so Auto + a
changed n_max still bounces a reload.
Frontend
- chat-settings-sheet.tsx: Switch swapped for Select modeled on the KV
Cache Dtype Select. Items: Auto / MTP / Ngram / MTP+Ngram / Off. Draft
Tokens input only visible when speculativeType is "mtp" or "mtp+ngram".
- chat-runtime-store.ts: initial value flips from "default" to "auto".
- use-chat-model-runtime.ts normalizeSpeculativeType mirrors the backend
canonicaliser so persisted "default" / "draft-mtp" / "ngram-mod" / chain
values hydrate to the right dropdown option.
- types/api.ts: docs the canonical wire vocabulary.
Tests
- 53 new assertions in test_llama_cpp_mtp_detection.py: full
_canonicalize_spec_mode table, a 23-row resolver matrix across
(requested mode) x (GPU/CPU) x (model size class), plus n_max override,
user-extra-args precedence, requested-mode round-trip, and graceful
degrade on an outdated llama-server without an MTP token.
- 165 existing backend tests still green. 218 total in the MTP /
server-args / reload-inheritance suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reset Speculative Decoding to Auto on model switch
When the user switches from model A to a different model B, clear the
runtime store's speculativeType + specDraftNMax (and their loaded*
shadows). The new load request then carries null, the backend
canonicalises that to "auto", and its platform-aware resolver runs
fresh for the new model.
Without this, a non-MTP model loaded with "Off" carried the Off choice
into a subsequent MTP load, suppressing MTP auto-promotion (and the
sub-3B ngram-mod fallback) until the user manually opened settings and
flipped the dropdown back to Auto. The clean-sweep deep probe caught
it as anomaly A-1.
The reset only fires when currentCheckpoint != modelId, so a
same-model reapply or forceReload still honours the user's current
spec choice. End-to-end probe on Qwen3.5-4B-GGUF (non-MTP, Off) ->
Qwen3.5-0.8B-MTP confirms: dropdown shows Auto, /api/inference/status
returns speculative_type=auto, studio.log shows the Auto sub-3B
fallback emitted --spec-type ngram-mod.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(peft): expose finetune_last_n_layers for parity with mlx-lm CLI
mlx-lm's lora CLI defaults `CONFIG_DEFAULTS['num_layers']=16`
(mlx_lm/lora.py:56), so it applies LoRA only to the LAST 16
transformer blocks. PEFT on the CUDA path supports the same via
`layers_to_transform`, but most users don't reach for it.
This commit adds a `finetune_last_n_layers` convenience parameter
to both `FastLlamaModel.get_peft_model` and
`FastBaseModel.get_peft_model` (vision/multi-modal). When set, it
fills `layers_to_transform` automatically with the last N blocks,
mirroring mlx-lm CLI's behavior AND
`unsloth_zoo.mlx.loader.FastMLXModel.get_peft_model`. A single
config value now controls layer-selection consistently across
CUDA, MLX (zoo), and mlx-lm CLI paths.
Default is None (= train all layers, current behavior unchanged).
When set, the value is clamped to [1, total_transformer_layers]
so callers can't accidentally over- or under-select. The total
is read from `config.num_hidden_layers` (or aliases), falling
through to `config.text_config.num_hidden_layers` for VLMs.
Why this matters: with the same fixture/seed, training the last
N layers vs all layers picks a different basin under stochastic
LoRA init. Empirically (n=15 seeds, gemma-3-270m-it single-row
LoRA memorization, MLX path) last-16 hits 67% greedy-decode
pass rate vs all-18 at 47%. The teacher-forced completion loss
is 0 in both — the model memorizes either way; only the first-
token argmax distribution differs. CUDA fp32 shows the same
pattern. Aligning the layer selection puts CUDA + MLX + mlx-lm
all in the same basin family for parity comparisons.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* peft: trim verbose finetune_last_n_layers comments
Per code-comment policy: parameter name is self-documenting, the clamp
and range() construction are obvious. Rationale (mlx-lm CLI parity,
empirical pass-rate data) lives in commit 106c1df4's message and the
PR description.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* loader: import FORCE_FLOAT32 from unsloth_zoo (single source of truth)
unsloth_zoo now owns the FORCE_FLOAT32 list in
unsloth_zoo/model_lists.py (re-exported as the top-level
`unsloth_zoo.FORCE_FLOAT32`). The CUDA loader here imports from there
so the MLX loader (unsloth_zoo.mlx.loader) and the CUDA loader stay in
sync from a single edit, and the bf16->fp16 downcast warning added in
unsloth-zoo PR #670 gates on the same list.
Companion to unsloth-zoo PR #670.
* loader: add inline FORCE_FLOAT32 fallback for old unsloth_zoo installs
If a user upgrades unsloth without upgrading unsloth_zoo, the previously
unconditional `from unsloth_zoo import FORCE_FLOAT32` would raise
ImportError at module import time, killing the whole package. Wrap the
import in try/except and fall back to an inline list that mirrors
unsloth_zoo.model_lists.FORCE_FLOAT32 byte-for-byte, so the module
loads cleanly on any zoo version while still preferring zoo as the
single source of truth when present.
* ci: add advisory lockfile supply-chain audit
Adds a fast, focused workflow that scans every checked-in npm and
cargo lockfile on PRs touching one. Default behaviour is advisory:
only public indicator-of-compromise strings, versions on the public
known-malicious list, and structurally broken lockfiles fail the
build. Structural anomalies (missing integrity hashes, non-default
registry, etc.) surface as :⚠️: annotations without gating
merges, so reviewers see the audit result inline on every PR
without changing the existing install behaviour.
Also commits the two missing npm lockfiles the audit needs:
studio/package-lock.json (Tauri CLI holder for desktop release)
and studio/backend/core/data_recipe/oxc-validator/package-lock.json
(oxc-parser runtime for the data-recipe validator). studio/setup.sh,
studio/setup.ps1, build.sh, and pyproject.toml are intentionally
left alone so the existing install path keeps working unchanged.
Audit script behaviour:
default mode -> exits 1 only on blocked-known-malicious,
known-ioc-string, malformed-lockfile,
missing-lockfile, unreadable-lockfile, or
missing-toml-parser
--strict -> promotes every finding to blocking (opt-in)
Adds a try/except around lockfile reads so a permissions error
prints a finding instead of crashing CI with a raw traceback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test(security): update cargo regression test for advisory mode
`scripts/lockfile_supply_chain_audit.py` now classifies
`non-registry-cargo-source` as an advisory finding by default
(returns exit 0 with a `:⚠️:` annotation) rather than
unconditionally blocking with exit 1. Update the existing
`test_malicious_cargo_lockfile_refused` to pass --strict so it
keeps verifying the "refuse to install" behavior it is named for,
and add a second test that pins the default-mode behavior:
advisory finding emitted, exit code 0.
* audit: escape Finding for GH Actions annotations
`:⚠️:` and `::error::` workflow commands truncate the
annotation message at the first newline unless the message is
%-encoded per the workflow-commands spec. Since `Finding.__str__`
returns three lines (kind+path, package, detail), the package
and detail fields were being dropped from the GitHub Actions UI.
Add a `_gha_escape()` helper that applies the spec'd escapes
(`%` -> `%25`, then `\r` -> `%0D`, then `\n` -> `%0A`; the `%`
replacement must happen first so the subsequent escapes are not
double-encoded), wrap every Finding rendered into a workflow
command with it, and pin both the helper and the end-to-end
single-line emission with two new regression tests.
Caught by gemini-code-assist on PR #5604.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: regenerate desktop launcher on `unsloth studio update`
Today `unsloth studio update` only mutates the venv. The macOS .app bundle,
the Linux .desktop file, and the shared launch-studio.sh stub bake their
paths and `studio_install_id` at install time and never refresh. Users who
update an existing Studio install report the Dock / Applications icon still
pointing at the old launcher; only a fresh `curl ... install.sh | sh`
fixes it because that path re-enters install.sh's create_studio_shortcuts.
Wire the same logic into the update path:
- install.sh: add --shortcuts-only. Skips the heavy install steps, resolves
STUDIO_HOME / OS / DATA_DIR through the existing _resolve_studio_destinations
+ platform detection, then calls create_studio_shortcuts and exits.
- unsloth_cli/commands/studio.py: after setup.sh succeeds, call install.sh
with --shortcuts-only. Prefers a local checkout's install.sh (when
STUDIO_LOCAL_REPO is set) or one shipped under _PACKAGE_ROOT, and falls
back to fetching the upstream installer from https://unsloth.ai/install.sh
for PyPI-installed users (the wheel does not ship install.sh).
Net effect: `unsloth studio update` now refreshes the macOS .app stub,
launcher script, studio.conf, and Linux .desktop entry on every update, so
the desktop icon stays in sync with the venv that setup.sh just updated.
Env-override and Tauri modes keep their existing behavior (no persistent
menu shortcuts, but the launch-studio.sh is still regenerated).
Windows is unchanged here; setup.ps1 already handles its own Start Menu /
Desktop .lnk creation on update.
* studio: also regenerate Windows .lnk shortcuts on update
Mirror the macOS fix: install.ps1 gains --shortcuts-only that short-circuits
to New-StudioShortcuts, and unsloth studio update calls it after setup.ps1
the same way it now does on macOS / Linux.
PyPI installs do not ship install.ps1, so the Python helper fetches the
upstream script from https://unsloth.ai/install.ps1 and pipes it into
powershell.exe -Command - with an explicit Install-UnslothStudio call
appended (irm | iex relies on the trailing @args, which is empty when
launched from stdin).
setup.ps1 alone never recreates the Start Menu / Desktop .lnk targets or
the launch-studio.{ps1,vbs} scripts, so without this update users on
Windows hit the same stale-icon regression that triggered the macOS PR.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: rename unsloth.exe to .deleteme before update on Windows
Pip's editable reinstall calls uninstall first, which deletes every RECORD
entry. unsloth.exe is one of them, and Windows refuses to delete a file
whose image is mapped into the running process tree. The first
unsloth studio update after install therefore fails with:
OSError: [WinError 32] The process cannot access the file because it
is being used by another process: ...\Scripts\unsloth.exe
Windows does allow renaming an in-use exe, so move it aside before
_run_setup_script kicks pip. pip then drops a fresh unsloth.exe at the
original path; the *.exe.deleteme left behind is cleaned up at the start
of the next update once the previous shim has exited.
* studio: rename unsloth.exe from setup.ps1 to reliably bypass exe lock
* studio: print python -m workaround when Windows exe lock blocks update
* studio: use python -c hint (unsloth_cli has no __main__)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* install.sh: reshape --shortcuts-only Tauri guard to pass exit-order test
* shorter comments in update / launcher regen logic
* studio update: env-mode passthrough + non-silent shortcuts-only error
* studio update: address codex/gemini PR review
- Strip install.ps1's `Install-UnslothStudio @args` auto-invoke before
appending an explicit `--shortcuts-only` call so PyPI Windows installs
don't re-run the full installer over stdin.
- subprocess.run(input=wrapper, ...) now uses encoding="utf-8" so box
drawing chars in install.ps1 don't UnicodeEncodeError on CP1252.
- Wrap _run_setup_script in try/except to restore unsloth.exe from
.deleteme if setup fails, and mirror that rollback inside setup.ps1
when install_python_stack.py exits non-zero.
- Capture subprocess return codes in _refresh_desktop_shortcuts and
echo a one-line warning on non-zero so silent stale-shortcut failures
surface.
- Drop --local from the Windows lock-recovery hint so users on PyPI
installs don't accidentally switch into editable-checkout mode.
- Quote $VENV_ABS_BIN/unsloth in the install.sh shortcuts-only error
so paths with spaces print legibly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio update: harden Windows refresh per multi-reviewer pass
- PowerShell stdin path now writes the wrapper to a UTF-8 BOM tempfile
and runs it via `-File`. `powershell.exe -Command -` decodes stdin
with the OEM code page, which mangles box-drawing chars in the
fetched install.ps1; -File reads the BOM and decodes UTF-8 cleanly.
- _restore_self_exe_lock_windows now treats a zero-byte unsloth.exe as
a partial-write and prefers the .deleteme copy. setup.ps1 mirrors
the same check.
- _release_self_exe_lock_windows uses os.replace for atomic overwrite
so a stale .deleteme from an aborted prior update doesn't break the
rename.
- Lock-recovery hint mentions that --local should be re-added when
the user installed from a repo checkout.
* studio update: respect Tauri context and tidy Windows .deleteme
Tauri's update.rs spawns `unsloth studio update`; without a signal,
the CLI's _refresh_desktop_shortcuts would call install.{sh,ps1}
--shortcuts-only and create duplicate ~/Applications/Unsloth Studio.app
(or .desktop / .lnk) entries that collide with the Tauri bundle.
- update.rs now sets UNSLOTH_TAURI_UPDATE=1 on the spawned child.
- studio.py's update() skips _refresh_desktop_shortcuts when that env
var is set; Tauri owns its own bundle entries.
- After a successful Windows update, drop the .deleteme orphan so
repeated updates don't accumulate stale binaries that could later
be promoted by _restore_self_exe_lock_windows on a cross-version
failure.
- Tempfile for the PyPI-fallback PowerShell path now uses an
unsloth-studio-refresh- prefix so AV/EDR rules and user greps can
identify it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio update: drop obsolete WinError 32 hint, echo Tauri skip
The rename trick in _release_self_exe_lock_windows + setup.ps1's
restore now handle the .exe-lock case in-flow; the printed hint
suggested re-running update via venv python, but that just re-enters
the same update() and hits the same failure if the rename didn't help.
Removing the misleading hint and its helper.
Also surface a one-line typer.echo when refresh is skipped under
UNSLOTH_TAURI_UPDATE so --verbose logs make the branch visible.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: cap auto-load cascade attempts
autoLoadSmallestModel walks every cached GGUF and safetensors repo with a
try/catch + continue, so a folder of broken caches (missing files, stale
llama.cpp prebuilt, GPU OOM) can fire dozens of failing POST /api/inference/load
calls in a row. Each call costs ~5 seconds (HF metadata probe + DNS guard
inside inference.py), so the user sees a runaway sequence of request_completed
log lines after sending one message that needed an auto-load.
Cap the total loadModel calls inside autoLoadSmallestModel at 3 (GGUF cascade
plus safetensors fallback share the same counter). Caching that fails three
times in a row is almost certainly an environment problem, not "we haven't
found the working one yet"; the default-Gemma download path still runs.
No behavior change on the happy path: success returns after the first hit
exactly like today, and the trust-remote-code skip path does not consume an
attempt slot.
* shorter comment on auto-load cap
* studio chat: extend autoload cap to default Gemma fallback
Cached cascade respected MAX_AUTO_LOAD_ATTEMPTS but the default-Gemma
download path skipped the budget, so a broken cache could still emit a
fourth /api/inference/load. Gate the fallback on the same cap (and bump
loadAttempts when we do call loadModel) so the total cross-path budget
is 3, matching the cap's intent.
* studio/frontend: reconcile stale must_change_password localStorage flag
The client OR's a localStorage flag against /api/auth/status everywhere it
gates change-password routing, but never clears the flag when the server
flips requires_password_change back to false. A user whose default admin
password was already rotated (change-password from another browser, the
CLI reset-password command, or a recreated auth DB) keeps that flag, so:
1. requirePasswordChangeFlow lets them sit on the change-password route.
2. Back to login bounces via requireGuest, hasActiveSession (which only
checks key presence, not validity), then getPostAuthRoute, which sends
the user back because the flag is still set.
End result: the user is pinned on change-password and cannot escape without
clearing localStorage by hand.
Fix the three places that compare server status to the flag:
- auth-guards.ts fetchAuthStatus: clear the local flag whenever the server
reports requires_password_change = false.
- auth-guards.ts requireGuest: call fetchAuthStatus before routing so a stale
flag cannot decide getPostAuthRoute.
- auth-form.tsx initializeAuthForm: same reconcile inside the page so the
change-password page redirects to login as soon as it loads when the
server no longer requires a change.
- api.ts redirectToAuth: same reconcile in the fetch wrapper's auth redirect.
After the reconcile the redundant mustChangePassword() OR clauses are no
longer load bearing for the change-password gates; the server's
fetchAuthStatus is now the single source of truth.
* shorter comments around auth-status reconcile
* studio/auth: make localStorage reconcile bidirectional
Round-5 robustness pass based on 20 parallel reviewers of head 96b9e465.
1. studio/backend/core/training/worker.py - BNB version pin / dynamo disable
/ _grouped_mm fallback block was still gated on torch.version.hip alone
despite the torchao stub block above already using the broad check. AMD
SDK / Radeon Windows wheels (torch.__version__ contains "rocm" but
torch.version.hip is None) silently skipped the Windows ROCm runtime
patches. Aligned to the same broad check (8/20 reviewers).
2. studio/backend/core/training/worker.py - _hip_ver_at_least() now also
parses the ROCm version out of torch.__version__ (e.g. "2.11.0+rocm7.13.0")
when torch.version.hip is missing, so the kernel-fix gate is correct for
SDK / Radeon wheels too.
3. studio/backend/core/training/worker.py - _grouped_mm_safe_impl with
offs=None now picks torch.bmm/matmul for 3-D inputs instead of always
calling torch.mm. The real _grouped_mm accepts 3-D batched matmul; the
prior fallback raised "self must be a matrix" on MoE workloads (2/20).
4. studio/backend/main.py - dropped the HIP_PATH / ROCM_PATH env-var gate
from the BNB block; probe torch directly. Runtime-only Radeon / AMD SDK
Windows installs do not set those SDK env vars but still ship ROCm torch
(5/20 reviewers).
5. install.sh - Strix override now collects every gfx token from
rocminfo / amd-smi (in enumeration order), then indexes by
HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES so a mixed Strix iGPU + non-
Strix dGPU host where the user selected the dGPU does NOT get rerouted
to the Strix per-gfx index. Mirrors the Python update path (5/20 reviewers).
6. install.sh - Strix detection chain now also probes `amd-smi static --asic`,
matching the PowerShell installer (1/20). Closes the gap on runtime-only
Strix hosts where `amd-smi list` does not surface a gfx token.
7. studio/install_python_stack.py - _has_rocm_gpu() now has the sysfs KFD
topology fallback (/sys/class/kfd/kfd/topology/nodes/*/gpu_id), matching
install.sh. On minimal package-managed installs without rocminfo /
amd-smi GUI tools, `studio update` can now detect the GPU and repair the
venv instead of returning early (2/20).
8. studio/install_python_stack.py - _detect_amd_gfx_codes() now falls back
to `amd-smi list` and `amd-smi static --asic` when rocminfo is missing
(2/20). Strix routing on runtime-only Radeon hosts now matches what
install.sh has done for a while.
9. studio/install_python_stack.py - Strix override now applies even when
has_hip_torch is True. The whole point of the override is to repair an
existing broken torch.version.hip == "7.1" install; skipping the
reinstall left users on the known _grouped_mm segfaulting stack (3/20).
Tests: 231 passed, 1 skipped. sim_5301 30 cases pass. sim_cross 12 pass.
Move the inline tool-call XML parser and stripper out of
studio/backend/core/inference/llama_cpp.py into a new
studio/backend/core/tool_healing.py so external inference servers
(llama-server wrappers, llama-swap, custom shims) can reuse the same
logic without importing the inference orchestrator, structlog, httpx,
or anything from torch / transformers / unsloth.
Closes#5502.
What this PR does:
- New file studio/backend/core/tool_healing.py contains the regex
constants (_TOOL_CLOSED_PATS, _TOOL_ALL_PATS, _TC_JSON_START_RE,
_TC_FUNC_START_RE, _TC_END_TAG_RE, _TC_FUNC_CLOSE_RE,
_TC_PARAM_START_RE, _TC_PARAM_CLOSE_RE), parse_tool_calls_from_text,
and strip_tool_call_markup. The regexes and function bodies are
byte-for-byte the same as the previous inline implementation in
llama_cpp.py; only the @staticmethod decorator and the closure-only
`if not auto_heal_tool_calls: return text` short-circuit are dropped
(the latter stays in the caller as a fast path when healing is off).
- studio/backend/core/inference/llama_cpp.py now imports the regexes
and helpers from .tool_healing. LlamaCppBackend._parse_tool_calls_from_text
becomes a one-line delegate; the _strip_tool_markup closure keeps the
auto_heal_tool_calls fast path and delegates the work.
- Helper module imports cleanly without torch, transformers, structlog,
httpx, or numpy. studio.backend.core itself is already stdlib-only
at import time (lazy __getattr__), so `from
studio.backend.core.tool_healing import parse_tool_calls_from_text,
strip_tool_call_markup` is the lightweight import path issue #5502
asked for.
No behaviour change for existing Studio paths. parse_tool_calls_from_text
and strip_tool_call_markup produce the same OpenAI-shape output the
old inline code produced for every input.
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Preserves per-turn OpenAI image_url content parts in the standard GGUF /v1/chat/completions path so multi-image chat history keeps each image attached to its original turn. Legacy top-level image_base64 is injected as a synthetic image_url part only when no message-level image exists. Tool use is disabled whenever any GGUF image is present. Fixes#5470.
* fix: patch loss functions for Qwen3_5ForConditionalGeneration to prevent OOM errors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Narrow except scope and simplify LOSS_MAPPING sweep
Replace bare except Exception with the only two compatibility errors we
actually care about so genuine bugs in the sweep surface. Drop the
redundant _key != "ForCausalLM" guard since the __name__ predicate
already excludes the patched entry (UnslothForCausalLMLoss != ForCausalLMLoss).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Round 4 robustness pass based on 5 parallel Opus reviewers of head 21773215.
Seven items from across regression / edge-case / error-paths / architecture
reviews:
1. studio/backend/main.py BNB gate: aligned with the broad ROCm check used
everywhere else in this PR (torch.version.hip OR 'rocm' in __version__).
AMD SDK / Radeon Linux wheels do not always populate torch.version.hip;
without this, main.py would silently skip BNB_ROCM_VERSION while worker.py
set it.
2. studio/install_python_stack.py _install_bnb_windows_rocm: init _ok = False
before the try block. Without this, if pip_install_try itself raises
(e.g. OSError on uv binary missing), the finally block restored env vars
correctly but the subsequent `if not _ok:` raised UnboundLocalError,
masking the original exception.
3. studio/install_python_stack.py _detect_windows_gfx_arch:
- Rewrote to use re.findall (not re.search) on both hipinfo and amd-smi
output, dedup tokens preserving order, and select via new
_pick_visible_index() helper.
- HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES (first comma entry, integer)
now picks the right GPU on multi-AMD-GPU hosts. Out-of-range or non-int
values fall back to the first GPU (matches detect_host behaviour in
install_llama_prebuilt.py).
4. studio/install_python_stack.py Strix override now consults the runtime
target before flipping:
- Previous behaviour intersected gfx_codes with {gfx1151, gfx1150} and
picked the first Strix arch, ignoring whether HIP_VISIBLE_DEVICES
selected a non-Strix sibling (e.g. discrete RX 7900 in a mixed APU+dGPU
box). Could install Strix-specific wheels onto a gfx1100 dGPU.
- Now resolves the runtime gfx via _pick_visible_index() and only
overrides when that runtime target is in the Strix set.
5. studio/backend/main.py + studio/backend/core/training/worker.py: ROCm
version dir scan no longer sorts lexically. Previous sort placed "10.0"
before "7.0" alphabetically, which would mis-prioritise ROCm 10.x bin
dirs once AMD ships them. New _ver_key() splits on "." and sorts
numerically with a string fallback.
6. install.sh Strix override URL: replaced ${var%/} (strips one trailing
slash) with a while-loop that strips all trailing slashes, matching
Python's .rstrip("/"). A user setting UNSLOTH_AMD_ROCM_MIRROR with
"http://corp/whl///" no longer ends up with "http://corp/whl///gfx1151/"
which strict pip proxies (artifactory, sonatype) 404 on.
7. studio/install_python_stack.py: bumped torch import probe timeout from
30s to 90s. PyTorch's lazy .so loading can take 60-90s on cold NFS or
USB-backed venvs. The shorter timeout was producing a false "torch
missing" classification and reinstalling a working ROCm torch.
Tests: 231 passed, 1 skipped. sim_5301 30 cases pass (added 7 new sims for
multi-GPU detection, Strix sibling handling, and _ok-init regression).
Robustness pass on top of 76137b2d. Four targeted fixes:
1. install.sh ROCm-tag routing normalisation.
`rocm7.2.1` would route to https://download.pytorch.org/whl/rocm7.2.1
which does not exist (PyTorch publishes major.minor URLs only). Same
for any future patch-level tag. Normalise every rocm{maj.min}* pattern
to the bare {maj.min} index URL.
2. install.ps1 + studio/setup.ps1 marketing-name fallback.
The gfx1151 row matched 890M / Strix Halo / HX 37x / HX 38x / AI 9 HX
but not the actual retail name 'AMD Radeon 8060S Graphics' shipped by
OEMs (Ryzen AI MAX+ 395). Add '8060S' to the regex.
3. install_python_stack.py Strix + ROCm 7.1 routing parity with install.sh.
The shell installer reroutes Strix Halo / Point + ROCm 7.1 to
repo.amd.com/rocm/whl/{gfx}/ (which serves torch 2.11.0+rocm7.13.0
with the upstream _grouped_mm fix). The Python `studio update` path
only warned and still installed the broken generic rocm7.1 wheel.
Mirror the override: detect gfx1151/gfx1150 on ROCm 7.1, route to
the AMD per-gfx index, honour UNSLOTH_AMD_ROCM_MIRROR override.
4. _detect_windows_gfx_arch amd-smi parsing tightened.
The amd-smi fallback added in the prior commit used a bare
`\bgfx[1-9][0-9a-z]{2,3}\b` match against the lowercased stdout,
which could pick up stray gfx references in warnings / device-name
strings. Anchor on labelled lines first (Target_Graphics_Version,
ASIC, Arch, gfx) and fall back to the bare match only when no
labelled line is present.
Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py;
sim_5301 23 cases pass (6 new sims for the Strix override + amd-smi parsing).
* studio: emit one comma-chained --spec-type for CPU/Mac MTP path
llama-server takes a single --spec-type whose value may be
comma-separated to chain implementations (e.g. ngram-mod,draft-mtp).
The CPU/Mac MTP branch in LlamaCppBackend.load_model was passing
--spec-type twice in the same invocation, which is not the documented
chaining mechanism and silently drops one of the two specs depending
on llama.cpp's argv handling.
Collapse the pair to --spec-type ngram-mod,{mtp_token} and update the
stale _extra_args_set_spec_type docstring that claimed llama-server
accumulates repeated --spec-type. Update the matching pass-through
fixture in test_llama_server_args.py.
* studio: align MTP ngram-mod knobs with llama.cpp upstream defaults
Two correctness fixes against the llama.cpp server README:
1. The CPU/Mac comma-chained branch was emitting
--spec-ngram-mod-n-max 6 with --spec-ngram-mod-n-min 48, which is
nonsensical (min > max). Per the upstream default the value is 64.
2. The standalone ngram-mod branch was emitting --spec-ngram-size-n,
--draft-min, --draft-max. llama.cpp removed those arg aliases for
ngram-mod (they live only on the ngram-simple / map families now);
the correct knobs are --spec-ngram-mod-n-match / n-min / n-max.
Also refresh the inline comment block to point at the server README
rather than the older docs/speculative.md draft- aliases.
Addresses findings from a 10x reviewer pass on the prior fix commit:
1. studio/backend/core/training/worker.py (parity with main.py):
- Gate the torchao stub block on torch.version.hip / 'rocm' in
torch.__version__ instead of HIP_PATH / ROCM_PATH env-var presence.
Same root cause as main.py: HIP SDK env vars stick around on CUDA hosts.
- Add module-level Windows ROCm DLL registration block. Worker subprocesses
inherit env vars but not the parent's add_dll_directory handles, so the
first `import torch` in the worker could fail to find amdhip64.dll when
HIP_PATH\bin is not on PATH. Mirrors main.py setup. Handles retained at
module scope via _ROCM_DLL_HANDLES.
- Promote _WINDOWS_ROCM_GROUPED_MM_LIB to module scope with `global` in
run_training_process so the torch.library.Library registration survives
past function return / mid-run garbage collection.
- Harden _torch_has_hip() to also accept 'rocm' in torch.__version__
(AMD SDK / Radeon wheels may not set torch.version.hip).
2. studio/install_python_stack.py:
- Don't roll back ROCm torch when bitsandbytes install fails. The prior
commit gated _rocm_windows_torch_installed on _install_bnb_windows_rocm()
returning True; if torch installed successfully but bnb failed, the flag
stayed False and later install steps could overwrite ROCm torch with the
generic CPU torch wheel. Set the flag after torch install; surface bnb
failure as a separate warning instead.
- _detect_windows_gfx_arch now probes in three tiers: UNSLOTH_ROCM_GFX_ARCH
env-var override (matches the PowerShell installer), then hipinfo (PATH
or HIP_PATH\bin), then amd-smi (`static --asic`, `list`). Without the
amd-smi fallback, runtime-only Radeon installs without hipinfo on PATH
made `studio update` return early and leave the venv on CPU torch.
- Linux torch-already-rocm probe in _ensure_rocm_torch now matches the
Windows probe shape: accepts torch.version.hip OR 'rocm' in
torch.__version__ to cover AMD SDK / Radeon Linux wheels.
3. studio/backend/utils/hardware/hardware.py:
- apply_gpu_ids() final-fallback torch probe accepts 'rocm' in
torch.__version__ in addition to torch.version.hip, matching
detect_hardware(). AMD SDK wheels could otherwise leak through with
CUDA-only visibility masks on a spawned ROCm worker.
Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py
(no test changes needed; the probe shape that prints the hip version (or
'rocm' sentinel) preserves the existing non-empty-string contract).
Not addressed in this commit (deferred or out of scope):
- Tag drift / lemonade checksum (PR 5303 surface, not this PR).
- install.sh rocm7.2.1 URL: small fix, separate.
- install.ps1 / setup.ps1 'Radeon 8060S' marketing-name fallback table.
- Strix Halo + ROCm 7.1 routing asymmetry in Python update path.
