* fix: Claude Code Anthropic API tool compatibility
* fix: merge Anthropic server tool selections
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix anthropic /v1/messages server-tool alias misrout
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden Anthropic /v1/messages tool validation
* fix: dispatch Anthropic server tools by only
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: reject Anthropic client tools missing 'name' at boundary
AnthropicTool.name was relaxed to Optional[str] to accommodate server-tool
declarations. A client tool with input_schema but no name now parses but
is silently dropped by anthropic_tools_to_openai, leaving tool calling
disabled. Surface as 400 instead.
* fix: reject Anthropic client tools with empty 'name'
isinstance(name, str) accepts an empty string, but anthropic_tools_to_openai
drops entries via 'if not name', producing the same silent-disable
fallthrough the boundary check is meant to prevent. Tighten to also reject
empty name.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* feat: add custom model v1/model loading
* fix: require base URL for local model catalog loading
* ux/studio-provider-model-loading-controls
* fix: normalize local provider base URLs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add custom model v1/model loading
* fix: require base URL for local model catalog loading
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Reverts PR #5615 to give the safetensors + MLX healing parity work more time to bake before re-merging. The reverted feature branch `studio-tools-multi-format` remains untouched, and the follow-up PR will layer the healing-parity commits on top.
Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe.
* studio/frontend: show Generation stopped placeholder when cancelled mid-thinking
Closes#5563.
When the user clicks Stop before any visible content has streamed in,
the running indicator disappears but no Parts have rendered yet, leaving
just the AssistantActionBar floating below the user prompt. That looks
broken (and is the exact failure mode behind the 'tools work, but I
don't see anything happening' bucket of reports).
Add a sibling CancelledIndicator next to GeneratingIndicator that fires
when content is empty AND status is incomplete with reason cancelled,
rendering a muted 'Generation stopped.' italic. The terminal-state
label is consistent with tool-fallback's existing 'Cancelled tool'
treatment and with reasoning's 'Thought for N seconds' summary.
* studio/frontend: shorten CancelledIndicator comment
Trim the 3-line explanation to a single line describing what the
placeholder is for.
* studio/frontend: use 'Cancelled.' to match tool-fallback wording
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: settings dialog fits viewport at tablet widths
The dialog used a fixed w-[820px] with sm:w-[820px] override, so any
viewport between 640px and 820px (iPad portrait at 768px is the
canonical case) saw the dialog overflow horizontally by 26px on each
side -- the right-edge scroll arrow and the active-tab chevron got
clipped against the viewport.
Replace the hard 820 with min(820px, calc(100vw-2rem)) on both max-w
and w so the dialog caps at the original 820px on desktop and shrinks
to fit (with a 1rem gutter) on narrower screens. max-sm: still drives
the full-bleed h-dvh/w-dvw layout under 640px.
* studio/frontend: keep mobile full-bleed override !important
Bot review: base !max-w-[min(...)] is !important so the regular
max-sm:max-w-none never wins, leaving a 1rem gutter on phones where
the previous code rendered a true full-bleed dialog. Bump the mobile
override to !important too.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
The composer's mic icon buttons used tooltip="Dictate" /
"Stop dictation" but no aria-label, so screen-reader users heard
only the empty SVG-only button. Every other composer icon button
(Send, Add Attachment, audio buttons, composer pills) carries an
explicit aria-label; the shared-composer.tsx implementation already
does too. Mirror that here for parity.
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
The settings dialog opens via a global Ctrl+, keydown handler in
__root.tsx, not via a <DialogTrigger>. Radix's FocusScope tries to
capture document.activeElement at mount as the focus-restore target,
but settings-dialog.tsx schedules a requestAnimationFrame that focuses
the active tab button right after mount, racing FocusScope's previous-
focus capture. On Escape or close-button click, focus then lands on
<body> instead of the textarea (or button, or wherever the user was).
A Playwright focus-management probe confirmed: open dialog, press Tab
15 times (trap holds), press Escape, document.activeElement === BODY.
This is a WCAG 2.4.3 (Focus Order) violation: keyboard-only users
have to re-Tab from the start of the page after every settings visit.
Fix: capture document.activeElement in the Zustand store at the moment
openDialog() runs, then restore via onCloseAutoFocus on DialogContent.
Use opener.isConnected so a stale node from a re-rendered tree falls
back to Radix's default. closeDialog deliberately does NOT clear the
opener slot - onCloseAutoFocus reads it on the render after open=false,
so clearing in the same set() would null it before restoration.
Probe re-run confirms focus restored to the TEXTAREA opener after
Escape, after close-button click, on both repeats. Tab + Shift+Tab
trap still holds (unchanged Radix behaviour).
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: compare composer blocks send when no model picked
Closes the racing-handle half of #5569. In Compare mode (GeneralCompare
shell with model1/model2 props), if the user sends a prompt before
picking models in either pane, the SharedComposer used to fall through
to the per-handle append branch. Both panes then raced
createOpenAIStreamAdapter -> autoLoadSmallestModel, one won, the other
dispatched into an unloaded slot and produced an empty bubble with a
1000000.0 tok/s readout. The per-pane picker state never observed the
global checkpoint change either, so both pickers stayed at
"Select model".
Add a guard before the content build: when handlesRef has model1/model2
keys but both selections are empty, surface a toast asking the user to
pick models first, leave the text in the composer for retry, and never
enter the racing dispatch path. Keeps the per-pane picker state as the
source of truth for which model is on each side.
The unphysical tok/s readout that the same path produced is separately
covered by PR #5570 (display guard).
* studio/frontend: tighten compare-mode guard to require both panes
Review feedback on #5574:
- Gemini: the redundant `model1 !== undefined && model2 !== undefined`
checks let the racing-handle dispatch slip through whenever the
Compare props arrive as undefined, which is the exact case the
guard is trying to block.
- Codex: with `isGeneralizedCompare` keyed on `model1?.id || model2?.id`,
a half-selected Compare (one model picked, one empty) still falls
into the generalized branch. The composer clears, the empty pane
gets the user message appended, and `startRun` only fires for the
side with an id, leaving the empty pane with a dangling prompt
and no response.
Switch `isGeneralizedCompare` to require BOTH panes (`&&`), drop the
undefined gate, and surface the "Pick a model in each pane" toast for
either the fully-empty or half-selected case. `hasCompareHandles` is
true only inside GeneralCompareContent, so LoraCompare and the
single-pane path stay unchanged.
* studio/frontend: shorten compare-mode no-model-guard comment
* studio/frontend: clarify compare-pane toast wording
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: include filename in attachment aria-label and img alt
When a chat has multiple attachments of the same kind, the rendered
tiles all share the generic accessible name "Image attachment" or
"Document attachment". Sighted users get the filename from the Radix
tooltip that pops on hover, but:
- screen-reader users hear "Image attachment, Image attachment,
Image attachment" with no way to distinguish three PNGs;
- touch-device users (no hover) lose the filename entirely;
- keyboard-only users would have to focus and read a tooltip that
isn't always announced.
Fold the filename into both the button's aria-label and the thumbnail
<img alt>, falling back to the existing labels when the attachment has
no filename. Sighted UX is unchanged: the Radix tooltip already shows
the same name on hover, and the visible aria-label has no rendered
counterpart.
Found while running a multi-image attach probe in the autonomous Studio
UX loop (cycle 8). Repro:
await page.evaluate(`Array.from(document.querySelectorAll(
'button[aria-label*="attachment" i]'
)).map(b => b.getAttribute('aria-label'))`)
Before: ["Image attachment", "Document attachment", "Add Attachment"]
After: ["Image attachment: test_red_circle.png",
"Document attachment: notes.txt",
"Add Attachment"]
* studio/frontend: shorten attachment a11y comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: show Loading fallback instead of blank pane on lazy route navigation
Closes#5567.
Train, Recipes and Export pages are imported via React.lazy() in their
respective createRoute calls, and the Suspense boundary around <Outlet />
in __root.tsx passes fallback={null}. The result is a 1-3 second
completely white pane between sidebar click and content paint, which is
the exact failure mode behind reports that those pages look broken or
stuck. /chat does not suffer from this because chat.tsx imports its
ChatPage synchronously.
Replace fallback={null} on both Suspense boundaries (hideNavbar and
sidebar layouts) with a small centered 'Loading...' label using the
same muted-foreground style as elsewhere in the app. Synchronous routes
(/chat) never suspend so they are unaffected; lazy routes now have a
visible terminal-state placeholder while their chunk loads.
* studio/frontend: also apply RouteFallback to the sidebar Suspense
The first revision only replaced the fallback={null} inside the
hideNavbar branch (used for onboarding / login). The primary lazy
boundary that wraps Train / Recipes / Export is inside the SidebarInset
branch at the other Suspense site, which kept rendering null and made
the page look stuck for the same window the original bug describes
(per bot review feedback on #5568).
Replace both Suspense fallbacks with RouteFallback so the "Loading..."
placeholder fires on every lazy route, not just on the auth flows.
* studio/frontend: shorten RouteFallback comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: keep theme classes mutually exclusive on <html>
The Sonner Toaster reads next-themes (mounted at provider.tsx with
attribute="class" defaultTheme="light"), so on first mount next-themes
adds a "light" class to <html>. Studio's own setTheme path
(features/settings/stores/theme-store.ts) only toggled "dark", so
after the user picked Dark in settings the document ended up with
html.className = "light dark". Harmless in CSS cascade because the
dark variables override, but reads as a UI defect in devtools and trips
CSS-aware tooling that branches on class lists.
Toggle "light" alongside "dark" in applyToDocument so the two classes
stay mutually exclusive regardless of how next-themes seeded the
initial class.
* studio/frontend: shorten theme-toggle comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/web: distinguish "offline" from "studio crashed" in error toast
When the user's browser loses network mid-request, authFetch caught the
fetch TypeError and surfaced "Studio isn't running -- please relaunch it."
That is a correct diagnosis in the Tauri desktop app (the supervisor died
in-process), but it is a misleading diagnosis in the web build where the
backend lives elsewhere: the user will start hunting for a dead process
when the actual problem is connectivity.
Branch on navigator.onLine === false (web build only) and surface
"You appear to be offline. Check your network connection and try again."
instead. Tauri keeps the original wording so it stays accurate there.
Found while running a slow-network UX probe and toggling
Network.emulateNetworkConditions {offline: true} mid-stream.
* studio/frontend: shorten offline-error wording comment
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio/frontend: guard message-timing badge against unphysical tok/s
llama.cpp can report `predicted_ms == 0` and `predicted_n == 0` on turns
that effectively produced no generation (most reliably reproduced today
on a Compare-mode pane that loses the auto-load race and dispatches a
generate against an unloaded slot, see issue #5569). The current display
trusts `predicted_per_second` verbatim, which turns into `Infinity` /
`1000000.0 tok/s` on the action toolbar of an otherwise empty bubble
and reads like a UI defect even when the underlying request did happen.
Require at least one predicted token, at least one millisecond of
generation time, and a finite rate before rendering. Falls back to the
total stream time formatter, which already handles the zero case
gracefully.
* studio/frontend: shorten predictedRate guard comment
* studio/frontend: tighten timing guard threshold and hide Generation row when suppressed
Raise the decode-window floor from 1ms to 10ms so race-lost panes that
emit a stray token in 1-2ms (still giving 1000-5000 tok/s) drop out
alongside the predicted_ms=0 case. Gate the tooltip's Generation row
on the same hasPredicted predicate as Speed so the tooltip never shows
'Generation: 0ms' with no Speed underneath.
* studio/frontend: accept sub-10ms decode windows in timing guard
Cycle-15 codex P2 flagged that the >= 10ms threshold hid legitimate
fast generation (cached single-token, small models). The original
Infinity-blocker was predicted_ms=0, so use >0 instead. predicted_n
>= 1 and Number.isFinite() still keep the no-op race-lost cases out.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio: respect prefers-reduced-motion across animations
Tailwind animate-in/out, Radix dialog/popover zoom-in/slide-in transforms,
and the infinite shine / shiny-text / icon-pop keyframes all run at their
full duration regardless of the user's OS-level reduced-motion preference.
A Playwright probe that emulated the media query confirmed every measured
transition was identical between no-preference and reduce, so users with
vestibular triggers see the same scaling overlays and continuous shimmers.
Add the canonical universal-selector override so animation-duration,
animation-iteration-count, and transition-duration collapse to ~0ms when
the preference is set, leaving end states intact. Probe re-run shows
settings-dialog animationDuration drop from 0.1s to 1e-05s and the 50ms
mid-open screenshot is byte-identical to the settled one.
* studio: exempt .animate-spin from reduced-motion collapse
The universal-selector rule from the previous commit froze every
animation including .animate-spin, which is used as the canonical
in-progress indicator across Studio: tool execution loaders
(tool-ui-python/terminal/web-search/code-execution/fallback/group),
sonner toast spinners, Tauri startup + update screens, and the
generic <Spinner /> primitive in components/ui/spinner.tsx.
Freezing those leaves reduced-motion users with no visual signal
that work is in flight, which trades one accessibility win for
another. WCAG treats progress indicators as "essential motion"
that should keep moving.
Restore .animate-spin with a 1.5s cadence (instead of the default
1s) so the rotation is still perceptible but less aggressive than
the no-preference path. animation-iteration-count goes back to
`infinite` so the spinner doesn't halt after one rotation.
Verified via a focused probe that injects a .animate-spin element
and a .animate-in fade element side by side:
no-preference spin=1s infinite fade=0.15s
reduce spin=1.5s infinite fade=1e-05s
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
The settings dialog sidebar was fixed at w-[200px], which left only
~92px of horizontal space for tab labels after icon, gap, and the
'New' badge for Connections/API. 'Connections' (11 chars at the
14.5px font weight medium) overflowed and rendered as 'Connectio...',
matching the paper-cut reported in issue #5572.
Bump the sidebar to w-[216px] -- 16 more pixels of label space, fully
within the existing dialog width and unchanged on mobile
(max-sm:w-full still drives the responsive layout).
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Adds tools, thinking blocks, code execution, and web search support to the safetensors / transformers and MLX inference backends in Studio, bringing them to parity with the GGUF path.
What ships
- safetensors / transformers agentic tool loop with cumulative-text state machine, tool-call XML parser, and template kwarg forwarding (tools / enable_thinking / reasoning_effort / preserve_thinking).
- MLX backend: same kwargs accepted on Apple Silicon; chat_template_info shipped through worker IPC; pills enable for Qwen / Qwen3 / Qwen3.5 / Gemma reasoning.
- Capability classifier (_detect_safetensors_features) gates supports_tools on actual parser-compatible emission markers (<tool_call> / <function=) so Llama-3 / Mistral / Gemma 4 do not advertise toggles the parser cannot honour.
- gpt-oss override stays: reasoning on, tools off (Harmony channel, not <tool_call> XML).
- CWE-209 hygiene: safetensors SSE error path emits a constant message and logs the trace server-side.
Validation
- 256 unit tests green (43 tool-loop, 11 capability advertise, 7 MLX backend, 5 main-added, 190 adjacent inference / anthropic / openai regression).
- Cross-OS staging CI green on ubuntu-latest / macos-14 / windows-latest plus a dedicated MLX cartesian probe against real unsloth/Qwen3.5-0.8B on macos-14 (CI 26098107440).
- Capability parity verified across Qwen3 / Qwen3.5 / Llama-3 / Mistral / Gemma / DeepSeek-R1 / gpt-oss (incl. BF16).
- Manual confirmation from Imagineer99 on Qwen3.5-2B: think + search + code exec working.
Closes the safetensors / MLX gap with the GGUF backend.
* studio: reserve VRAM headroom for the MTP draft cache in auto-fit
When MTP is going to engage on this load, _fit_context_to_vram now
budgets 0.85 of available VRAM instead of 0.90, leaving room for
llama.cpp's secondary MTP draft KV cache + compute graph buffers.
Motivation: a user report on RTX 5090 (32 GB) showed Qwen3.6-27B-MTP-GGUF
UD-Q4_K_XL at native auto-context running roughly half the speed of
the same model with a slightly smaller context. The most parsimonious
explanation is a VRAM cliff: at native context the target's KV
already eats the 90% budget, then llama-server allocates the draft
cache + draft graph on top and spills into a slower partial-offload
path. Reducing the budget by 5% on MTP loads avoids the spill without
penalising non-MTP loads. On hardware with abundant VRAM (B200, etc.)
the fit is unchanged because the requested context already fits in
the tighter budget too.
MTP detection mirrors the auto-promotion logic in load_model: the
GGUF advertises nextn_predict_layers, or the model identifier /
local path matches the -MTP marker, and the user has not explicitly
opted out via speculative_type="off" or --spec-type extra args.
Tests: two new cases in test_kv_cache_estimation.py verify that
mtp_engaged=True yields a context less-than-or-equal-to the
non-MTP path on a tight budget, and that kv_on_gpu=False still
short-circuits regardless of mtp_engaged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: gate _mtp_will_engage on canonical-mode resolver
After PR #5582 introduced the 5-mode Speculative Decoding dropdown plus
_canonicalize_spec_mode, the auto-fit MTP-engaged predicate becomes:
* forced mtp / mtp+ngram -> always engage MTP (extra VRAM needed)
* auto + MTP GGUF (>= 3B) -> engages MTP via auto-promotion
* auto + MTP GGUF (sub-3B) -> falls back to ngram-mod (no extra VRAM)
* ngram / ngram-simple / off -> never engage MTP
* user --spec-type in extra_args -> resolver suppressed; no headroom
The old gate triggered on "anything but off", so it over-reserved the
0.85 budget when the user explicitly picked Ngram (no MTP) or when
Auto fell back to ngram-mod on a sub-3B MTP model. The 5% headroom
cost was minor but unnecessary.
Mirrors the same logic already encoded in _build_speculative_flags so
the auto-fit budget and the actual emission agree on whether MTP is
running.
All 361 backend tests pass.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: add --spec-draft-n-max toggle for MTP speculative decoding
Surface llama-server's --spec-draft-n-max as a first-class
LoadRequest field so users can tune the MTP draft tree size from
the chat settings panel. Default behaviour is unchanged: when the
caller omits spec_draft_n_max, the existing platform defaults still
apply (6 on GPU, 3 on CPU/Mac).
Why this matters: on context-constrained loads the draft KV cache
competes with the target model's KV cache for VRAM. Lowering
spec_draft_n_max reduces that pressure, lets a larger user context
fit, and recovers throughput; raising it pays off when draft
acceptance is high enough to amortise the extra cache.
Backend
- LoadRequest gains an optional spec_draft_n_max: int (1..16).
- LlamaCppBackend.load_model accepts and persists the override on
self._spec_draft_n_max, used in place of the hardcoded 6/3 in the
MTP emit branch.
- LoadResponse and InferenceStatusResponse echo the active value
(None when the platform default is in effect) so the UI can
hydrate the input on refresh.
- _already_in_target_state and _request_matches_loaded_settings
compare spec_draft_n_max alongside speculative_type so a value
change triggers a reload rather than no-op'ing.
- strip_shadowing_flags now strips inherited --spec-* extras when
either speculative_type or spec_draft_n_max is in fields_set, so
an inherited --spec-draft-n-max cannot last-wins-override a fresh
request's first-class field.
Frontend
- LoadModelRequest, LoadModelResponse, InferenceStatusResponse
TypeScript shapes get spec_draft_n_max.
- chat-runtime-store gains specDraftNMax / loadedSpecDraftNMax and
a setter, hydrated from /v1/status and /v1/load.
- chat-settings-sheet renders a "Draft Tokens" numeric input
directly under the Speculative Decoding switch when that switch
is on. Toggling the switch off clears the override; the Reset
button restores the loaded value.
Tests
- Four new regression tests cover _already_in_target_state with
matching / mismatching / non-MTP / unset spec_draft_n_max.
- Existing test_llama_server_args.py and test_llama_cpp_mtp_detection.py
green: 141 passed locally.
* studio: add --spec-draft-p-min and --spec-draft-p-split to spec strip set
llama.cpp server documents --spec-draft-p-min (default 0.75, min draft
acceptance probability) and --spec-draft-p-split (default 0.10). Both
are first-class spec-decoding knobs that should travel with the rest
of the --spec-* family when an Apply re-sets speculative_type, so an
inherited override doesn't leak across a fresh load.
* studio/tests: skip MTP capability-probe tests on Windows
The four probe_server_capabilities tests use a bash stub written to
tmp_path/llama-server, which Windows' subprocess can't execute
directly (no shebang resolution, .bat / .cmd would be needed). Mark
them skipif sys.platform == 'win32' so the rest of the MTP plumbing
suite stays green on Windows CI. Unix coverage is unchanged.
* studio: lower MTP GPU default --spec-draft-n-max from 6 to 2
Bench on B200 / Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL across five prompt
types (essay, code, story, math, science) with greedy temp=0:
prompt OFF n=1 n=2 n=3 n=6
essay 79.1 93.4 93.8 84.7 64.6
code 79.1 104.4 116.6 113.5 103.0
story 79.1 99.2 105.7 101.8 88.9
math 79.1 100.8 110.8 111.8 98.2
science 79.1 100.1 110.8 110.8 102.9
The previous hardcoded GPU default of 6 was 17% SLOWER than spec-off
on the essay prompt (64.6 vs 79.1 t/s) and 11-50% slower than n=2 on
the rest. n=2 wins on 4/5 prompts with a 1.18x-1.47x speedup vs OFF;
n=3 wins on the math prompt by a hair. n=6 collapses once acceptance
rate drops past n=3 -- wasted draft decode dominates the per-step
budget.
Matches the dataset README ("n_max=2 is the sweet spot for 36 of 42
quants"). Keeps CPU/Mac default at 3, which empirically tracks the
narrower ngram+MTP chained budget on those platforms.
Users who want the old behaviour can pass spec_draft_n_max in
LoadRequest (the toggle this PR also adds) or --spec-draft-n-max via
llama_extra_args.
* studio: skip MTP auto-promote on sub-2B models, backfill chat usage
Two MTP-visibility fixes uncovered while bisecting llama.cpp post-#22673
on Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL on B200.
Size gate. Direct llama-server bench (no Studio measurement loop) at
n_predict=192 across 9 prompts shows MTP regresses vs spec-off on
sub-2B dense models because draft cost exceeds savings:
Qwen3.5-0.8B Q4_K_XL GPU: 452.0 OFF -> 283.4 t/s n=2 (0.63x)
CPU: 84.5 OFF -> 64.9 t/s n=3 (0.77x)
Qwen3.5-4B Q4_K_XL GPU: 241.0 OFF -> 258.2 t/s n=2 (1.07x)
Qwen3.5-9B Q4_K_XL GPU: 201.6 OFF -> 228.9 t/s n=2 (1.14x)
Qwen3.5-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-35B-A3B Q4 GPU: 192.3 OFF -> 223.2 t/s n=2 (1.16x)
The 2B inflection is sharp. Skip auto-promote to draft-mtp when the
identifier reports <2.0B params; users can still force via --spec-type
or the Speculative Decoding toggle. Mirror the gate in the
reload-skip check so a sub-2B reload-with-default does not bounce a
spec-off backend.
Chat-completions usage. llama-server's final SSE chunk emits both an
OpenAI-style usage block and a custom timings block. timings.predicted_n
is always populated, but usage.completion_tokens is zero on some
server builds. The Studio chat UI computes generation t/s from
meta.usage.completion_tokens / totalStreamTime, so a zero
completion_tokens makes the UI fall back to wall-clock time
(including SSE / proxy / template overhead) which dilutes MTP gains and
makes ON look the same as OFF.
Add _backfill_usage_from_timings: if usage.completion_tokens is missing
or zero AND timings has predicted_n/prompt_n, synthesize a complete
usage dict. Apply at the streaming metadata yield in
generate_chat_completion and at the three accumulator/yield sites in
generate_chat_completion_with_tools so per-iteration counts are not
silently lost across tool calls.
Tests cover both the gate (sub-2B skips, 2B+ promotes) and the
backfill (zero usage filled, real usage preserved, empty timings
passthrough).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe + emit legacy ngram-mod flags for pre-rename llama-server
llama.cpp upstream renamed the ngram-mod tuning knobs:
--draft-max -> --spec-ngram-mod-n-max (and --spec-draft-n-max)
--draft-min -> --spec-ngram-mod-n-min (and --spec-draft-n-min)
--spec-ngram-size-n -> --spec-ngram-mod-n-match
The new names are real flags on post-rename builds and stub removal
entries on the same builds (with description "argument has been
removed"). Pre-rename builds only carry the legacy names as real
flags. Studio was emitting the new names unconditionally, so a user
running a pre-rename llama-server (e.g. an older prebuilt or a
hand-installed binary) would see "unknown argument" errors when the
ngram-mod path engages, or silent drop of the ngram knobs.
Extend `probe_server_capabilities` to parse the help text into
per-flag description blocks and tell real flags apart from removal
stubs by the "argument has been removed" marker. Add three new probe
fields: `ngram_mod_flavor` ("new" / "legacy" / None),
`supports_ngram_mod`, and `spec_draft_n_max_flag` (the actual n_max
flag the binary accepts). Cached by (path, mtime) the same way as
`mtp_token`.
Add `_build_ngram_mod_flags(caps, ...)` that picks the right flag
set, returning [] when neither is usable so callers can drop ngram
chaining entirely on minimal binaries.
Wire both call sites to use the probe-driven flag set:
- CPU/Mac MTP comma-chain (--spec-type ngram-mod,draft-mtp) emits
legacy or new knobs as appropriate. If neither set is available,
degrade to MTP-only (warn but still engage spec).
- Standalone --spec-type ngram-mod branch uses the same helper.
Tests cover post-rename detection, legacy detection, removal-stub
discrimination, minimal-binary case, and all three branches of
`_build_ngram_mod_flags` plus custom n_match/n_min/n_max values.
Verified against three real binaries (Studio bundled 726704a, my
build of 45b455e HEAD, and the MTP merge baseline 2555826) all
correctly reporting ngram_mod_flavor=new.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: sub-3B MTP falls back to ngram-mod, not off
Earlier sub-2B gate disabled speculative decoding entirely for tiny
dense MTP models because the MTP draft head's per-token cost exceeds
the acceptance savings at that scale. The "fully off" fallback was
conservative -- ngram-mod has near-zero idle cost on diverse content
and consistently outperforms both off and draft-mtp at sub-3B.
Clean-methodology bench (each of 9 distinct prompts run once after
two unrelated warmup prompts so the ngram-mod hash pool is
realistically populated but never holds the exact deterministic
output we're about to measure):
Q4_K_XL on B200:
0.8B OFF=451 draft-mtp n=2=263 (0.58x) ngram-only=498 (1.10x)
2B OFF=377 draft-mtp n=2=308 (0.82x) ngram-only=369 (1.00x)
4B OFF=240 draft-mtp n=2=260 (1.08x) -- 4B+ wins with MTP
Q4_K_XL on x86 48 cores:
0.8B OFF= 80 chained n=2= 69 (0.86x) ngram-only= 95 (1.19x)
2B OFF= 62 chained n=2= 51 (0.83x) ngram-only= 63 (1.01x)
4B OFF= 31 chained n=2= 41 (1.33x)
Change:
- Raise the MTP-skip threshold from 2.0B to 3.0B (2B falls below it).
- When skipping the MTP head, fall back to --spec-type ngram-mod via
the probe-driven _build_ngram_mod_flags helper. Works on both
post-rename and pre-rename llama-server builds.
- If the binary advertises neither ngram-mod flavor, fall back to
spec-off (older binaries that don't support ngram-mod at all).
- Mirror the same fallback in _already_in_target_state so a sub-3B
reload-with-default does not bounce a ngram-mod backend.
Tests updated: monkeypatch probe_server_capabilities so the gate
behavior is deterministic regardless of which llama-server happens
to be on the host. +1 new test for the "binary has no ngram-mod
support" branch; renamed prior 2B/0.8B tests to reflect new semantics.
This generalizes the size gate to be probe-driven instead of a hard
"disable spec" branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: 5-mode Speculative Decoding dropdown (Auto / MTP / Ngram / MTP+Ngram / Off)
Replace the Chat Settings Speculative Decoding on/off Switch with a 5-option
Select. Auto preserves today's platform-aware resolver (MTP on MTP GGUFs,
ngram-mod fallback for sub-3B, --spec-default for non-MTP). The other 3 modes
force the user's choice on BOTH GPU and CPU: MTP emits draft-mtp only (no
ngram chain on CPU), Ngram emits ngram-mod only, MTP+Ngram emits the
ngram-mod,draft-mtp chain on both platforms. Off is the existing fully-off
state, kept so the Switch's "disable" capability isn't lost.
Backend
- New module-level _canonicalize_spec_mode(value) maps any accepted input
(canonical, legacy "default" / "draft-mtp" / "ngram-mod" / "ngram-simple",
or comma-chained "ngram-mod,draft-mtp") onto one of auto / mtp / ngram /
mtp+ngram / off / ngram-simple / None. Lets external callers and old
persisted UI state round-trip without breaking.
- LlamaCppBackend grows a _requested_spec_mode field + requested_spec_mode
property storing the canonical UI mode the user requested. Status
responses round-trip this instead of the resolved internal flag, so the
dropdown restores the picked value after reload / refresh (Auto on a 27B
MTP GGUF resolves to draft-mtp internally but the dropdown stays on
"Auto").
- The resolver block in load_model is extracted into a unit-testable
_build_speculative_flags method. Forced MTP / MTP+Ngram on a sub-3B or
non-MTP GGUF logs a warning and engages anyway (user override > the
Auto-path sub-3B fallback).
- _already_in_target_state and routes/inference._request_matches_loaded_settings
now compare canonical-requested mode, dropping the old auto-promotion
mirror. spec_draft_n_max still gates on the resolved spec so Auto + a
changed n_max still bounces a reload.
Frontend
- chat-settings-sheet.tsx: Switch swapped for Select modeled on the KV
Cache Dtype Select. Items: Auto / MTP / Ngram / MTP+Ngram / Off. Draft
Tokens input only visible when speculativeType is "mtp" or "mtp+ngram".
- chat-runtime-store.ts: initial value flips from "default" to "auto".
- use-chat-model-runtime.ts normalizeSpeculativeType mirrors the backend
canonicaliser so persisted "default" / "draft-mtp" / "ngram-mod" / chain
values hydrate to the right dropdown option.
- types/api.ts: docs the canonical wire vocabulary.
Tests
- 53 new assertions in test_llama_cpp_mtp_detection.py: full
_canonicalize_spec_mode table, a 23-row resolver matrix across
(requested mode) x (GPU/CPU) x (model size class), plus n_max override,
user-extra-args precedence, requested-mode round-trip, and graceful
degrade on an outdated llama-server without an MTP token.
- 165 existing backend tests still green. 218 total in the MTP /
server-args / reload-inheritance suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reset Speculative Decoding to Auto on model switch
When the user switches from model A to a different model B, clear the
runtime store's speculativeType + specDraftNMax (and their loaded*
shadows). The new load request then carries null, the backend
canonicalises that to "auto", and its platform-aware resolver runs
fresh for the new model.
Without this, a non-MTP model loaded with "Off" carried the Off choice
into a subsequent MTP load, suppressing MTP auto-promotion (and the
sub-3B ngram-mod fallback) until the user manually opened settings and
flipped the dropdown back to Auto. The clean-sweep deep probe caught
it as anomaly A-1.
The reset only fires when currentCheckpoint != modelId, so a
same-model reapply or forceReload still honours the user's current
spec choice. End-to-end probe on Qwen3.5-4B-GGUF (non-MTP, Off) ->
Qwen3.5-0.8B-MTP confirms: dropdown shows Auto, /api/inference/status
returns speculative_type=auto, studio.log shows the Auto sub-3B
fallback emitted --spec-type ngram-mod.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* ci: add advisory lockfile supply-chain audit
Adds a fast, focused workflow that scans every checked-in npm and
cargo lockfile on PRs touching one. Default behaviour is advisory:
only public indicator-of-compromise strings, versions on the public
known-malicious list, and structurally broken lockfiles fail the
build. Structural anomalies (missing integrity hashes, non-default
registry, etc.) surface as :⚠️: annotations without gating
merges, so reviewers see the audit result inline on every PR
without changing the existing install behaviour.
Also commits the two missing npm lockfiles the audit needs:
studio/package-lock.json (Tauri CLI holder for desktop release)
and studio/backend/core/data_recipe/oxc-validator/package-lock.json
(oxc-parser runtime for the data-recipe validator). studio/setup.sh,
studio/setup.ps1, build.sh, and pyproject.toml are intentionally
left alone so the existing install path keeps working unchanged.
Audit script behaviour:
default mode -> exits 1 only on blocked-known-malicious,
known-ioc-string, malformed-lockfile,
missing-lockfile, unreadable-lockfile, or
missing-toml-parser
--strict -> promotes every finding to blocking (opt-in)
Adds a try/except around lockfile reads so a permissions error
prints a finding instead of crashing CI with a raw traceback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test(security): update cargo regression test for advisory mode
`scripts/lockfile_supply_chain_audit.py` now classifies
`non-registry-cargo-source` as an advisory finding by default
(returns exit 0 with a `:⚠️:` annotation) rather than
unconditionally blocking with exit 1. Update the existing
`test_malicious_cargo_lockfile_refused` to pass --strict so it
keeps verifying the "refuse to install" behavior it is named for,
and add a second test that pins the default-mode behavior:
advisory finding emitted, exit code 0.
* audit: escape Finding for GH Actions annotations
`:⚠️:` and `::error::` workflow commands truncate the
annotation message at the first newline unless the message is
%-encoded per the workflow-commands spec. Since `Finding.__str__`
returns three lines (kind+path, package, detail), the package
and detail fields were being dropped from the GitHub Actions UI.
Add a `_gha_escape()` helper that applies the spec'd escapes
(`%` -> `%25`, then `\r` -> `%0D`, then `\n` -> `%0A`; the `%`
replacement must happen first so the subsequent escapes are not
double-encoded), wrap every Finding rendered into a workflow
command with it, and pin both the helper and the end-to-end
single-line emission with two new regression tests.
Caught by gemini-code-assist on PR #5604.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: regenerate desktop launcher on `unsloth studio update`
Today `unsloth studio update` only mutates the venv. The macOS .app bundle,
the Linux .desktop file, and the shared launch-studio.sh stub bake their
paths and `studio_install_id` at install time and never refresh. Users who
update an existing Studio install report the Dock / Applications icon still
pointing at the old launcher; only a fresh `curl ... install.sh | sh`
fixes it because that path re-enters install.sh's create_studio_shortcuts.
Wire the same logic into the update path:
- install.sh: add --shortcuts-only. Skips the heavy install steps, resolves
STUDIO_HOME / OS / DATA_DIR through the existing _resolve_studio_destinations
+ platform detection, then calls create_studio_shortcuts and exits.
- unsloth_cli/commands/studio.py: after setup.sh succeeds, call install.sh
with --shortcuts-only. Prefers a local checkout's install.sh (when
STUDIO_LOCAL_REPO is set) or one shipped under _PACKAGE_ROOT, and falls
back to fetching the upstream installer from https://unsloth.ai/install.sh
for PyPI-installed users (the wheel does not ship install.sh).
Net effect: `unsloth studio update` now refreshes the macOS .app stub,
launcher script, studio.conf, and Linux .desktop entry on every update, so
the desktop icon stays in sync with the venv that setup.sh just updated.
Env-override and Tauri modes keep their existing behavior (no persistent
menu shortcuts, but the launch-studio.sh is still regenerated).
Windows is unchanged here; setup.ps1 already handles its own Start Menu /
Desktop .lnk creation on update.
* studio: also regenerate Windows .lnk shortcuts on update
Mirror the macOS fix: install.ps1 gains --shortcuts-only that short-circuits
to New-StudioShortcuts, and unsloth studio update calls it after setup.ps1
the same way it now does on macOS / Linux.
PyPI installs do not ship install.ps1, so the Python helper fetches the
upstream script from https://unsloth.ai/install.ps1 and pipes it into
powershell.exe -Command - with an explicit Install-UnslothStudio call
appended (irm | iex relies on the trailing @args, which is empty when
launched from stdin).
setup.ps1 alone never recreates the Start Menu / Desktop .lnk targets or
the launch-studio.{ps1,vbs} scripts, so without this update users on
Windows hit the same stale-icon regression that triggered the macOS PR.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: rename unsloth.exe to .deleteme before update on Windows
Pip's editable reinstall calls uninstall first, which deletes every RECORD
entry. unsloth.exe is one of them, and Windows refuses to delete a file
whose image is mapped into the running process tree. The first
unsloth studio update after install therefore fails with:
OSError: [WinError 32] The process cannot access the file because it
is being used by another process: ...\Scripts\unsloth.exe
Windows does allow renaming an in-use exe, so move it aside before
_run_setup_script kicks pip. pip then drops a fresh unsloth.exe at the
original path; the *.exe.deleteme left behind is cleaned up at the start
of the next update once the previous shim has exited.
* studio: rename unsloth.exe from setup.ps1 to reliably bypass exe lock
* studio: print python -m workaround when Windows exe lock blocks update
* studio: use python -c hint (unsloth_cli has no __main__)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* install.sh: reshape --shortcuts-only Tauri guard to pass exit-order test
* shorter comments in update / launcher regen logic
* studio update: env-mode passthrough + non-silent shortcuts-only error
* studio update: address codex/gemini PR review
- Strip install.ps1's `Install-UnslothStudio @args` auto-invoke before
appending an explicit `--shortcuts-only` call so PyPI Windows installs
don't re-run the full installer over stdin.
- subprocess.run(input=wrapper, ...) now uses encoding="utf-8" so box
drawing chars in install.ps1 don't UnicodeEncodeError on CP1252.
- Wrap _run_setup_script in try/except to restore unsloth.exe from
.deleteme if setup fails, and mirror that rollback inside setup.ps1
when install_python_stack.py exits non-zero.
- Capture subprocess return codes in _refresh_desktop_shortcuts and
echo a one-line warning on non-zero so silent stale-shortcut failures
surface.
- Drop --local from the Windows lock-recovery hint so users on PyPI
installs don't accidentally switch into editable-checkout mode.
- Quote $VENV_ABS_BIN/unsloth in the install.sh shortcuts-only error
so paths with spaces print legibly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio update: harden Windows refresh per multi-reviewer pass
- PowerShell stdin path now writes the wrapper to a UTF-8 BOM tempfile
and runs it via `-File`. `powershell.exe -Command -` decodes stdin
with the OEM code page, which mangles box-drawing chars in the
fetched install.ps1; -File reads the BOM and decodes UTF-8 cleanly.
- _restore_self_exe_lock_windows now treats a zero-byte unsloth.exe as
a partial-write and prefers the .deleteme copy. setup.ps1 mirrors
the same check.
- _release_self_exe_lock_windows uses os.replace for atomic overwrite
so a stale .deleteme from an aborted prior update doesn't break the
rename.
- Lock-recovery hint mentions that --local should be re-added when
the user installed from a repo checkout.
* studio update: respect Tauri context and tidy Windows .deleteme
Tauri's update.rs spawns `unsloth studio update`; without a signal,
the CLI's _refresh_desktop_shortcuts would call install.{sh,ps1}
--shortcuts-only and create duplicate ~/Applications/Unsloth Studio.app
(or .desktop / .lnk) entries that collide with the Tauri bundle.
- update.rs now sets UNSLOTH_TAURI_UPDATE=1 on the spawned child.
- studio.py's update() skips _refresh_desktop_shortcuts when that env
var is set; Tauri owns its own bundle entries.
- After a successful Windows update, drop the .deleteme orphan so
repeated updates don't accumulate stale binaries that could later
be promoted by _restore_self_exe_lock_windows on a cross-version
failure.
- Tempfile for the PyPI-fallback PowerShell path now uses an
unsloth-studio-refresh- prefix so AV/EDR rules and user greps can
identify it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio update: drop obsolete WinError 32 hint, echo Tauri skip
The rename trick in _release_self_exe_lock_windows + setup.ps1's
restore now handle the .exe-lock case in-flow; the printed hint
suggested re-running update via venv python, but that just re-enters
the same update() and hits the same failure if the rename didn't help.
Removing the misleading hint and its helper.
Also surface a one-line typer.echo when refresh is skipped under
UNSLOTH_TAURI_UPDATE so --verbose logs make the branch visible.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: cap auto-load cascade attempts
autoLoadSmallestModel walks every cached GGUF and safetensors repo with a
try/catch + continue, so a folder of broken caches (missing files, stale
llama.cpp prebuilt, GPU OOM) can fire dozens of failing POST /api/inference/load
calls in a row. Each call costs ~5 seconds (HF metadata probe + DNS guard
inside inference.py), so the user sees a runaway sequence of request_completed
log lines after sending one message that needed an auto-load.
Cap the total loadModel calls inside autoLoadSmallestModel at 3 (GGUF cascade
plus safetensors fallback share the same counter). Caching that fails three
times in a row is almost certainly an environment problem, not "we haven't
found the working one yet"; the default-Gemma download path still runs.
No behavior change on the happy path: success returns after the first hit
exactly like today, and the trust-remote-code skip path does not consume an
attempt slot.
* shorter comment on auto-load cap
* studio chat: extend autoload cap to default Gemma fallback
Cached cascade respected MAX_AUTO_LOAD_ATTEMPTS but the default-Gemma
download path skipped the budget, so a broken cache could still emit a
fourth /api/inference/load. Gate the fallback on the same cap (and bump
loadAttempts when we do call loadModel) so the total cross-path budget
is 3, matching the cap's intent.
* studio/frontend: reconcile stale must_change_password localStorage flag
The client OR's a localStorage flag against /api/auth/status everywhere it
gates change-password routing, but never clears the flag when the server
flips requires_password_change back to false. A user whose default admin
password was already rotated (change-password from another browser, the
CLI reset-password command, or a recreated auth DB) keeps that flag, so:
1. requirePasswordChangeFlow lets them sit on the change-password route.
2. Back to login bounces via requireGuest, hasActiveSession (which only
checks key presence, not validity), then getPostAuthRoute, which sends
the user back because the flag is still set.
End result: the user is pinned on change-password and cannot escape without
clearing localStorage by hand.
Fix the three places that compare server status to the flag:
- auth-guards.ts fetchAuthStatus: clear the local flag whenever the server
reports requires_password_change = false.
- auth-guards.ts requireGuest: call fetchAuthStatus before routing so a stale
flag cannot decide getPostAuthRoute.
- auth-form.tsx initializeAuthForm: same reconcile inside the page so the
change-password page redirects to login as soon as it loads when the
server no longer requires a change.
- api.ts redirectToAuth: same reconcile in the fetch wrapper's auth redirect.
After the reconcile the redundant mustChangePassword() OR clauses are no
longer load bearing for the change-password gates; the server's
fetchAuthStatus is now the single source of truth.
* shorter comments around auth-status reconcile
* studio/auth: make localStorage reconcile bidirectional
Move the inline tool-call XML parser and stripper out of
studio/backend/core/inference/llama_cpp.py into a new
studio/backend/core/tool_healing.py so external inference servers
(llama-server wrappers, llama-swap, custom shims) can reuse the same
logic without importing the inference orchestrator, structlog, httpx,
or anything from torch / transformers / unsloth.
Closes#5502.
What this PR does:
- New file studio/backend/core/tool_healing.py contains the regex
constants (_TOOL_CLOSED_PATS, _TOOL_ALL_PATS, _TC_JSON_START_RE,
_TC_FUNC_START_RE, _TC_END_TAG_RE, _TC_FUNC_CLOSE_RE,
_TC_PARAM_START_RE, _TC_PARAM_CLOSE_RE), parse_tool_calls_from_text,
and strip_tool_call_markup. The regexes and function bodies are
byte-for-byte the same as the previous inline implementation in
llama_cpp.py; only the @staticmethod decorator and the closure-only
`if not auto_heal_tool_calls: return text` short-circuit are dropped
(the latter stays in the caller as a fast path when healing is off).
- studio/backend/core/inference/llama_cpp.py now imports the regexes
and helpers from .tool_healing. LlamaCppBackend._parse_tool_calls_from_text
becomes a one-line delegate; the _strip_tool_markup closure keeps the
auto_heal_tool_calls fast path and delegates the work.
- Helper module imports cleanly without torch, transformers, structlog,
httpx, or numpy. studio.backend.core itself is already stdlib-only
at import time (lazy __getattr__), so `from
studio.backend.core.tool_healing import parse_tool_calls_from_text,
strip_tool_call_markup` is the lightweight import path issue #5502
asked for.
No behaviour change for existing Studio paths. parse_tool_calls_from_text
and strip_tool_call_markup produce the same OpenAI-shape output the
old inline code produced for every input.
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Preserves per-turn OpenAI image_url content parts in the standard GGUF /v1/chat/completions path so multi-image chat history keeps each image attached to its original turn. Legacy top-level image_base64 is injected as a synthetic image_url part only when no message-level image exists. Tool use is disabled whenever any GGUF image is present. Fixes#5470.
* studio: emit one comma-chained --spec-type for CPU/Mac MTP path
llama-server takes a single --spec-type whose value may be
comma-separated to chain implementations (e.g. ngram-mod,draft-mtp).
The CPU/Mac MTP branch in LlamaCppBackend.load_model was passing
--spec-type twice in the same invocation, which is not the documented
chaining mechanism and silently drops one of the two specs depending
on llama.cpp's argv handling.
Collapse the pair to --spec-type ngram-mod,{mtp_token} and update the
stale _extra_args_set_spec_type docstring that claimed llama-server
accumulates repeated --spec-type. Update the matching pass-through
fixture in test_llama_server_args.py.
* studio: align MTP ngram-mod knobs with llama.cpp upstream defaults
Two correctness fixes against the llama.cpp server README:
1. The CPU/Mac comma-chained branch was emitting
--spec-ngram-mod-n-max 6 with --spec-ngram-mod-n-min 48, which is
nonsensical (min > max). Per the upstream default the value is 64.
2. The standalone ngram-mod branch was emitting --spec-ngram-size-n,
--draft-min, --draft-max. llama.cpp removed those arg aliases for
ngram-mod (they live only on the ngram-simple / map families now);
the correct knobs are --spec-ngram-mod-n-match / n-min / n-max.
Also refresh the inline comment block to point at the server README
rather than the older docs/speculative.md draft- aliases.
Two related issues on the chat toasts:
1. Close X did nothing. The lib/toast.ts wrapper defaulted every toast
to `dismissible: false` (originally to keep swipe capture from
stealing text selection). In sonner v2, `dismissible: false` makes
the close-button onClick a no-op, so the X looked clickable but
never dismissed the toast. The Toaster already sets
`swipeDirections={[]}` in components/ui/sonner.tsx, so the
per-toast swipe workaround is unnecessary and harmful. Replace the
wrapper with a thin re-export of sonner.
2. Close X hover collapsed to a near-black circle in light mode.
Sonner's default close-button styling uses fixed gray-scale tokens
(--gray2 hover, --gray12 text) that ignore the theme attribute.
Once the Toaster's inline style overrides --normal-bg with
var(--popover), the base background follows the app theme but the
hover state does not, so the hover bg lands on a color that has no
contrast with the X glyph. Pin both base and hover to theme tokens
(--popover, --muted, --popover-foreground, --border) so contrast
stays visible in both light and dark modes.
Repro: open chat, load any cached model, hover the X on the
"<name> loaded" toast in light mode -- before this change the circle
turned dark and the click did nothing; after, the circle stays light
and the click dismisses the toast.
* studio: engage draft-mtp on vision MTP GGUFs
The draft-mtp auto-promotion in LlamaCppBackend.load_model was gated on
not effective_is_vision, and the spec-emit branch repeated the same
guard. Every Unsloth -MTP GGUF repo ships an mmproj projector, so
effective_is_vision was always True for those repos and the MTP speedup
silently never engaged out of the box.
llama.cpp #22673 explicitly states MTP is compatible with vision input.
The bundled b9204 server happily loads both: a manual run with
--mmproj ... --spec-type draft-mtp --spec-draft-n-max 6 logs
"loaded multimodal model" followed by
"adding speculative implementation 'draft-mtp'".
Drop the vision gate from both sites and rewrite the matching short
circuit in _already_in_target_state so reload checks reach the auto
promotion path on vision MTP loads. Add three regression tests covering
vision MTP match (auto and default), and non MTP vision repo unaffected.
Verified on a B200 with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
base decode 179.7 t/s vs MTP decode 253.8 t/s, draft acceptance 0.57,
1.41x speedup on a 255 token completion. mmproj still loads and image
input remains available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: prefer Qwen3.5 -MTP GGUF variants in default model lists
With the vision gate dropped in the previous commit, draft-mtp now
auto-engages on -MTP GGUF repos out of the box. Swap the four Qwen3.5
recommended entries in DEFAULT_MODELS_GGUF and DEFAULT_MODELS_STANDARD
to their -MTP-GGUF counterparts so new users get the speedup by default:
unsloth/Qwen3.5-4B-GGUF -> unsloth/Qwen3.5-4B-MTP-GGUF
unsloth/Qwen3.5-9B-GGUF -> unsloth/Qwen3.5-9B-MTP-GGUF
unsloth/Qwen3.5-35B-A3B-GGUF -> unsloth/Qwen3.5-35B-A3B-MTP-GGUF
unsloth/Qwen3.5-0.8B-GGUF -> unsloth/Qwen3.5-0.8B-MTP-GGUF
All four HF repos exist (HEAD 200) and ship the same UD-Q4_K_XL quant
layout as the non-MTP variants. Non-Qwen3.5 entries are untouched.
* bump version to 2026.5.4
Picks up the studio MTP vision-gate fix and the Qwen3.5 -MTP default
swap in this PR.
* studio: prefer Qwen3.6-35B-A3B-MTP-GGUF in default model lists
Same rationale as the previous Qwen3.5 swap. The Qwen3.6 MTP variant
exists at unsloth/Qwen3.6-35B-A3B-MTP-GGUF (HF HEAD 200) and now
auto-engages draft-mtp out of the box with the gate fix.
* studio: drop --spec-draft-n-max from 6 to 3 for draft-mtp
n=6 is too greedy: on Qwen3.6 the draft has to guess 6 tokens ahead
and acceptance crashes to ~0.45, leaving only ~14% throughput gain.
PR ggml-org/llama.cpp#22673's author benched n=3 at ~0.72 acceptance
and 2 to 3x speedup on the same Qwen3.6 family, and the README sample
command uses n=2 or n=3. Match that.
CPU/Mac branch already uses n=3, so this aligns both paths.
* studio: set --spec-draft-n-max back to 6 for draft-mtp on GPU
Reverts the n=3 tuning. n=6 is the original default; user-side comparisons
hold the larger draft window steady so the toggle (next commit) is the
primary on/off lever.
* studio: add Speculative Decoding toggle under Max Tokens
Adds a top-level kill switch (panel-switch under Max Tokens, mirroring
Auto-Healing Tool Calls) that forces the /load request's
speculative_type to "off" when disabled. The backend "off" branch in
LlamaCppBackend.load_model skips both the draft-mtp auto-promotion and
the spec-emit branch, so neither --spec-type draft-mtp nor
--spec-default reaches llama-server.
Wiring:
- chat-runtime-store: new speculativeDecodingEnabled bool, default
true, persisted to localStorage under unsloth_speculative_decoding,
plus a setSpeculativeDecodingEnabled setter.
- chat-settings-sheet: SpeculativeDecodingToggle rendered immediately
beneath the Max Tokens slider for non-external models.
- use-chat-model-runtime: when speculativeDecodingEnabled is false,
override speculative_type to "off" in the loadModel call so the
switch wins over any pre-existing speculativeType state (including
the existing per-model toggle in Model Settings).
Verified end to end on unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
toggle ON emits --spec-type draft-mtp --spec-draft-n-max 6; toggle
OFF emits zero --spec-* flags on the same MTP GGUF.
* studio: relocate Speculative Decoding toggle into Model Settings
Move the toggle out from under Max Tokens and back into the Model
Settings section, directly beneath KV Cache Dtype, where the existing
Apply/Reset workflow already drives a reload on dirty. This way flipping
the switch in the UI actually picks up: the section becomes dirty,
Apply re-runs /load with the new speculative_type.
Drop the !currentModelIsMultimodal gate so vision MTP GGUFs can also
disable speculative decoding from the UI.
Switch the toggle's off-value from null to "off" so the backend's "off"
short-circuit fires for MTP models too (null normalises to None which
re-triggers the draft-mtp auto-promotion).
Tooltip now reads "Faster generation with 0% accuracy hit".
Remove the now-redundant speculativeDecodingEnabled bool + setter from
the runtime store and the load-time override in use-chat-model-runtime;
the toggle binds directly to speculativeType.
* studio: restore OOM/TIGHT badge on recommended GGUF rows
The recommended-list row passed vramStatus=null for any GGUF repo
because the existing useRecommendedModelVram hook reads safetensors
totals from HF model info, which GGUF-only repos do not expose. As a
result, an OOM Q-quant repo would render with only a "GGUF" badge and
no visual signal that nothing in it fits.
Add useGgufRecommendedFit: per repo, fetch the variant list via the
existing /api/models/gguf-variants endpoint, take the smallest
variant's size_bytes, and classify with the same 0.7*GPU + 0.7*RAM
thresholds as GgufVariantExpander. Session-scoped cache + in-flight
dedup so a repo is requested at most once.
Wire the result into the three GGUF row sites in pickers.tsx so OOM
and TIGHT badges show on the collapsed cards.
* Revert "studio: restore OOM/TIGHT badge on recommended GGUF rows"
This reverts commit 07793b1240df72b13e51d6dc15f63c4ee8c6cba9.
The new useGgufRecommendedFit hook was treating the symptom. PR #5561
identified the real root cause: useGpuInfo was calling /api/system
with plain fetch instead of authFetch, so the session-auth check
failed silently and gpu.available stayed false everywhere. With no
GPU info, every fit check (variant expander, recommended carousel)
fell back to "no signal" and dropped the OOM/TIGHT badges.
Reverting the over-engineered hook and applying the authFetch fix
in the next commit, which restores the existing badges with one line.
* chore: replace qwen suggested with MTP variant
* fix: restore GPU info auth for GGUF fit badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* studio/chat: release stuck IME flag when compositionend never fires
Chrome on Windows talking to a WSL-hosted Studio (issue #5546) fires
compositionstart + compositionupdate but no compositionend after the
IME commits. The earlier hardening in #5327 cleared the stale flag on
the next non-composing input event, which never arrives in this
sequence, so composingRef stays true forever and the Send button stays
disabled even though the committed CJK text is already in the textarea.
Add a watchdog in both useImeComposerInputHandlers (main + edit
composer) and SharedComposer (compare mode) that runs the same reset
the missing compositionend would have done. The timer is rearmed on
every compositionupdate and on every non-composing input so it only
fires when the IME pipeline has actually gone quiet — normal candidate
selection keeps it alive, the WSL stuck case lets it expire.
Extends the existing IME Playwright smoke with a stuck-compositionend
repro and adds a static guard so the watchdog can't be removed without
the regression tests catching it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-pin composing flag on IME keydown to close#5546 watchdog gap
The stuck-compositionend watchdog (PR #5551) releases composingRef after
2500 ms of IME silence so Send unwedges in the WSL+Chrome case. The same
release also fires during a long candidate-window pause in healthy IMEs,
which lets a subsequent IME-confirm Enter slip preedit text through
handleSubmit (main composer) or click-Send through send() (compare composer).
Add a keydown gate to both composers: when the browser still reports
nativeEvent.isComposing or keyCode 229, re-pin composingRef and cancel
any pending watchdog so the next form-submit / send() guard refuses.
The Send button stays visually enabled (avoids re-introducing the
stuck-UI bug) but the submit path is blocked until a real compositionend
or non-composing input arrives. Mirrors the existing isComposing guard
shape in shared-composer.onKeyDown.
Tests:
- tests/studio/test_composer_rtl_bidi_attribute.py: two new static
guards asserting the keydown gate wiring in both composer files.
- tests/studio/playwright_chat_ime_i18n.py: new section 6c repro that
fires the IME-confirm keydown after the watchdog has cleared, then
triggers form.requestSubmit() and asserts the preedit text is not
cleared (would indicate a leaked submit).
Verified across Chromium / Firefox / WebKit via a side-by-side pre-PR
vs post-PR simulation (54 scenarios, zero pageerror or console.error).
The #5546 stuck-end repro still passes (Send re-enables 2.5-3 s after
the silent commit) and the new keydown-repin probe confirms the submit
gate refuses on all three engines.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-arm IME watchdog after keydown re-pin (Codex P1)
The keydown re-pin added in 2c3c9793 closed the watchdog-race for
healthy IMEs, but on the same WSL+Chrome no-compositionend path this
PR targets it would re-lock Send permanently: setting composingRef=true
and only *clearing* the watchdog leaves the flag pinned forever if no
follow-up compositionend or non-composing input ever arrives.
Swap clearStuckTimer/clearStuckImeTimer for refreshStuckTimer/
refreshStuckImeTimer in both composer keydown gates so the watchdog
fires once more after every IME keypress. Same visual contract — Send
stays enabled — the submit gate just keeps a 2.5s window before
re-releasing instead of staying locked.
Extends the playwright IME smoke with section 6d: clears composing via
the watchdog, fires an IME keydown, then waits past the re-armed
watchdog window and asserts the form submit actually flushes the
textarea. Two new static guards in test_composer_rtl_bidi_attribute
lock the refresh call into both keydown handlers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio/frontend: hide Current password input on first boot
PR #5490 added a third Current password input to the change-password form
so the admin-forced must_change_password reset path could supply a current
password (the bootstrap is empty in that path). The side effect is that the
dominant first-boot UX, which has window.__UNSLOTH_BOOTSTRAP__ present and
silently fed into currentPassword, now shows three visible inputs instead
of the two it had before.
Render the Current password input only when window.__UNSLOTH_BOOTSTRAP__
is absent. The loadBootstrap effect already seeds the password state from
the bootstrap and currentPassword keeps the bootstrap fallback, so
handleSubmit sees the same value as before. On admin-forced resets where
the bootstrap is undefined, the Current password input still appears so
the user can type their actual current password.
Verified end-to-end against a local install via UNSLOTH_STUDIO_HOME +
install.sh --local with Playwright driving the page: bootstrap present
renders two inputs (New, Confirm) and completes change-password into
/chat; bootstrap suppressed via a non-configurable property descriptor
init script renders the three inputs (Current, New, Confirm) and keeps
the #5490 fix intact.
* studio/frontend: add deterministic input-count tests for auth-form
Pure-source pytest covering the change-password JSX contract. No
browser, no Studio boot, no JS toolchain -- runs on any CI runner.
Complements the Playwright probe in tests/studio/playwright_chat_ui.py
which exercises the same contract end to end.
Pins seven invariants with explicit failure reasons:
1. hasBootstrapPassword is derived from window.__UNSLOTH_BOOTSTRAP__
so a future swap to a localStorage flag or prop cannot silently
drift from the backend's _inject_bootstrap contract in
studio/backend/main.py.
2. Exactly one !hasBootstrapPassword conditional exists; multiple
would split rendering into branches these tests cannot reason
about.
3. The Current password input sits inside that conditional, so it
never renders on first boot (the regression PR #5490 introduced
and that this fix reverses).
4. The New password input sits outside it, so it always renders in
change-password mode (admin-forced reset still works).
5. Confirm password: same as New.
6. The change-password JSX subtree declares exactly current /
new / confirm; a fourth password input would almost certainly
break the 2-input first-boot contract.
7. The login JSX subtree declares exactly one password input.
Verified the tests fail loudly on the pre-fix auth-form.tsx at
c4575ca0 (5/7 fail with descriptive reasons) and pass on the fixed
version (7/7).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: soften toast shadow and tighten vertical padding
Sonner's defaults felt heavy in the chat header surface: a 16px
all-around padding made the box taller than the two-line content
warranted, and the 4/12/0.10 drop shadow read as a hard slab
against the light background. Trim padding to 10px vertical
(horizontal unchanged at 16px) and dial the shadow back to
0 2px 6px / 0.08 so the toast still lifts off the surface without
casting a heavy halo.
* studio/frontend: annotate why toast override needs !important
Sonner injects its base styles at runtime from inside its JS bundle,
so a plain cascade tie can lose depending on injection order. One
short comment above the override saves the next reader the dig.
* studio/frontend: boost toast shadow opacity in dark mode
Sonner's lighter 0.08 shadow disappears on the dark popover surface:
quantitative measurement of the shadow band (10px below the toast)
across Chromium / Firefox / WebKit showed only a ~3% luminance drop
vs background, well below perceptual threshold. Bump the dark-mode
opacity to 0.3, matching the existing .shadow-border light/dark ratio
(0.1 -> 0.3) and bringing the toast in line with .menu-soft-surface's
dark-mode shadow (0.28). Light mode keeps the original 0.08.
* studio: install flash-linear-attention and tilelang for Qwen3.5 family
Studio currently only installs causal-conv1d for qwen3.5 / qwen3.6 /
qwen3-next models. Without flash-linear-attention installed alongside
it, transformers' Qwen3.5 fast-path gate stays False and the model
falls back to a pure-PyTorch loop for the GatedDeltaNet layers. In a
60-step run on unsloth/Qwen3.5-2B on B200, this fallback costs ~2.35x
vs the full fast path.
On top of that, FLA dispatches its hottest GDN kernels through a
TileLang backend when tilelang is importable. Adding tilelang plus a
pinned apache-tvm-ffi gives another ~26% on the same workload (4.73
s/step to 3.50 s/step) and is what users have been getting indirectly
when they install mamba-ssm (mamba-ssm transitively pulls tilelang and
pins apache-tvm-ffi<=0.1.9, which is the last working version on
sm_100; 0.1.10 and 0.1.11 crash Triton with misaligned address).
Changes:
* _ensure_flash_linear_attention: pure-Python PyPI install gated on
the same model match set as _ensure_causal_conv1d_fast_path.
* _ensure_tilelang_backend: installs apache-tvm-ffi==0.1.9 and
tilelang==0.1.8 in one pip resolve so the tvm-ffi pin wins over
tilelang's >=0.1.2 constraint. Gated on the Qwen3.5 family only;
SSM models (Nemotron-H, Falcon-H1, Granite-H, LFM2) do not use
FLA's GDN dispatch.
* UNSLOTH_STUDIO_SKIP_TILELANG_INSTALL=1 escape hatch matching the
flash-attn pattern.
* Orchestration block reordered: causal-conv1d -> fla -> mamba-ssm
-> tilelang -> flash-attn (long context).
* 7 new tests covering the new helpers, including SSM-model skip,
skip-env, full Qwen3 family name variants, and graceful pip
install failure.
Combined Qwen3.5-2B-Vision step time on B200 in our bench goes from
5.0 s/step (current Studio: causal-conv1d only) to 3.5 s/step
(causal-conv1d + fla + tilelang), a 1.43x speedup with no notebook
or user code changes required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: accept new grad_norm arg in MLX smoke _on_step callback
The MLX trainer's step callback now passes a ninth positional argument
(grad_norm) per unsloth_zoo/mlx/trainer.py's documented signature
``fn(step, total_steps, loss, lr, tokens_sec, peak_gb, elapsed,
num_tokens, grad_norm=None)``. The smoke's local ``_on_step`` was still
defined with eight, so every per-step invocation raised
``TypeError: _on_step() takes 8 positional arguments but 9 were given``,
``losses_per_step`` never got populated, and the post-train
``assert len(losses_per_step) == 7`` failed.
Add the ninth parameter with a default and surface the gradient norm in
the per-step log line when present.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: retrigger after zoo drift + IPython fixes landed in main
* tests/studio: pin max_grad_value=0 in MLX smoke so max_grad_norm=1.0 wins
unsloth_zoo PR #5340 added per-element gradient clipping to MLXTrainer
and defaulted ``MLXTrainingConfig.max_grad_value = 5.0``. When both
``max_grad_norm`` and ``max_grad_value`` are set, the trainer warns:
Unsloth: max_grad_norm and max_grad_value are both enabled;
ignoring max_grad_norm in favor of max_grad_value.
and silently drops the test's ``max_grad_norm=1.0``. +-5.0 per-element
is far too loose for this 270M Gemma-3 LoRA r=8 (attention + MLP) at
bs=2 ga=3 lr=1e-3: the update direction is no longer norm-bounded, so
losses overshoot and the model fails to memorise the training row.
Reproduced on a CUDA mirror (scripts/cuda_mlx_mirror_sim.py):
norm_1 (max_grad_norm=1.0, no clip): losses 7.64 -> 0.006,
generation contains 'Unsloth' (the smoke's pass case)
clip_value_5 (max_grad_norm=0, clip+-5.0): losses 7.29 -> 8.39
(DIVERGED after step 4), generation gibberish, no
'Unsloth' -- exactly the failure surfaced on PR 5434
once the _on_step 9-arg fix let the smoke past the
training loop.
Pin ``max_grad_value=0.0`` so the smoke uses the same ``max_grad_norm=
1.0`` clipping it was designed against. Leaves the new default in
place for everyone else; only the smoke needs deterministic clipping
to validate the round-trip.
* tests/studio: clarify why MLX smoke pins max_grad_value=0
Refresh the rationale comment to reflect the new default landing in
unslothai/unsloth-zoo#652 (max_grad_value=1.0, not 5.0). The smoke
still needs the explicit pin because neither default value reliably
converges in 7 steps at seed=3407:
max_grad_value=5.0 -- diverges after step 4 (loss 7.3 -> 8.4)
max_grad_value=1.0 -- stalls (loss ~3.2 plateau across seeds)
max_grad_value=0.5/0.25/0.1 -- noisier still
max_grad_norm=1.0 -- cleanly drops loss to <0.01, emits "Unsloth!"
Mention both the historical 5.0 default and the new 1.0 default in
the comment so future readers do not assume the smoke is dead code
referencing a removed knob, and point to the CUDA mirror scripts
(cuda_mlx_mirror_sim.py + cuda_mlx_clip1_vs_norm1.py) for the
empirical evidence.
No behaviour change; comment-only refresh.
* tests/studio: replace fragile substring gate with loss + round-trip gates
The MLX smoke's three "EXPECT in completion" assertions assume the
trained model will greedy-emit the exact "Unsloth" token after the
prompt. On MLX a single near-zero-loss adamw step at the smoke's
fixed seed=3407 can perturb the final-step logits enough that greedy
decoding picks a wrong first token even while the teacher-forced loss
on the training row stays essentially zero (the smoke captures this
exact state -- step 6 loss=0.049, step 7 grad=36.7, step 7 loss=0.17;
completion goes from "Unsloth!" to "5 lbs!"). Reproduced extensively
on CUDA via scripts/cuda_mlx_step7_*.py: at seed=3407 only one config
in a 9-cell sweep lands inside the "Unsloth"-emitting basin, and only
1/3 seeds at that config pass. This is a property of the assertion,
not of save/reload correctness.
Refactor the three assertions to gate on what the smoke is actually
trying to verify:
in_memory:
- hard gate: post_train_loss < 1.0 (training memorised the row).
- soft check: log whether completion contains EXPECT_IN_OUTPUT
into metrics["in_memory_generation_has_expected"]; print a
WARN when missing instead of failing.
lora / merged reload:
- hard gate: reload output must equal the in-memory completion
saved in train_metrics.json. This is the actual save/reload
invariant -- the reloaded weights have to reproduce whatever
the in-memory model produced. Falls back to the original
gibberish gate if train_metrics.json is unavailable.
gguf reload:
- hard gate: llama.cpp produced usable, non-empty output after
the prompt (>=4 chars). llama.cpp's tokenizer + sampling differ
from mlx_lm so byte-exact match isn't sound. Log
gguf_has_expected for visibility.
Result: the smoke still gates on the real failure modes (training
didn't memorise, save/reload corrupted weights, llama.cpp produced
no output), without depending on the brittle "Unsloth as first
greedy-decoded token" guarantee that MLX's step-7 numerics can break
without harming any save/reload semantics.
Cross-version constraint: no transformers / trl API touched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: gate MLX reload on training-row loss, not greedy text
The strict reload assertion (out == in_mem_out) failed on macOS:
in-memory completion was '5 lbs!' and the reloaded completion was
'_________________________'. Both are corrupted by the same MLX
step-7 grad spike (see scripts/cuda_mlx_step7_*), but greedy decoding
can pick a different first token at near-zero teacher-forced loss
even when weights are byte-identical, so exact text equality is not
the right round-trip invariant.
Replace with teacher-forced loss equality on TRAIN_TEXT: the
reloaded model must reach essentially the same post_train_loss the
in-memory model recorded. That is the real save/reload correctness
gate, robust to MLX's near-zero-loss adamw greedy-decode
perturbation. Falls back to a non-empty-body check when
train_metrics.json is missing.
CUDA mirror at this seed converges cleanly to ~0.006 loss; on MLX
post_train_loss < 1.0 still holds via the existing memorisation
gate. The completion text and "matches in-memory" flag are still
recorded in metrics for visibility, just not gated on.
* ci: retrigger Backend CI after transient pwsh-startup timeout
* ci: retrigger MLX dispatch after pytorch CDN DNS flake
* studio: harden FLA + tilelang installers per reviewer feedback
Addresses bot review on #5434:
* Narrow `_ensure_flash_linear_attention` from `_model_wants_causal_conv1d`
(which also matches Nemotron-H / Falcon-H1 / Granite-H / LFM2) to
`_model_wants_tilelang` (Qwen3.5 / Qwen3.6 / Qwen3-Next only). True
SSM families take the mamba_ssm path and never call FLA's GDN
kernels, so installing FLA there is wasted bandwidth.
* Pin both `flash-linear-attention==0.5.0` and `fla-core==0.5.0` and
install with `--no-deps`. Otherwise pip resolves fla-core's
declared `torch>=2.7.0` requirement and may silently upgrade the
Studio venv's torch on environments running torch 2.4/2.5/2.6.
* Skip both installs on Python <3.10 (FLA, fla-core, and tilelang
all declare `Requires-Python: >=3.10`). On older interpreters the
pip install would fail every launch and leave the worker on the
slow torch fallback while still claiming to have set up the fast
path.
* Skip tilelang install on non-Linux platforms. `tilelang==0.1.8`
only publishes Linux x86_64 / aarch64 and macOS arm64 wheels.
Falling back to its 93MB sdist on a Studio worker is undesirable.
* Detect an existing `apache-tvm-ffi` 0.1.10 / 0.1.11 install and
force a reinstall to 0.1.9 with `--force-reinstall --no-deps`.
Previously the import-only probe returned early and left the
broken version in place, which crashes Triton on sm_100.
* Add a 600s timeout to the tilelang and FLA subprocess.run calls,
matching the existing flash-attn install pattern, so a network
hang cannot block the training subprocess indefinitely.
* 13 new / updated tests covering all six guards plus the
pinned-spec, timeout, and force-reinstall code paths.
Total: 21 passing tests (8 original + 13 new / updated).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: address reviewer.py P1/P2 findings on FLA + tilelang installers
Twelve-reviewer aggregated review on this PR flagged several real
correctness bugs in the first hardening pass. Fixes:
P1:
* Add UNSLOTH_STUDIO_SKIP_FLA_INSTALL escape hatch for symmetry
with UNSLOTH_STUDIO_SKIP_TILELANG_INSTALL and the existing
UNSLOTH_STUDIO_SKIP_FLASHATTN_INSTALL.
* Install einops alongside fla-core. `--no-deps` was suppressing
fla-core's only non-torch runtime dep, so on a clean venv
`import fla.modules` raised ModuleNotFoundError even though pip
exited 0.
* Drop --no-deps from the tilelang force-reinstall path. tilelang
needs z3-solver, ml-dtypes, cloudpickle, etc. at runtime;
--force-reinstall --no-deps left libz3.so missing and
`import tilelang` raised OSError on the next training subprocess.
* Skip FLA install when installed torch is below 2.7.0
(fla-core declares torch>=2.7.0). Otherwise users on Studio's
supported torch 2.4/2.5/2.6 stacks get an incompatible FLA
installed silently.
P2:
* Replace bare `except ImportError` probes with helpers that catch
`Exception` so a broken native package (OSError on missing
.so, RuntimeError in __init__, ...) does not kill the worker
before the fallback path can run.
* Tighten the tilelang platform guard from "any linux" to
"linux + machine in {x86_64, aarch64, ...}" so ppc64le / s390x /
armv7 do not fall through and download the 93 MB tilelang sdist.
* Add --only-binary=:all: to the tilelang install command. The
comment already said we never want the sdist; now the pip
invocation enforces it.
* Verify both FLA and tilelang are importable after pip exits 0;
if not, report and continue on the fallback path.
6 new tests bring the suite to 27 passing (was 21).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: pin packaging + triton with FLA --no-deps install
An end-to-end install simulation in a fresh venv caught a real
regression: `fla/utils.py` does `from packaging import version` and
`import triton` at module load, but fla-core's METADATA only declares
einops + torch. With `--no-deps` the worker would land FLA in any
runtime that lacks packaging (e.g. minimal torch builds) and the
post-install import probe would fall back to the torch GDN loop
silently.
Add `packaging` and `triton` to `_FLA_RUNTIME_DEPS` so the install
spec list always carries them. Tests updated to assert both are now in
the install command.
* studio: hook transformers' fast-path gates for just-in-time FLA + causal-conv1d install
The substring-based detection in this PR (`_model_wants_tilelang` /
`_model_wants_causal_conv1d`) is brittle: it depends on what the user
typed for the model name, not on what the architecture actually needs.
Users typing custom model paths, future Qwen3.7 / non-Qwen GDN
architectures, and any model whose author renamed it would silently
fall back to the torch loop.
The correct signal is the one transformers itself uses to gate the
fast path. `transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py`
does at module import time:
if is_causal_conv1d_available():
from causal_conv1d import causal_conv1d_fn, causal_conv1d_update
if is_flash_linear_attention_available():
from fla.modules import FusedRMSNormGated
from fla.ops.gated_delta_rule import (
chunk_gated_delta_rule, fused_recurrent_gated_delta_rule,
)
Wrap both gates so the first call (always at modeling import, before
any forward pass) installs the matching kernel synchronously and
delegates to the original function. Any model whose architecture
queries those gates auto-triggers the install; models that never
query them (Llama, Gemma, dense Qwen, ...) never pay the cost.
Mechanics:
- Split `_ensure_flash_linear_attention` and `_ensure_tilelang_backend`
into `_unconditional` variants (no substring gate, retains python
/ torch / platform / skip-env guards) plus thin substring wrappers
used by the legacy fallback path.
- New `_install_fast_path_hooks(event_queue)` patches both gates on
`transformers.utils.import_utils` AND sweeps `sys.modules` so any
modeling file that already did `from ... import is_X` sees the
wrapper (the local binding survives a module-level reassignment).
- Wrappers clear the original's `lru_cache` before delegating, install
on False, re-check, and short-circuit on subsequent calls.
- Set `UNSLOTH_STUDIO_SKIP_FAST_PATH_HOOKS=1` to fall back to the
substring path.
Verified end-to-end against `transformers.models.qwen3_5_moe`:
PRE_STATE fla=False tilelang=False causal_conv1d=False
HOOK_INSTALLED
Hook fired for is_causal_conv1d_available; installing kernel...
Installing prebuilt causal-conv1d wheel...
Hook fired for is_flash_linear_attention_available; installing kernel...
Installing flash-linear-attention==0.5.0 (with fla-core==0.5.0) for the fast path...
Installed flash-linear-attention for the FLA fast path
Installing TileLang backend (apache-tvm-ffi==0.1.9, tilelang==0.1.8)...
Installed TileLang backend for FLA fast path
MODELING_IMPORT_OK
FAST_PATH_SYMBOLS {"chunk_gated_delta_rule": true,
"fused_recurrent_gated_delta_rule": true,
"FusedRMSNormGated": true,
"causal_conv1d_fn": true,
"causal_conv1d_update": true}
POST_STATE fla=True tilelang=True causal_conv1d=True
Adds 9 new tests covering: install-on-False, skip-on-True, idempotency,
install-failure handling, env-disable, lru_cache clear, sys.modules
rebind, missing-transformers fallback, substring fallback. Total
test count is now 36 (was 27).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: address reviewer.py n=12 findings on the FLA hook path
Eight issues reproduced by parallel reviewers against 6ce495a; all
fixed and covered by regression tests. 45 pytest cases pass (was 36);
end-to-end Qwen3.5_MoE modeling-import drill still loads all five
fast-path symbols.
P1 fixes:
1. TileLang loses the Qwen-family guard on the normal FLA hook path
(10/12 reviewers, reproduced with allenai/OLMo-Hybrid-1B). The
hook unconditionally installed tilelang for any FLA-using model.
- Threaded `model_name` through `_install_fast_path_hooks(event_queue,
model_name)`.
- `_fla_install` now gates tilelang on
`_model_wants_tilelang(model_name)` AND a successful FLA install.
2. TileLang repair `--force-reinstall` (without `--no-deps`) could
replace `torch==2.12.0+cu130` with `torch==2.12.0`. Split repair
into TWO steps:
step 1: `--force-reinstall --no-deps apache-tvm-ffi==0.1.9`
step 2: regular install of tilelang + apache-tvm-ffi
Step 1 surgically downgrades the broken package; step 2 resolves
missing transitive deps (z3-solver, ml-dtypes) without
--force-reinstall, so it never replaces torch.
3. Hook could return True after the installer's deep import probe
failed: when pip exits 0 but `import fla.modules` raises, the old
wrapper re-called `original()` (transformers' metadata check) and
trusted it. Refactored:
- `_ensure_flash_linear_attention_unconditional(...) -> bool`
- `_ensure_tilelang_backend_unconditional(...) -> bool`
The wrapper now uses the installer's bool directly.
4. SSM models (Nemotron-H, Falcon-H1, Granite-H) use
`lazy_load_kernel("causal-conv1d")` and never call
`is_causal_conv1d_available()`, so the hook never fires for them.
The orchestrator now always runs `_ensure_causal_conv1d_fast_path`
outside the hook-mode if/else.
P2 fixes:
5. `_rebind_in_already_imported_modules` invoked transformers' lazy
module `__getattr__` (hundreds of "Accessing X from .models..."
warnings, ~3.4s overhead). Switched to `module.__dict__.get(...)`
which only sees real module-level bindings.
6. TileLang installed even when FLA was skipped (Torch <2.7) or
failed (timeout, post-install probe failed). Now gated on the
installer's bool return.
7. TileLang repair was skipped when FLA was already True but tilelang
missing or apache-tvm-ffi on the broken list. Added an optional
`post_available_fn` to the wrapper; the FLA hook's
`_fla_post_available` runs `_ensure_tilelang_backend_unconditional`
when (model wants tilelang) AND (tilelang missing OR tvm-ffi broken).
8. `_flash_linear_attention_importable()` only checks deep import,
not version. Added `_flash_linear_attention_current()` that
compares against the pinned `flash-linear-attention==0.5.0` /
`fla-core==0.5.0`; older versions trigger `--force-reinstall
--no-deps` so torch stays untouched.
Helpers extracted to keep the surface tight:
- `_pip_install_cmd(*args)` builds `uv pip install` or
`python -m pip install` depending on uv availability.
- `_run_pip(cmd, event_queue, label)` runs a pip command with
timeout / failure handling and a status emission.
Regression tests added:
- test_hook_does_not_install_tilelang_for_non_qwen_fla_model
- test_hook_does_install_tilelang_for_qwen35
- test_tilelang_repair_does_not_touch_torch_cuda_stack
- test_hook_trusts_installer_bool_not_metadata
- test_rebind_does_not_trigger_module_getattr
- test_hook_skips_tilelang_when_fla_install_is_skipped
- test_hook_runs_tilelang_repair_when_fla_already_true
- test_fla_installer_force_reinstalls_when_older_version_present
- test_run_training_process_eagerly_installs_causal_conv1d_in_normal_mode
Existing tests updated for the new `_install_fast_path_hooks` signature
and the two-step tilelang repair flow.
End-to-end re-verified against transformers.models.qwen3_5_moe:
PRE_STATE fla=False, hook fires for both gates, FLA + tilelang +
causal-conv1d install, all 5 fast-path symbols non-None.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix double-install of tilelang on the FLA hook install path
Backend CI surfaced a test-isolation bug introduced by the
post_available_fn mechanism for finding #7. The wrapper ran
`post_available_fn` in BOTH paths (install ran AND gate already True),
but `_fla_install` already chains tilelang on the install path, so the
post-available step then called tilelang install AGAIN.
This was masked locally because tilelang was installed in the
workspace venv (post_available short-circuited on
`_tilelang_importable()` returning True). CI starts with no tilelang,
so the second call actually fired and the mock recorded two calls.
Fix: only run `post_available_fn` when the install path did NOT run.
That preserves the finding #7 semantics (tilelang repair when FLA
already True but tilelang missing or tvm-ffi broken) without
duplicating the chained install on the gate-was-False path.
Also tightened `test_hook_skips_install_when_gate_already_true` to
monkeypatch `_tilelang_importable=True` and
`_installed_tvm_ffi_version=0.1.9` so it stays a pure "no install at
all" test regardless of the venv's actual state.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: retrigger Mac Studio GGUF after transient HF DNS resolve flake
* studio: skip tilelang on HIP / ROCm torch (Strix Halo crash report)
h34v3nzc0dex tested PR 5434 on Strix Halo (gfx1151, ROCm 7.13,
torch 2.11.0+rocm7.13.0) and hit a hard regression:
File ".../fla/ops/common/backends/tilelang/__init__.py", line 92,
in chunk_bwd_dqkwg
File ".../tilelang/jit/kernel.py", line 137, in __init__
File ".../tilelang/tileop/gemm/__init__.py", line 143,
in _select_gemm_instruction
tvm.error.InternalError: Check failed: (0) is false:
Unsupported target for gemm:
hip -keys=hip,gpu -mcpu=gfx1151 ...
`tilelang==0.1.8` ships no HIP GEMM instruction; `_select_gemm_instruction`
raises at lower-time, not import-time. So:
- pip install succeeds
- `import tilelang` succeeds
- `TileLangBackend.is_available()` returns True
- FLA's dispatcher picks TileLang for `chunk_bwd_dqkwg`
- training subprocess dies at first GDN backward, no graceful fallback
The PR's existing platform gate (`_tilelang_platform_supported`)
checked only `sys.platform == "linux"` and `platform.machine()`, both
of which look identical on a ROCm box.
Fix has two layers:
1. INSTALL GATE: new `_torch_has_hip()` helper checks
`torch.version.hip is not None`. `_tilelang_platform_supported`
now returns False on HIP torch, so the install never fires.
2. RUNTIME GATE: even with the install skipped, a user could have
tilelang already present (e.g. venv carried over from a CUDA box).
`_install_fast_path_hooks` now calls
`os.environ.setdefault("FLA_TILELANG", "0")` when HIP is detected,
which is the env-var FLA's `TileLangBackend` already honors. Users
who know they have a HIP-aware tilelang fork can override by
setting `FLA_TILELANG=1` explicitly.
This costs nothing on CUDA (the gate is a no-op when
`torch.version.hip is None`), and removes the crash for AMD users.
The benchmark numbers in the PR description (1.43x on B200 sm_100)
are not affected.
The other halves of the PR are confirmed working on gfx1151 by the
same report:
- `flash-linear-attention 0.5.0` runs at production scale
(B=1 T=8192 H=16 K=128 V=128 and others) with no patches.
- `causal-conv1d` runs at the shapes the fast-path gate cares
about. (A separate Ubuntu 24.04 `--gcc-install-dir` build
workaround is needed for the source-build path; that mirrors
bbf004c's llama.cpp fix and is out of scope here.)
Tests added:
- test_tilelang_platform_unsupported_on_hip_torch
- test_tilelang_install_skipped_on_hip_torch
- test_install_fast_path_hooks_sets_fla_tilelang_zero_on_hip
- test_install_fast_path_hooks_respects_user_fla_tilelang_override
- test_install_fast_path_hooks_does_not_set_fla_tilelang_on_cuda
Total 50 passing (was 45).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: retrigger Windows Studio UI after transient Playwright tab-lookup flake
* studio: auto-discover FLA-using model types from installed transformers
Drop the hand-maintained `_TILELANG_MODEL_SUBSTRINGS` tuple
(qwen3.5 / qwen3_5 / qwen3.6 / qwen3_6 / qwen3-next / qwen3_next)
and derive the allowlist by scanning the installed
`transformers/models/*/modeling_*.py` for `from fla.` imports.
A model "wants tilelang" iff its modeling file imports an FLA op,
which is the same signal `is_flash_linear_attention_available()` is
the runtime test for. The scan happens once per worker subprocess
and is cached for the process lifetime; an empty result (eg
transformers not importable) means "no tilelang pre-install" --
the FLA runtime hook still drives the install via the gate when
the loaded model actually probes it.
Verified against the live installed transformers, the auto-derived
set is {qwen3_5, qwen3_5_moe, qwen3_next}, with `_model_wants_tilelang`
matching the HF Hub names `unsloth/Qwen3.5-2B`, `Qwen/Qwen3.5-MoE-A3B`,
`mlx-community/qwen3-next-80b`, and correctly rejecting Llama,
Mistral, Nemotron-H, Falcon-H1, etc. Future GDN models (Qwen3.7,
OLMo-Hybrid-FA, ...) are picked up automatically once they ship in
transformers; no further worker edits needed.
Also trim docstrings / comments through the FLA / tilelang / HIP /
hook block: constants get 1-line trailing comments, function
docstrings collapse to 1-3 lines, and the fast-path-hooks banner
shrinks from a 27-line block to 4 lines. The file drops from 2847
to 2630 lines without losing the load-bearing WHY notes
(--no-deps protects torch; `__dict__.get` avoids lazy-module
__getattr__; two-step tvm-ffi repair keeps torch off the dep
graph; HIP setdefault disables FLA's TileLang dispatch even with
tilelang already installed).
7 new tests (50 -> 57 total): discovery returns only FLA-using
model_types; discovery cache reuse; missing transformers handled;
OSError on a modeling file is non-fatal; `_model_wants_tilelang`
matches real HF repo names across separator variants; empty
discovery -> always False; normalization across `-`, `.`, `/`,
space.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: hermetize the non-allowlist hook test against transformers 5.4.0+
transformers 5.4.0 added `olmo_hybrid` as an FLA-using model_type, so
the auto-discovered allowlist now includes it -- and the test's prior
choice of `allenai/OLMo-Hybrid-1B` as a "non-Qwen FLA-only" example
became an allowlist member. CI on Python 3.11 / 3.13 caught this.
Swap to a guaranteed-not-in-allowlist fake model_name AND patch
_discover_fla_model_types to a known {qwen3_5, qwen3_5_moe, qwen3_next}
set so the test stays valid as upstream transformers adds new
FLA-using architectures.
Renames the test to reflect the actual semantic under test:
"outside-allowlist -> no tilelang".
* ci: retrigger Windows Studio API after llama.cpp prebuilt staging WinError 5 flake
* tests: move MLX smoke gate changes to dedicated PR #5537
The seven MLX smoke commits in this PR's history (_on_step grad_norm,
max_grad_value pin, loss + round-trip gates) are unrelated to the
FLA / tilelang work. They now live in #5537 so this PR's diff is
limited to the studio worker installer changes.
Net effect on tests/studio/run_real_mlx_smoke.py vs main: zero.
* studio: friendlier install banners (drop hook / gate-name jargon)
User-visible status text now reads:
Installing flash-linear-attention==<ver> for faster training...
Installing TileLang==<ver> for faster training...
Installing causal-conv1d for faster training...
Installing flash-attn for faster training...
Removed the transient "Hook fired for is_flash_linear_attention_available;
installing kernel..." banner — the install banner that immediately follows
already tells the user what is happening, in plain English.
The internal logger.info messages (server-side log) still carry the
gate names + "Hook fired ..." for debugging.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: add dismissable toasts with corner close button
- Enable Sonner's close button globally on the Toaster, so every toast
(model load progress, model loaded, load failure, etc.) gets an X that
users can click to dismiss without waiting for the auto-dismiss timer.
This matches the Claude desktop notification behavior.
- Drop the per-toast 'closeButton: false' overrides in the model load
runtime so they inherit the global default. The existing 'onDismiss'
handler already flips state to show an inline header status, so the
X on the loading toast hides the toast without canceling the load
(Cancel still aborts).
- Pin the close button to the top-right corner inside the toast box.
Overrides Sonner's left-side default placement, outside-corner
translate, and hardcoded 'top: 0'. Top is set via a small rule in
index.css because Sonner does not expose it as a CSS variable.
- Add a small offset on the Toaster so toasts sit at the chat header
line, shifted left of the parameters and settings buttons on the
right edge instead of stacking on top of them.
- Bump the post-load success and failure durations from 2s and 5s to
8s so users actually have time to read and click the new close X
before the toast auto-dismisses.
* studio: explicit boolean for closeButton prop to satisfy biome
* studio: keep close button X visible in dark mode
Two defensive fixes for the dark-mode close button visibility:
- Use resolvedTheme so sonner's data-sonner-theme always matches the
class next-themes applies to <html>. Passing theme can be 'system',
which makes sonner resolve via its own media query; that can disagree
with next-themes (Tauri webview, hydration races, OS quirks), leaving
CSS vars dark while sonner still applies its light close-button colors
(dark X on dark background).
- Bump the close-icon stroke from sonner's default 1.5 to 2.25 so the X
is readable on a 12x12 svg sitting on dark backgrounds.
---------
Co-authored-by: shimmyshimmer <datta_mike@hotmail.com>
* studio/frontend: make toast and inline error text selectable and copyable
Sonner toasts and the inline model-load error in the chat header were
showing copyable content (backend tracebacks, model-load failures, log
lines) that users could not actually select with the mouse.
Two underlying issues:
1. Sonner's swipe-to-dismiss handler calls `setPointerCapture` in
`onPointerDown`, which preempts the browser's text-selection
gesture. The capture only happens when `dismissible` is true. CSS
alone cannot work around this.
2. The inline model-load error truncated with `text-overflow: ellipsis`
and parked the full string in a native `title=` tooltip, which
browsers render as an OS tooltip that cannot be selected.
Fixes:
- New `@/lib/toast` wrapper that defaults `dismissible: false` on every
toast (callable plus `.success` / `.error` / `.info` / `.warning` /
`.loading` / `.message` / `.custom`). API is identical to sonner's
`toast`, so the 18 call sites just swap their import path. Callers
can opt back into swipe-to-dismiss with `dismissible: true`.
- `<Toaster>` sets `swipeDirections={[]}` to make the intent explicit.
- `index.css` forces `user-select: text` on toast text content and
keeps `user-select: none` on toast buttons.
- New `<CopyableErrorChip>` component replaces the truncated inline
error in the chat header. The chip shows the truncated message
inline and opens a popover with the full, wrap-friendly, selectable
message and a one-click Copy button.
Toasts still auto-dismiss after their `duration`, close buttons and
action buttons still work.
* studio/frontend: tighten code comments in selectable-toast change
* studio/frontend: address PR review on selectable-toast change
Three review-driven fixes:
1. CopyableErrorChip clears the copied->reset setTimeout on unmount via
a useRef + useEffect cleanup so setState cannot fire on an unmounted
component.
2. index.css restricts `cursor: text` to text-bearing toast nodes
(`[data-title]`, `[data-description]`, `p`, `span`). The toast
container keeps its default cursor and no longer pretends to be an
editable surface. `user-select: text` still applies to the full toast
tree so a drag-select starting on padding still works.
3. Toast wrapper now also injects `dismissible: false` into the second
argument of `toast.promise(p, data?)`, covering the loading /
success / error toasts created from a single promise call. Explicit
`dismissible: true` in the data continues to win.
A fourth review point asked us to drop the wrapper and instead pass
`toastOptions={{ dismissible: false }}` to <Toaster>. Sonner v2.0.7's
Toaster only forwards `duration`, `className`, `descriptionClassName`,
`closeButton`, `style`, `unstyled`, `classNames`, `cancelButtonStyle`,
`actionButtonStyle`, and `closeButtonAriaLabel` from `toastOptions`
(see index.mjs lines 1144-1164). `dismissible` is not forwarded, so the
global-option approach is a runtime no-op (verified empirically across
Chromium / Firefox / WebKit). Wrapper is required.
* studio/frontend: drop chip aria-label override so message reads via SR
The CopyableErrorChip trigger set a fixed `aria-label`, which overrides
the visible message in the accessibility tree. Inside the chat header's
`role="status"` region this caused screen readers to announce the
generic label instead of the actual model-load error, a regression
versus the old plain-text status div.
Removed the `ariaLabel` prop and the default override. The button's
visible message text is now its accessible name, so the full
(untruncated) error is announced. Truncation stays purely visual via
CSS. Caller in chat-page.tsx dropped the prop too.
Added a Playwright assertion that the trigger's accessible name
contains the error message across Chromium, Firefox, and WebKit.
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
* studio/frontend: grow chat composer to 16 rows and inset scrollbar
Raise the composer textarea cap from 6 to 16 rows so the input keeps
expanding as you type longer prompts. Also nudge the textarea in with
mt-2 / mr-3 so the internal scrollbar no longer sits flush against
the rounded edges of the chat composer surface.
* studio/frontend: lower composer cap from 16 to 12 rows
Keeps the composer growing past the previous 6-row cap while staying
conservative enough that a fully expanded textarea does not cover the
scroll-to-bottom button or a large slice of recent messages.
* studio/frontend: use symmetric mx-3 inset on composer-input
Replaces mr-3 with mx-3 (and width calc(100%-1.5rem)) so the textarea
sits inset from both edges of the chat composer surface. Keeps the
scrollbar tucked in regardless of writing direction: LTR scrolls on
the right, RTL scrolls on the left, and both edges are now ~16px in
from the surface (4px surface px-1 + 12px mx-3).
The spinner inside ThreadWelcome subscribed to the global
generatingStatus from chat-runtime-store, so any in-flight warmup
(or stale leak from a prior run) surfaced Generating on the empty
Chat with your model surface, even while the user was still typing.
On the normal path the welcome view is gone the moment a message
is submitted, and the assistant bubble already renders its own
per-message GeneratingIndicator. Remove the welcome-screen spinner,
its component, and the now-unused LoaderIcon import.
* studio: register /settings route that opens the settings dialog
Navigating to /settings used to render Not Found because the route
was never registered. The settings dialog only opened via the user
menu, so /settings was a broken deep link if shared. Add a route
that calls useSettingsDialogStore.openDialog() and redirects to the
post-auth landing page so the modal appears on top of the chat.
* studio: harden Connections dialog provider sync and allow manual model IDs
Two related fixes for the Connections panel.
1. Keep localStorage providers when the server returns an empty list.
The dialog used to sync from /api/providers/ on mount and unconditionally
overwrite the Zustand provider store with the server result. When the
server had no enabled configs but the local store had entries (legacy
users, fresh dev installs, or providers created via earlier paths),
opening the dialog silently wiped them. The model picker reads from the
same store, so the chat header reverted from 'gpt-4o . OpenAI' to the
raw 'external::openai-1::gpt-4o' key. Treat the server as authoritative
only when it actually has rows; otherwise keep the local view.
2. Accept manual model IDs alongside the live catalog for remote-mode
providers (DeepSeek, OpenAI, etc.). Previously the only way to save was
to load the available-models catalog via a live API call, which fails
in air-gapped setups, behind 502s, or when the user already knows the
exact model ID. Add a Textarea fallback in the same render block, and
relax the validation to accept manual IDs even when availableModels is
empty. The validation message now points users at the manual path.
* studio: restrict manual model ID entry to openrouter among remote providers
Address review feedback: major remote providers (openai, anthropic,
gemini, mistral, cohere, deepseek, ...) expose large per-model
parameter surfaces that differ across models, so accepting pasted
model IDs leads to mismatched parameter expectations and frustrating
runtime errors. Keep their catalog curated by hiding the manual
textarea and falling back to the prior 'Load available models first'
validation toast for them.
OpenRouter drops unsupported parameters server-side, so manual entry
remains useful there; keep the textarea and the union save path for
it. Custom and curated backends already gated via isCustomProvider /
isCuratedModelList and continue to require manual entry as before.
* studio: shorten code comments in chat-providers-dialog.tsx
Trim three multi-line comment blocks to single lines per review.
* Studio: gate image input on a usable mmproj for GGUF vision models
* Improve image gating and model capability sync
Tighten image-handling and model capability syncing across the chat flow. Key changes:
- chat-adapter: Replace per-message current-user image check with a simpler gate that blocks if ANY image is present in the outbound payload when the selected model cannot handle vision. Show the toast reason and flip the per-thread running flag on→off to avoid hanging wait promises before throwing.
- shared-composer: Simplify and correct image-attachment gating for single vs compare modes. Use an attach-time gate that defers to send/ensureModelLoaded in compare mode, introduce attachUnavailableReason, and only block immediately for single-mode. Remove an unused models selector.
- shared-composer: Sync the runtime models[] entry with the response from ensureModelLoaded so UI/send gates read fresh capabilities (isVision, isGguf, isAudio, audioType, hasAudioInput). This addresses catalog lag (e.g., GGUF mmproj arriving after the catalog snapshot).
- UX tweak: the file-picker button no longer outright blocks on image availability; addFiles still filters images per-file and toasts appropriately.
These changes prevent mid-stream server rejections, avoid deadlocks, and ensure model capability checks are accurate when attaching images or audio.
* studio: only pass --mmproj to llama-server when effective_is_vision
When a text-only GGUF (static is_vision=False) was paired with a
family-matching mmproj path, the launcher appended both --mmproj and
--spec-default, leaving llama-server in an inconsistent state while
Studio reported is_vision=False. Gate the --mmproj flag on
effective_is_vision so the launch command tracks the runtime
capability the rest of Studio sees.
* studio: reject image content in streaming /v1/responses for non-vision GGUF
_responses_stream forwards the OpenAI request body directly to
llama-server's /v1/chat/completions, bypassing the image-vs-vision
guard that openai_chat_completions enforces for the wrapped path.
Add the same check at the top of the streaming entry point so an
SDK client that posts an image to a non-vision GGUF receives a
typed 400 instead of an opaque downstream error.
* studio: gate external chat providers in the image input helper
External selections (cohere, deepseek, mistral, openrouter, ...) live
in externalProviders, not in runtime.models[], so activeModel is
undefined for them and the helper short-circuited to allow. Result:
images attached to a non-vision external chat model were dropped
silently downstream instead of rejected up front.
Add providerTypeSupportsVision to external-providers.ts (false for
known text-only providers, true for known vision-capable ones, null
for unknown / custom self-hosted) and thread externalSupportsVision
+ externalModelLabel through the helper. shared-composer.tsx,
runtime-provider.tsx (VisionImageAdapter.add), and chat-adapter.ts
pre-stream gate all resolve the provider type and pass it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/worker): inject --gcc-install-dir for HIP source builds on Ubuntu 24.04
On Ubuntu 24.04 + ROCm clang-20, the HIP source-build fallback in
`_install_package_wheel_first` (causal-conv1d, mamba-ssm source fallback,
flash-attn source fallback) dies at:
/opt/rocm-X.Y/lib/llvm/lib/clang/20/include/__clang_hip_runtime_wrapper.h:112:10:
fatal error: 'cstdlib' file not found
Root cause: clang-20 picks the highest-numbered /usr/lib/gcc/x86_64-linux-gnu/<N>
runtime dir by default. On 24.04 that's gcc-14, whose runtime objects ship in
the gcc-14 package but whose C++ headers (/usr/include/c++/14) come from
libstdc++-14-dev — NOT in the default apt set. libstdc++-13-dev IS in the
default set, so /usr/include/c++/13 exists. clang has no way to discover
that asymmetry and the build fails.
Fix: new `_hipcc_gcc_install_dir()` helper iterates gcc 14 → 11 and returns
the first /usr/lib/gcc/x86_64-linux-gnu/<N> dir where BOTH the runtime AND
/usr/include/c++/<N> exist. The HIP branch of `_install_package_wheel_first`
appends `--gcc-install-dir=<that path>` to HIPCC_COMPILE_FLAGS_APPEND before
invoking pip. Respects an existing `--gcc-install-dir` in the env var
(user-set takes precedence); preserves any other flags the user has set
(appends to the end rather than overwriting). No-op on non-HIP, non-Linux,
non-x86_64.
Mirrors the same fix bbf004c added to studio/setup.sh for the llama.cpp HIP
build branch (#5301), but via env var since pip-driven source builds can't
take CMake flags directly.
Verified on Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151) / Ubuntu 24.04 /
ROCm 7.13 nightly: `_hipcc_gcc_install_dir()` returns
`/usr/lib/gcc/x86_64-linux-gnu/13`, which matches the manual workaround
that already lets `pip install causal-conv1d` succeed on this hardware.
Tests added (8 new in test_training_worker_flash_attn.py):
- test_hipcc_gcc_install_dir_picks_highest_with_headers
- test_hipcc_gcc_install_dir_picks_14_when_headers_exist
- test_hipcc_gcc_install_dir_returns_none_when_no_match
- test_hipcc_gcc_install_dir_returns_none_on_non_linux
- test_hipcc_gcc_install_dir_returns_none_on_non_x86_64
- test_install_injects_gcc_install_dir_on_hip_source_build
- test_install_appends_to_existing_hipcc_compile_flags
- test_install_respects_user_gcc_install_dir
- test_install_does_not_inject_env_on_cuda
Per @danielhanchen's suggestion in
https://github.com/unslothai/unsloth/pull/5434#issuecomment-4469980122
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: apply gemini-code-assist suggestion on _run_kwargs env handling
Use _run_kwargs.get("env", os.environ).copy() + key-mutation instead of
rebuilding env from os.environ directly. Today both forms are equivalent
(no earlier code in _install_package_wheel_first sets _run_kwargs["env"]),
but the .get().copy() pattern survives any future env modification added
upstream of this block without silently throwing it away.
No behavioural change; tests already assert the final HIPCC_COMPILE_FLAGS_APPEND
value, not the env-construction pattern.
Per https://github.com/unslothai/unsloth/pull/5517#discussion_r... (gemini-code-assist[bot])
---------
Co-authored-by: h34v3nzc0dex <h34v3nzc0dex@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: extend offline DNS auto-detect to inference parent + training
#5505 fixed the GGUF/llama-server load path. Studio still has two
adjacent code paths that burn ~30-60s of soft-failed timeouts before
the worker subprocess starts when DNS to huggingface.co is dead and
the model is already in the local HF cache.
Inference parent process (routes/inference.py:load_model):
* ModelConfig.from_identifier now runs inside _hf_offline_if_dns_dead
so the LoRA-detect hf_model_info call and the urllib config probes
in utils/transformers_version.py short-circuit when DNS is dead.
* utils/models/model_config.py: extracted the inline HF_HUB_OFFLINE/
TRANSFORMERS_OFFLINE check used by list_gguf_variants and
detect_gguf_model_remote into a shared _env_offline() helper, then
reused it to gate the LoRA-detect hf_model_info call.
* utils/transformers_version.py: _check_tokenizer_config_needs_v5 and
_check_config_needs_550 now early-return False when offline instead
of issuing a 10s urllib.urlopen against huggingface.co/raw/main.
Training worker (core/training/worker.py:run_training_process):
* Add the same 2s DNS probe used by core/inference/worker.py at the
top of the training subprocess. On failure, set HF_HUB_OFFLINE,
TRANSFORMERS_OFFLINE, and HF_DATASETS_OFFLINE before the rest of
the subprocess imports torch/transformers/unsloth, so every
from_pretrained, snapshot_download, and load_dataset call below
resolves from cache. Scope is per-subprocess; the orchestrator
always spawns a fresh worker per training run.
Training trainer (core/training/trainer.py:load_model):
* Skip the proactive hf_model_info gated-repo probe when _env_offline()
is true. The API is unreachable anyway, and a gated model that is
already cached is exactly the scenario the user is trying to train
against. from_pretrained surfaces the real error if access is
actually denied.
Tests (tests/test_offline_inference_parent.py, 7 new cases):
* _env_offline truthy/falsy parsing across HF_HUB_OFFLINE and
TRANSFORMERS_OFFLINE.
* transformers_version urllib short-circuit when offline.
* LoRA detect hf_model_info skip when offline.
Existing tests/test_offline_gguf_cache_fallback.py still passes
(26 cases) because the inline env check was extracted, not changed.
* tests: prefer real httpx over stub in offline-test files
The studio test stub convention only included the 6 httpx exception
names that existed callers needed. Newer huggingface_hub (1.15+)
imports HTTPError, Response, Request, HTTPStatusError, AsyncClient,
and more at module import time. When httpx is truly absent the stub
chase becomes a treadmill.
Use the real package when installed (the CI install list already
includes httpx, so this is the production environment). Fall back to
the stub only when httpx is genuinely missing.
No code under test changes.
* studio: detect cached LoRA adapters offline; tighten test
Two follow-ups from the review pass on #5512:
* ModelConfig.from_identifier no longer skips the remote LoRA-detect
hf_model_info call when _env_offline() is true. huggingface_hub
short-circuits the call via OfflineModeIsEnabled in ~0ms when
HF_HUB_OFFLINE is set, so the original 25s concern was moot once
routes/inference.py wrapped the call in _hf_offline_if_dns_dead.
Skipping the API meant users with a cached LoRA adapter
(adapter_config.json on disk) got is_lora=False and the load
failed. After the API call (which raises fast offline) a new
cache-fallback walks the HF cache snapshot for adapter_config.json
via the existing _iter_hf_cache_snapshots helper.
* test_hf_model_info_not_called_when_offline replaced. The old test
raised AssertionError inside production code that catches Exception,
so it passed even if the call happened. New tests use MagicMock and
assert call_count >= 1, plus a fixture that stages a fake HF cache
with adapter_config.json to verify the offline cache detection.
Test count goes from 7 to 8 in test_offline_inference_parent.py.
Combined with test_offline_gguf_cache_fallback.py: 34 pass in 9.75s.
* Fix/adjust offline training DNS probe per PR #5505 review
Same fix as #5505's _probe_dns_dead refactor: run gethostbyname on a
daemon thread with join timeout so concurrent sockets in the parent
interpreter never inherit a process-wide socket.setdefaulttimeout
mutation. Adds a static-pin regression test that the inference parent
file does not regress on this.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose code comments per review feedback
Shorten the longer explanatory comments added by this PR while keeping
the WHY of each non-obvious branch:
- trainer.py: collapse the 5-line proactive gated-check comment.
- training/worker.py: trim the offline auto-detect preamble and the
"logger isn't configured" note.
- routes/inference.py: shorten the DNS-probe wrap rationale.
- transformers_version.py: collapse the two urllib short-circuit notes.
- model_config.py: shorten the LoRA detect + cache-fallback notes.
- tests/test_offline_inference_parent.py: tighter module docstring,
trim class docstrings, drop multi-line explainer comments inside the
tests; behaviour and coverage unchanged (9/9 tests still pass).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: warn when llama.cpp prebuilt is at least 3 days behind
Layered on #5528. Generalises the MTP-specific staleness warning to
every llama.cpp prebuilt update, not just the ones that add MTP. If
the installed prebuilt is at least 3 days old AND its tag differs
from the latest published tag on the helper release repo (default
unslothai/llama.cpp), Studio nudges the user to run
"unsloth studio update".
How it works
Reads the install marker UNSLOTH_PREBUILT_INFO.json that
install_llama_prebuilt.py already writes to install_dir. The marker
carries the installed tag, the helper repo, and an installed_at_utc
timestamp. Studio compares those against the latest published tag
from the GitHub releases API for the helper repo.
GitHub fetch is cached at two levels:
- Process-level memo for /status hot path.
- Disk-level cache (24h TTL) at ~/.unsloth/studio/cache/llama_cpp_freshness/
so cold-start Studio launches do not always hit the API.
On a transient fetch failure (offline, rate-limited) we keep the
last-good disk value alive rather than poisoning the cache with None.
The check fails open: if anything is missing (marker, timestamp,
GitHub response), stale stays False so users never see a misleading
banner.
Surfaced in two places
1. Startup banner (logs + stderr) in main.py:lifespan(), alongside the
MTP capability probe added in #5528. Single line, e.g.:
WARNING: llama.cpp prebuilt is 5 days behind: installed b9190,
latest b9300. Run "unsloth studio update" to refresh.
2. /api/inference/status now returns:
llama_cpp_prebuilt_stale: bool
llama_cpp_installed_tag: str | None
llama_cpp_latest_tag: str | None
so the frontend can render a banner / popup with the actual tag
delta the user is missing.
3-day threshold
Mirrors the typical Unsloth llama.cpp release cadence. Anything
shorter would nag users who restart Studio at the wrong moment;
longer leaves real bugs sitting on the user's machine. Configurable
via the threshold_days kwarg if a future call site wants a different
window.
Tests
17 new cases in tests/test_llama_cpp_freshness.py cover marker
discovery in both cmake and root install layouts, missing / invalid
marker, GitHub fetch caching across process restarts (disk cache hit
after the in-memory cache is reset), the stale / not-stale decision
matrix (tag mismatch + age threshold), fail-open behaviour when
GitHub is unreachable, custom threshold, singular/plural day in the
warning string, and unparseable installed_at_utc. The broader
205-test inference regression suite still passes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Roland Tannous