PR 5549 cycle-15 codex review surfaced two P2 issues:
1. Trailing-plan regex matched "Now let me know if you want another
example" (the closer "let me know" was negative-lookahead-guarded
only on the bare alternative, not on "now let me"). Refactor to a
single `(?:now\s+)?let me(?!\s+know\b)` alternative so both
prefixes share the same guard.
2. Anthropic adapter opened a new text block immediately on the
iteration boundary marker. If the reprompted iteration started
with a tool_call rather than text, that just-opened text block was
immediately closed, emitting a zero-length text content block
between intent text and tool_use. Defer the open: close the
current block + bump block_index + reset cursor at the boundary,
and let `_handle_content` lazy-open when real text arrives.
Codex 02:42Z on #5549 caught that flagging the post-tool empty-status
event with boundary=True (cycle-12 commit 610c387) double-handles the
cursor reset in the Anthropic streaming path.
AnthropicStreamEmitter._handle_tool_end already:
- closes the open tool_use block,
- emits tool_result,
- increments block_index,
- opens a fresh text block, and
- resets _prev_text = "".
When llama_cpp.py then yielded boundary=True on the very next event,
_handle_boundary fired _close_block + _open_text_block on that freshly
opened (still-empty) text block. Result: every tool call produced a
spurious content_block_stop + content_block_start pair before the
post-tool model text streamed.
Fix:
- llama_cpp.py: drop boundary=True from the post-tool status emit
(line ~4768). Keep boundary=True only at the auto-continue site
(line ~4500), which has no preceding tool_end to do the cursor
work.
- routes/inference.py OpenAI-compat tool stream: mirror the
Anthropic semantics by resetting prev_text on BOTH tool_start AND
tool_end, so the post-tool empty-status no longer needs to do it.
Add backend/tests/test_anthropic_messages.py::TestAnthropicStreamEmitter::
test_post_tool_empty_status_does_not_double_close as a regression
test: content -> tool_start -> tool_end -> empty status -> content
must not bump block_index past tool_end's increment, and the post-tool
content must land in the text block tool_end opened.
95 tests pass across the anthropic + trailing-plan suites.
Codex P2 on #5549 flagged that the cycle-5 cursor-reset (efa43c4) keyed
on every empty-status event, but generate_chat_completion_with_tools
emits empty status events in five places, only two of which are real
iteration boundaries:
- Line 4497: emitted right before `continue` after a re-prompt /
auto-continue. The next iteration starts a fresh assistant turn.
BOUNDARY: reset cursor.
- Line 4766: emitted right before `continue` after a tool call. The
next iteration regenerates with tool results in history. BOUNDARY:
reset cursor.
- Line 4501: emitted at metadata-yield after normal streaming. Stream
is about to end, no new iteration follows. NOT a boundary.
- Line 4584: emitted in DRAINING-no-tool-call fallback path. Stream
is about to end with buffered content_accum. NOT a boundary.
- Line 4794: emitted at the final exit of the generator. NOT a
boundary.
Treating all five as boundaries gave every Anthropic-streaming response
an extra content_block_stop + content_block_start pair around its final
text and around every tool call.
Fix by tagging the two real boundary sites with `"boundary": True` and
tightening both the Anthropic emitter (`anthropic_compat.py`) and the
OpenAI-compat tool path + Anthropic non-streaming path
(`routes/inference.py`) to reset the cumulative-text cursor only when
that flag is set. Plain empty-status events keep their existing badge-
clear semantics on the frontend (`tool_status` SSE with content "").
Add two regression tests in
`backend/tests/test_anthropic_messages.py::TestAnthropicStreamEmitter`:
- test_boundary_flag_closes_block_and_resets_cursor: a boundary=True
status closes the open text block and the next content delta
streams from zero.
- test_empty_status_without_boundary_does_not_close_block: a plain
empty status leaves block_index unchanged and the next content
delta is diffed against the previous text length.
107 tests pass across the four anthropic + trailing-plan test files.
While drafting backend/tests/test_trailing_plan.py for the changes
landed in b4e0985, the new tests surfaced a deeper false-positive the
earlier regex tightening missed.
For input "Let me explain:\n1. The function returns 42.\n\nThat's the
answer." the previous `(?:\s*(?:[-*•]|\d+\.)\s+[^\n]+\n?)+` allowed the
regex engine to backtrack and treat the in-prose "42." substring as a
second list-item marker: iter 1 consumed "1. The function returns 4"
and iter 2 consumed "2.\n\nThat's the answer." (with `\s+` greedily
crossing the empty-line break). The pattern then satisfied `\s*\Z` and
the buffer fired a spurious `Continue.` retry on what was already a
fully-formed answer.
Tighten the per-item boundary:
- `[ \t]*` before the marker (no newlines): forces the marker to sit
at the start of its own line. A mid-prose "42." cannot satisfy this
because the engine cannot rewind past the preceding `\n` without
invalidating the previous iteration's `[^\n]+\n` close.
- `[ \t]+` between the marker and content: blocks an `\s+`-driven
cross-newline reach into a closing paragraph.
- `(?:\n|\Z)` at end of item: a real line break OR end of buffer.
Preserves the "list at EOB with no trailing newline" case while
eliminating the backtrack route.
Land backend/tests/test_trailing_plan.py at the same time, covering:
- `_TRAILING_PLAN_INTENT` "let me know" closer exclusion (cycle-3 fix).
- `_TRAILING_PLAN_LIST` correctly fires on genuine trailing lists
(dash, asterisk, unicode bullet, numeric).
- `_TRAILING_PLAN_LIST` does NOT fire on list + closing paragraph,
list + closing sentence, list embedded mid-text, or the "42."
in-prose digit case.
- `_TRAILING_PLAN_COLON` fires on bare trailing intent-colons only.
- `_trailing_plan_hit` composite cases.
- The 600-char window slicing.
31 cases, all pass. Pins the regex behaviour against future regressions
inside the repo (the prior pin script lived only in the probing
workspace, not the studio tree).
Codex P2 review on #5549 surfaced two related risks in the auto-continue
plumbing:
1. `_TRAILING_PLAN_LIST` was compiled with `(?ims)`. The `m` flag makes
the terminal `\s*$` match end-of-line, so a complete answer like
"Here's my plan:\n- a\n- b\n\nDone, that should work." still matched
the list-block sub-pattern and tripped a spurious `Continue.` retry.
Drop the `m` (and the unused `s`) flag and re-anchor with `\Z` so the
list pattern only fires when the list is genuinely the last thing in
the buffer.
2. The agent loop pre-reserved `_MAX_REPROMPTS + _MAX_CONTINUES` (= 6)
extra iterations on top of the caller's `max_tool_iterations`
unconditionally. That weakens the caller-provided budget: a turn
that never trips the reprompt or continue path could still run up
to N+6 full iterations and execute their tool calls.
Switch the bound to a dynamic cap that grows only as reprompts /
continues are actually consumed: `iteration < max_tool_iterations +
_reprompt_count + _continue_count`. With both counters at zero the
loop honors the caller cap exactly; once a continue or reprompt
fires it earns its own slot back.
Implemented with `itertools.count()` so the existing `continue`
statements in the loop body keep their semantics.
Regex behaviour pinned by `scripts/r6_trailing_plan_regex_test.py`
(updated separately for the new list-tail case).
Codex P2 on #5549 flagged that the auto-continue branch yields only
a `{"type":"status","text":""}` event between turns; the Anthropic
streaming emitter (`AnthropicStreamEmitter`) and the non-streaming
tool path (`_anthropic_tool_non_streaming`) both ignore `status`
events, so their cumulative-text cursor still holds the previous
turn's full length when the continuation starts streaming. Shorter
continuations get dropped entirely and longer ones lose their prefix.
Treat the empty-text status as an auto-continue boundary in both
paths:
- `AnthropicStreamEmitter`: close any open text block, open a fresh
one (matches the `tool_end` reset pattern), and clear `_prev_text`.
- `_anthropic_tool_non_streaming`: clear `prev_text` so the next
`content` event's diff baseline is empty.
Non-empty status events (tool progress text) keep their existing
no-op semantics.
Codex review on #5549 flagged that endings like
"If you need anything else, let me know." match the trailing-plan
intent pattern (the regex matches "let me <anything>." at end of
buffer). On a finished, user-facing closing this fires the auto-
continue branch up to three times, costing latency / tokens and
appending unrelated text after the response.
Add a negative lookahead so "let me" only counts as a mid-plan signal
when it is NOT immediately followed by "know". Other intent phrases
("now let me", "i'll now", "i'm going to", "i will now", "let's now")
already require a planning verb so they are unaffected.
Verified against `scripts/r6_trailing_plan_regex_test.py`: closing
"let me know" variants no longer match; "let me clone/check/run …"
still does.
The existing intent-signal re-prompt fires only when tools are armed
and the response is short. Models often stop mid-plan in other shapes
too: a trailing "Let me clone the repo.", a "Let me ...:" header
followed by a numbered list, or a bare trailing colon. When this
happens the turn ends with the structured workload only partially
delivered.
Add a neutral "Continue." nudge that runs alongside the tool-coercive
re-prompt:
- _TRAILING_PLAN_INTENT, _TRAILING_PLAN_LIST, _TRAILING_PLAN_COLON
cover the three observed shapes, scanned over the last 600 chars.
- _trailing_plan_hit() returns True if any of them match.
- _MAX_CONTINUES (3) is independent of _MAX_REPROMPTS so the two
paths cannot starve each other.
- _continue_count threads through the agentic loop; auto-continue
fires with "Continue." regardless of tool armament.
Regex tested against the patterns above plus negative controls
(complete sentences, benign "let me" earlier in the buffer) before
landing.
* studio: read Playwright default model from defaults.py without importing it
The Playwright Chat UI job installs Studio with --no-torch and does not
have structlog. Importing core.inference.defaults pulls in
core/inference/__init__.py (eager orchestrator -> structlog) and
defaults.py's own `import utils.hardware.hardware as hw` (also
structlog), so the test died before the first page action.
Read DEFAULT_MODELS_GGUF as a literal via ast.literal_eval. Zero side
effects, no new test deps, the EXPECTED_DEFAULT_MODEL override still
wins.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: engage draft-mtp on vision MTP GGUFs
The draft-mtp auto-promotion in LlamaCppBackend.load_model was gated on
not effective_is_vision, and the spec-emit branch repeated the same
guard. Every Unsloth -MTP GGUF repo ships an mmproj projector, so
effective_is_vision was always True for those repos and the MTP speedup
silently never engaged out of the box.
llama.cpp #22673 explicitly states MTP is compatible with vision input.
The bundled b9204 server happily loads both: a manual run with
--mmproj ... --spec-type draft-mtp --spec-draft-n-max 6 logs
"loaded multimodal model" followed by
"adding speculative implementation 'draft-mtp'".
Drop the vision gate from both sites and rewrite the matching short
circuit in _already_in_target_state so reload checks reach the auto
promotion path on vision MTP loads. Add three regression tests covering
vision MTP match (auto and default), and non MTP vision repo unaffected.
Verified on a B200 with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
base decode 179.7 t/s vs MTP decode 253.8 t/s, draft acceptance 0.57,
1.41x speedup on a 255 token completion. mmproj still loads and image
input remains available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: prefer Qwen3.5 -MTP GGUF variants in default model lists
With the vision gate dropped in the previous commit, draft-mtp now
auto-engages on -MTP GGUF repos out of the box. Swap the four Qwen3.5
recommended entries in DEFAULT_MODELS_GGUF and DEFAULT_MODELS_STANDARD
to their -MTP-GGUF counterparts so new users get the speedup by default:
unsloth/Qwen3.5-4B-GGUF -> unsloth/Qwen3.5-4B-MTP-GGUF
unsloth/Qwen3.5-9B-GGUF -> unsloth/Qwen3.5-9B-MTP-GGUF
unsloth/Qwen3.5-35B-A3B-GGUF -> unsloth/Qwen3.5-35B-A3B-MTP-GGUF
unsloth/Qwen3.5-0.8B-GGUF -> unsloth/Qwen3.5-0.8B-MTP-GGUF
All four HF repos exist (HEAD 200) and ship the same UD-Q4_K_XL quant
layout as the non-MTP variants. Non-Qwen3.5 entries are untouched.
* bump version to 2026.5.4
Picks up the studio MTP vision-gate fix and the Qwen3.5 -MTP default
swap in this PR.
* studio: prefer Qwen3.6-35B-A3B-MTP-GGUF in default model lists
Same rationale as the previous Qwen3.5 swap. The Qwen3.6 MTP variant
exists at unsloth/Qwen3.6-35B-A3B-MTP-GGUF (HF HEAD 200) and now
auto-engages draft-mtp out of the box with the gate fix.
* studio: drop --spec-draft-n-max from 6 to 3 for draft-mtp
n=6 is too greedy: on Qwen3.6 the draft has to guess 6 tokens ahead
and acceptance crashes to ~0.45, leaving only ~14% throughput gain.
PR ggml-org/llama.cpp#22673's author benched n=3 at ~0.72 acceptance
and 2 to 3x speedup on the same Qwen3.6 family, and the README sample
command uses n=2 or n=3. Match that.
CPU/Mac branch already uses n=3, so this aligns both paths.
* studio: set --spec-draft-n-max back to 6 for draft-mtp on GPU
Reverts the n=3 tuning. n=6 is the original default; user-side comparisons
hold the larger draft window steady so the toggle (next commit) is the
primary on/off lever.
* studio: add Speculative Decoding toggle under Max Tokens
Adds a top-level kill switch (panel-switch under Max Tokens, mirroring
Auto-Healing Tool Calls) that forces the /load request's
speculative_type to "off" when disabled. The backend "off" branch in
LlamaCppBackend.load_model skips both the draft-mtp auto-promotion and
the spec-emit branch, so neither --spec-type draft-mtp nor
--spec-default reaches llama-server.
Wiring:
- chat-runtime-store: new speculativeDecodingEnabled bool, default
true, persisted to localStorage under unsloth_speculative_decoding,
plus a setSpeculativeDecodingEnabled setter.
- chat-settings-sheet: SpeculativeDecodingToggle rendered immediately
beneath the Max Tokens slider for non-external models.
- use-chat-model-runtime: when speculativeDecodingEnabled is false,
override speculative_type to "off" in the loadModel call so the
switch wins over any pre-existing speculativeType state (including
the existing per-model toggle in Model Settings).
Verified end to end on unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
toggle ON emits --spec-type draft-mtp --spec-draft-n-max 6; toggle
OFF emits zero --spec-* flags on the same MTP GGUF.
* studio: relocate Speculative Decoding toggle into Model Settings
Move the toggle out from under Max Tokens and back into the Model
Settings section, directly beneath KV Cache Dtype, where the existing
Apply/Reset workflow already drives a reload on dirty. This way flipping
the switch in the UI actually picks up: the section becomes dirty,
Apply re-runs /load with the new speculative_type.
Drop the !currentModelIsMultimodal gate so vision MTP GGUFs can also
disable speculative decoding from the UI.
Switch the toggle's off-value from null to "off" so the backend's "off"
short-circuit fires for MTP models too (null normalises to None which
re-triggers the draft-mtp auto-promotion).
Tooltip now reads "Faster generation with 0% accuracy hit".
Remove the now-redundant speculativeDecodingEnabled bool + setter from
the runtime store and the load-time override in use-chat-model-runtime;
the toggle binds directly to speculativeType.
* studio: restore OOM/TIGHT badge on recommended GGUF rows
The recommended-list row passed vramStatus=null for any GGUF repo
because the existing useRecommendedModelVram hook reads safetensors
totals from HF model info, which GGUF-only repos do not expose. As a
result, an OOM Q-quant repo would render with only a "GGUF" badge and
no visual signal that nothing in it fits.
Add useGgufRecommendedFit: per repo, fetch the variant list via the
existing /api/models/gguf-variants endpoint, take the smallest
variant's size_bytes, and classify with the same 0.7*GPU + 0.7*RAM
thresholds as GgufVariantExpander. Session-scoped cache + in-flight
dedup so a repo is requested at most once.
Wire the result into the three GGUF row sites in pickers.tsx so OOM
and TIGHT badges show on the collapsed cards.
* Revert "studio: restore OOM/TIGHT badge on recommended GGUF rows"
This reverts commit 07793b1240df72b13e51d6dc15f63c4ee8c6cba9.
The new useGgufRecommendedFit hook was treating the symptom. PR #5561
identified the real root cause: useGpuInfo was calling /api/system
with plain fetch instead of authFetch, so the session-auth check
failed silently and gpu.available stayed false everywhere. With no
GPU info, every fit check (variant expander, recommended carousel)
fell back to "no signal" and dropped the OOM/TIGHT badges.
Reverting the over-engineered hook and applying the authFetch fix
in the next commit, which restores the existing badges with one line.
* chore: replace qwen suggested with MTP variant
* fix: restore GPU info auth for GGUF fit badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
unsloth 2026.5.3 was just published to PyPI. Update install.sh and
install.ps1 so fresh installs pull the new release (5 occurrences each).
Co-authored-by: Daniel Han <info@unsloth.ai>
* studio/chat: release stuck IME flag when compositionend never fires
Chrome on Windows talking to a WSL-hosted Studio (issue #5546) fires
compositionstart + compositionupdate but no compositionend after the
IME commits. The earlier hardening in #5327 cleared the stale flag on
the next non-composing input event, which never arrives in this
sequence, so composingRef stays true forever and the Send button stays
disabled even though the committed CJK text is already in the textarea.
Add a watchdog in both useImeComposerInputHandlers (main + edit
composer) and SharedComposer (compare mode) that runs the same reset
the missing compositionend would have done. The timer is rearmed on
every compositionupdate and on every non-composing input so it only
fires when the IME pipeline has actually gone quiet — normal candidate
selection keeps it alive, the WSL stuck case lets it expire.
Extends the existing IME Playwright smoke with a stuck-compositionend
repro and adds a static guard so the watchdog can't be removed without
the regression tests catching it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-pin composing flag on IME keydown to close#5546 watchdog gap
The stuck-compositionend watchdog (PR #5551) releases composingRef after
2500 ms of IME silence so Send unwedges in the WSL+Chrome case. The same
release also fires during a long candidate-window pause in healthy IMEs,
which lets a subsequent IME-confirm Enter slip preedit text through
handleSubmit (main composer) or click-Send through send() (compare composer).
Add a keydown gate to both composers: when the browser still reports
nativeEvent.isComposing or keyCode 229, re-pin composingRef and cancel
any pending watchdog so the next form-submit / send() guard refuses.
The Send button stays visually enabled (avoids re-introducing the
stuck-UI bug) but the submit path is blocked until a real compositionend
or non-composing input arrives. Mirrors the existing isComposing guard
shape in shared-composer.onKeyDown.
Tests:
- tests/studio/test_composer_rtl_bidi_attribute.py: two new static
guards asserting the keydown gate wiring in both composer files.
- tests/studio/playwright_chat_ime_i18n.py: new section 6c repro that
fires the IME-confirm keydown after the watchdog has cleared, then
triggers form.requestSubmit() and asserts the preedit text is not
cleared (would indicate a leaked submit).
Verified across Chromium / Firefox / WebKit via a side-by-side pre-PR
vs post-PR simulation (54 scenarios, zero pageerror or console.error).
The #5546 stuck-end repro still passes (Send re-enables 2.5-3 s after
the silent commit) and the new keydown-repin probe confirms the submit
gate refuses on all three engines.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-arm IME watchdog after keydown re-pin (Codex P1)
The keydown re-pin added in 2c3c9793 closed the watchdog-race for
healthy IMEs, but on the same WSL+Chrome no-compositionend path this
PR targets it would re-lock Send permanently: setting composingRef=true
and only *clearing* the watchdog leaves the flag pinned forever if no
follow-up compositionend or non-composing input ever arrives.
Swap clearStuckTimer/clearStuckImeTimer for refreshStuckTimer/
refreshStuckImeTimer in both composer keydown gates so the watchdog
fires once more after every IME keypress. Same visual contract — Send
stays enabled — the submit gate just keeps a 2.5s window before
re-releasing instead of staying locked.
Extends the playwright IME smoke with section 6d: clears composing via
the watchdog, fires an IME keydown, then waits past the re-armed
watchdog window and asserts the form submit actually flushes the
textarea. Two new static guards in test_composer_rtl_bidi_attribute
lock the refresh call into both keydown handlers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Two near-identical Discord button images existed under images/, with
the only effective difference being the rendered button width. Keep
the narrower variant (formerly the lowercase "discord button.png")
and remove the wider "Discord button.png", consolidating to a single
"Discord button.png" file.
* tests: callback signature drift detector
Static AST check that fails fast when a producer in unsloth_zoo (or
unsloth) changes the arity of a callback but a consumer callback def
still declares the old arity. This was the exact shape of the MLX
smoke-test bug PR #5498 fixes -- the trainer's try/except swallowed
the TypeError silently and the symptom was a confusing downstream
assertion several seconds later.
What the detector does:
* Producer side: walks every .py and finds classes that own a
self._<name>_callbacks list, populated via .append() from an
add_<name>_callback method, and invoked via
`for cb in self._<name>_callbacks: cb(arg1, ..., argN)`. The
arity at the call site is the canonical expected arity.
* Consumer side: walks every <obj>.add_<name>_callback(fn) call,
resolves fn to a def or lambda in the same file, and asserts
arity matches. Consumers that use *args or **kwargs are
tolerantly accepted as any arity.
* Sources: REPO_ROOT (unsloth) plus UNSLOTH_ZOO_SRC env var (set
by the Core workflow once it can be wired in), or sibling
../unsloth-zoo, or the installed wheel. Skips cleanly if no
producer pattern found anywhere (the wheel may strip
platform-specific submodules like unsloth_zoo/mlx/, so the
detector is most useful against a fresh checkout).
Validated end-to-end:
* Reverted run_real_mlx_smoke.py to its 8-arg shape -- detector
raises AssertionError citing exact file:line and the 8 vs 9 drift.
* Restored the 9-arg shape -- detector PASSes.
* Total runtime ~7 s in pytest.
Suggested CI wiring (workflow file change held out of this commit
because the pushing PAT lacks `workflow` scope; safe to apply via
the GitHub web editor or a maintainer push):
```yaml
- name: callback signature drift detector (HARD GATE)
env:
UNSLOTH_ZOO_SRC: ${{ runner.temp }}/unsloth-zoo
run: |
python -m pytest -v --tb=short tests/test_callback_signature_drift.py
```
Drop the step into .github/workflows/consolidated-tests-ci.yml right
after the existing public-api drift detector step. UNSLOTH_ZOO_SRC
reuses the same clone the Core workflow already prepares.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: wire callback-signature drift detector into Core matrix
Drops a 6-line pytest step right after the public-api drift detector,
with UNSLOTH_ZOO_SRC pointed at the freshly cloned $RUNNER_TEMP/unsloth-zoo
so the detector sees unsloth_zoo/mlx/ (the wheel strips it).
Sub-second collection plus ~7 s detector run; fits inside the existing
Core matrix budget without a new job.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* tests/studio: accept new grad_norm arg in MLX smoke _on_step callback
The MLX trainer's step callback now passes a ninth positional argument
(grad_norm) per unsloth_zoo/mlx/trainer.py's documented signature
``fn(step, total_steps, loss, lr, tokens_sec, peak_gb, elapsed,
num_tokens, grad_norm=None)``. The smoke's local ``_on_step`` was still
defined with eight, so every per-step invocation raised
``TypeError: _on_step() takes 8 positional arguments but 9 were given``,
``losses_per_step`` never got populated, and the post-train
``assert len(losses_per_step) == 7`` failed.
Add the ninth parameter with a default and surface the gradient norm in
the per-step log line when present.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: pin max_grad_value=0 in MLX smoke so max_grad_norm=1.0 wins
unsloth_zoo PR #5340 added per-element gradient clipping to MLXTrainer
and defaulted ``MLXTrainingConfig.max_grad_value = 5.0``. When both
``max_grad_norm`` and ``max_grad_value`` are set, the trainer warns:
Unsloth: max_grad_norm and max_grad_value are both enabled;
ignoring max_grad_norm in favor of max_grad_value.
and silently drops the test's ``max_grad_norm=1.0``. +-5.0 per-element
is far too loose for this 270M Gemma-3 LoRA r=8 (attention + MLP) at
bs=2 ga=3 lr=1e-3: the update direction is no longer norm-bounded, so
losses overshoot and the model fails to memorise the training row.
Reproduced on a CUDA mirror (scripts/cuda_mlx_mirror_sim.py):
norm_1 (max_grad_norm=1.0, no clip): losses 7.64 -> 0.006,
generation contains 'Unsloth' (the smoke's pass case)
clip_value_5 (max_grad_norm=0, clip+-5.0): losses 7.29 -> 8.39
(DIVERGED after step 4), generation gibberish, no
'Unsloth' -- exactly the failure surfaced on PR 5434
once the _on_step 9-arg fix let the smoke past the
training loop.
Pin ``max_grad_value=0.0`` so the smoke uses the same ``max_grad_norm=
1.0`` clipping it was designed against. Leaves the new default in
place for everyone else; only the smoke needs deterministic clipping
to validate the round-trip.
* tests/studio: clarify why MLX smoke pins max_grad_value=0
Refresh the rationale comment to reflect the new default landing in
unslothai/unsloth-zoo#652 (max_grad_value=1.0, not 5.0). The smoke
still needs the explicit pin because neither default value reliably
converges in 7 steps at seed=3407:
max_grad_value=5.0 -- diverges after step 4 (loss 7.3 -> 8.4)
max_grad_value=1.0 -- stalls (loss ~3.2 plateau across seeds)
max_grad_value=0.5/0.25/0.1 -- noisier still
max_grad_norm=1.0 -- cleanly drops loss to <0.01, emits "Unsloth!"
Mention both the historical 5.0 default and the new 1.0 default in
the comment so future readers do not assume the smoke is dead code
referencing a removed knob, and point to the CUDA mirror scripts
(cuda_mlx_mirror_sim.py + cuda_mlx_clip1_vs_norm1.py) for the
empirical evidence.
No behaviour change; comment-only refresh.
* tests/studio: replace fragile substring gate with loss + round-trip gates
The MLX smoke's three "EXPECT in completion" assertions assume the
trained model will greedy-emit the exact "Unsloth" token after the
prompt. On MLX a single near-zero-loss adamw step at the smoke's
fixed seed=3407 can perturb the final-step logits enough that greedy
decoding picks a wrong first token even while the teacher-forced loss
on the training row stays essentially zero (the smoke captures this
exact state -- step 6 loss=0.049, step 7 grad=36.7, step 7 loss=0.17;
completion goes from "Unsloth!" to "5 lbs!"). Reproduced extensively
on CUDA via scripts/cuda_mlx_step7_*.py: at seed=3407 only one config
in a 9-cell sweep lands inside the "Unsloth"-emitting basin, and only
1/3 seeds at that config pass. This is a property of the assertion,
not of save/reload correctness.
Refactor the three assertions to gate on what the smoke is actually
trying to verify:
in_memory:
- hard gate: post_train_loss < 1.0 (training memorised the row).
- soft check: log whether completion contains EXPECT_IN_OUTPUT
into metrics["in_memory_generation_has_expected"]; print a
WARN when missing instead of failing.
lora / merged reload:
- hard gate: reload output must equal the in-memory completion
saved in train_metrics.json. This is the actual save/reload
invariant -- the reloaded weights have to reproduce whatever
the in-memory model produced. Falls back to the original
gibberish gate if train_metrics.json is unavailable.
gguf reload:
- hard gate: llama.cpp produced usable, non-empty output after
the prompt (>=4 chars). llama.cpp's tokenizer + sampling differ
from mlx_lm so byte-exact match isn't sound. Log
gguf_has_expected for visibility.
Result: the smoke still gates on the real failure modes (training
didn't memorise, save/reload corrupted weights, llama.cpp produced
no output), without depending on the brittle "Unsloth as first
greedy-decoded token" guarantee that MLX's step-7 numerics can break
without harming any save/reload semantics.
Cross-version constraint: no transformers / trl API touched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: gate MLX reload on training-row loss, not greedy text
The strict reload assertion (out == in_mem_out) failed on macOS:
in-memory completion was '5 lbs!' and the reloaded completion was
'_________________________'. Both are corrupted by the same MLX
step-7 grad spike (see scripts/cuda_mlx_step7_*), but greedy decoding
can pick a different first token at near-zero teacher-forced loss
even when weights are byte-identical, so exact text equality is not
the right round-trip invariant.
Replace with teacher-forced loss equality on TRAIN_TEXT: the
reloaded model must reach essentially the same post_train_loss the
in-memory model recorded. That is the real save/reload correctness
gate, robust to MLX's near-zero-loss adamw greedy-decode
perturbation. Falls back to a non-empty-body check when
train_metrics.json is missing.
CUDA mirror at this seed converges cleanly to ~0.006 loss; on MLX
post_train_loss < 1.0 still holds via the existing memorisation
gate. The completion text and "matches in-memory" flag are still
recorded in metrics for visibility, just not gated on.
* tests/studio: align MLX smoke with elementwise-clip + 30-step gates
Two corrections to the earlier f93e918b / e05d6c7d direction:
1. max_grad_value=0.0, max_grad_norm=1.0 picked the memory-heavy
norm clip. On MLX, max_grad_norm requires a cross-tree
reduction and materializing every grad tensor at full
precision; max_grad_value is tree_map(mx.clip) per leaf with
no reduction. MLXTrainingConfig defaults to max_grad_value=1.0
for exactly this reason. Flip the smoke to
max_grad_norm=0.0, max_grad_value=1.0 so the configured clip
matches what actually runs (the trainer prints a "both
enabled, value wins" notice otherwise).
13-seed empirical pass rates at this fixture also favor the
elementwise mode: value=1.0 62%, norm=1.0 46%, value=5.0 33%,
value=0.5 77%. Cheaper default = higher pass rate, no
tradeoff. (See PR #5498 / staging-2#119 rounds A-AT.)
2. max_steps=7 was below the convergence horizon at every clip
tested. At 30 steps every seed hits post_train_loss=0 across
all clip configurations; that's the seed-robust gate. Bump
max_steps 7 -> 30, tighten the memorisation gate from
post_loss < 1.0 to post_loss < 0.1.
3. Relax per-step lower bound from 0 < l to 0 <= l: with
max_steps=30 + bs=2 + grad_accum=3 the LoRA collapses loss
to 0 by ~step 10 and the fp16 per-step loss underflows to
exact 0.0 from then on. That's the success signal, not a bug.
Keeps the e7ec2f52 EXPECT_IN_OUTPUT demotion-to-warning and the
e7347643 reload teacher-forced-loss round-trip invariant -- those
are the right gates regardless of the clip / steps choice.
* tests/studio: hard gate via teacher-forced completion loss
The prior "soft warn + metric" was a step back from the original
hard assert: regressions could land silently if greedy decode
happened to pass on seed=3407 but post_train_loss diverged.
A true hard gate is needed.
Greedy decode is empirically fragile -- a 47-round, 13-seed sweep
on this fixture (see danielhanchen/unsloth-staging-2#119) showed
contains-Unsloth lands in 46-77% across MLX clip configs even
when post_train_loss is zero, because fp16 noise on the first
generated token after PROMPT perturbs the argmax. Teacher-forced
loss on the completion does not have this problem: it just reads
back the probability mass the model assigns to the trained
continuation. In every config where post_train_loss < 0.1, the
completion loss is essentially zero.
Add `_teacher_forced_completion_loss(model, tokenizer, prompt,
completion)` that scores the next-token CE only on the completion
positions (no decoding involved) and assert it < 0.5. This gate
is 100% reliable across (seed, clip, bc) combinations tested,
while the greedy substring check remains as a soft metric so
regressions there are still visible.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: hide Current password input on first boot
PR #5490 added a third Current password input to the change-password form
so the admin-forced must_change_password reset path could supply a current
password (the bootstrap is empty in that path). The side effect is that the
dominant first-boot UX, which has window.__UNSLOTH_BOOTSTRAP__ present and
silently fed into currentPassword, now shows three visible inputs instead
of the two it had before.
Render the Current password input only when window.__UNSLOTH_BOOTSTRAP__
is absent. The loadBootstrap effect already seeds the password state from
the bootstrap and currentPassword keeps the bootstrap fallback, so
handleSubmit sees the same value as before. On admin-forced resets where
the bootstrap is undefined, the Current password input still appears so
the user can type their actual current password.
Verified end-to-end against a local install via UNSLOTH_STUDIO_HOME +
install.sh --local with Playwright driving the page: bootstrap present
renders two inputs (New, Confirm) and completes change-password into
/chat; bootstrap suppressed via a non-configurable property descriptor
init script renders the three inputs (Current, New, Confirm) and keeps
the #5490 fix intact.
* studio/frontend: add deterministic input-count tests for auth-form
Pure-source pytest covering the change-password JSX contract. No
browser, no Studio boot, no JS toolchain -- runs on any CI runner.
Complements the Playwright probe in tests/studio/playwright_chat_ui.py
which exercises the same contract end to end.
Pins seven invariants with explicit failure reasons:
1. hasBootstrapPassword is derived from window.__UNSLOTH_BOOTSTRAP__
so a future swap to a localStorage flag or prop cannot silently
drift from the backend's _inject_bootstrap contract in
studio/backend/main.py.
2. Exactly one !hasBootstrapPassword conditional exists; multiple
would split rendering into branches these tests cannot reason
about.
3. The Current password input sits inside that conditional, so it
never renders on first boot (the regression PR #5490 introduced
and that this fix reverses).
4. The New password input sits outside it, so it always renders in
change-password mode (admin-forced reset still works).
5. Confirm password: same as New.
6. The change-password JSX subtree declares exactly current /
new / confirm; a fourth password input would almost certainly
break the 2-input first-boot contract.
7. The login JSX subtree declares exactly one password input.
Verified the tests fail loudly on the pre-fix auth-form.tsx at
c4575ca0 (5/7 fail with descriptive reasons) and pass on the fixed
version (7/7).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fast_generate: unify legacy/new logits kwarg + fix Mistral merge site
Two related issues caught by review on PR #5538:
1. unsloth_fast_generate (models/llama.py)
The previous patch promoted num_logits_to_keep -> logits_to_keep
unconditionally whenever the caller supplied num_logits_to_keep,
and only popped num_logits_to_keep (not logits_to_keep). On
transformers older than 4.50 (legacy spelling is the only one the
model forward accepts), the promotion broke things; symmetrically,
a caller supplying logits_to_keep on those older transformers also
went unchecked.
Switch to the unified normalize-then-inspect pattern from the
review:
_provided_num = kwargs.pop("num_logits_to_keep", None)
_provided_logits = kwargs.pop("logits_to_keep", None)
_provided = _provided_logits if _provided_logits is not None else _provided_num
_fwd_params = inspect.signature(self.forward).parameters
if "logits_to_keep" in _fwd_params:
kwargs["logits_to_keep"] = _provided if _provided is not None else 1
elif "num_logits_to_keep" in _fwd_params:
kwargs["num_logits_to_keep"] = _provided if _provided is not None else 1
Inspect the runtime forward signature first, then choose the
spelling it actually accepts, then route either user-supplied value
under that spelling. Backward-compatible in both directions.
2. MistralForCausalLM_fast_forward (models/mistral.py)
The max(num_logits_to_keep, logits_to_keep) merge was inside the
`if UNSLOTH_RETURN_HIDDEN_STATES:` block, so it only fired on the
GRPO hidden-states path. On the normal generation path the elif at
line 316 only checked num_logits_to_keep, so a caller (including
unsloth_fast_generate itself) passing logits_to_keep=1 ended up
computing full prompt logits instead of slicing to the last token.
For long prompts that reintroduces the large prefill logits
allocation the default keep=1 was avoiding.
Move the max() merge above the env-var branching so the normal
generation path slices correctly too. Llama already did this
merge at the top (unsloth/models/llama.py:1501); Mistral now
matches.
No behaviour change on the default GRPO / SFT paths. Targets only the
edge cases the review flagged.
* fast_generate: preserve caller logits kwarg when signature inspect fails
If `inspect.signature(self.forward)` raises TypeError/ValueError (opaque
C-extension or compiled wrappers), the previous fix set `_fwd_params = {}`
which silently dropped the caller-supplied `logits_to_keep` /
`num_logits_to_keep`. Fall back to the spelling the caller used (default
`logits_to_keep=1` when neither was supplied) so generation still honors
the requested logits slice.
* fast_forward: do not max() int against tensor logits_to_keep
HF accepts logits_to_keep as a 1-D LongTensor of positions for
selective decode. The merge in mistral.py (added by this PR) and
the pre-existing one in llama.py both run max(int, Tensor), which
casts the comparison to a bool and raises on multi-element tensors.
Branch on type and skip the merge when either argument is a tensor;
downstream int-slice path is unchanged, so tensor callers fall
through with num_logits_to_keep == 0, matching pre-merge behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate/forward: shorten kwarg-merge comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: soften toast shadow and tighten vertical padding
Sonner's defaults felt heavy in the chat header surface: a 16px
all-around padding made the box taller than the two-line content
warranted, and the 4/12/0.10 drop shadow read as a hard slab
against the light background. Trim padding to 10px vertical
(horizontal unchanged at 16px) and dial the shadow back to
0 2px 6px / 0.08 so the toast still lifts off the surface without
casting a heavy halo.
* studio/frontend: annotate why toast override needs !important
Sonner injects its base styles at runtime from inside its JS bundle,
so a plain cascade tie can lose depending on injection order. One
short comment above the override saves the next reader the dig.
* studio/frontend: boost toast shadow opacity in dark mode
Sonner's lighter 0.08 shadow disappears on the dark popover surface:
quantitative measurement of the shadow band (10px below the toast)
across Chromium / Firefox / WebKit showed only a ~3% luminance drop
vs background, well below perceptual threshold. Bump the dark-mode
opacity to 0.3, matching the existing .shadow-border light/dark ratio
(0.1 -> 0.3) and bringing the toast in line with .menu-soft-surface's
dark-mode shadow (0.28). Light mode keeps the original 0.08.
* studio: install flash-linear-attention and tilelang for Qwen3.5 family
Studio currently only installs causal-conv1d for qwen3.5 / qwen3.6 /
qwen3-next models. Without flash-linear-attention installed alongside
it, transformers' Qwen3.5 fast-path gate stays False and the model
falls back to a pure-PyTorch loop for the GatedDeltaNet layers. In a
60-step run on unsloth/Qwen3.5-2B on B200, this fallback costs ~2.35x
vs the full fast path.
On top of that, FLA dispatches its hottest GDN kernels through a
TileLang backend when tilelang is importable. Adding tilelang plus a
pinned apache-tvm-ffi gives another ~26% on the same workload (4.73
s/step to 3.50 s/step) and is what users have been getting indirectly
when they install mamba-ssm (mamba-ssm transitively pulls tilelang and
pins apache-tvm-ffi<=0.1.9, which is the last working version on
sm_100; 0.1.10 and 0.1.11 crash Triton with misaligned address).
Changes:
* _ensure_flash_linear_attention: pure-Python PyPI install gated on
the same model match set as _ensure_causal_conv1d_fast_path.
* _ensure_tilelang_backend: installs apache-tvm-ffi==0.1.9 and
tilelang==0.1.8 in one pip resolve so the tvm-ffi pin wins over
tilelang's >=0.1.2 constraint. Gated on the Qwen3.5 family only;
SSM models (Nemotron-H, Falcon-H1, Granite-H, LFM2) do not use
FLA's GDN dispatch.
* UNSLOTH_STUDIO_SKIP_TILELANG_INSTALL=1 escape hatch matching the
flash-attn pattern.
* Orchestration block reordered: causal-conv1d -> fla -> mamba-ssm
-> tilelang -> flash-attn (long context).
* 7 new tests covering the new helpers, including SSM-model skip,
skip-env, full Qwen3 family name variants, and graceful pip
install failure.
Combined Qwen3.5-2B-Vision step time on B200 in our bench goes from
5.0 s/step (current Studio: causal-conv1d only) to 3.5 s/step
(causal-conv1d + fla + tilelang), a 1.43x speedup with no notebook
or user code changes required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: accept new grad_norm arg in MLX smoke _on_step callback
The MLX trainer's step callback now passes a ninth positional argument
(grad_norm) per unsloth_zoo/mlx/trainer.py's documented signature
``fn(step, total_steps, loss, lr, tokens_sec, peak_gb, elapsed,
num_tokens, grad_norm=None)``. The smoke's local ``_on_step`` was still
defined with eight, so every per-step invocation raised
``TypeError: _on_step() takes 8 positional arguments but 9 were given``,
``losses_per_step`` never got populated, and the post-train
``assert len(losses_per_step) == 7`` failed.
Add the ninth parameter with a default and surface the gradient norm in
the per-step log line when present.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: retrigger after zoo drift + IPython fixes landed in main
* tests/studio: pin max_grad_value=0 in MLX smoke so max_grad_norm=1.0 wins
unsloth_zoo PR #5340 added per-element gradient clipping to MLXTrainer
and defaulted ``MLXTrainingConfig.max_grad_value = 5.0``. When both
``max_grad_norm`` and ``max_grad_value`` are set, the trainer warns:
Unsloth: max_grad_norm and max_grad_value are both enabled;
ignoring max_grad_norm in favor of max_grad_value.
and silently drops the test's ``max_grad_norm=1.0``. +-5.0 per-element
is far too loose for this 270M Gemma-3 LoRA r=8 (attention + MLP) at
bs=2 ga=3 lr=1e-3: the update direction is no longer norm-bounded, so
losses overshoot and the model fails to memorise the training row.
Reproduced on a CUDA mirror (scripts/cuda_mlx_mirror_sim.py):
norm_1 (max_grad_norm=1.0, no clip): losses 7.64 -> 0.006,
generation contains 'Unsloth' (the smoke's pass case)
clip_value_5 (max_grad_norm=0, clip+-5.0): losses 7.29 -> 8.39
(DIVERGED after step 4), generation gibberish, no
'Unsloth' -- exactly the failure surfaced on PR 5434
once the _on_step 9-arg fix let the smoke past the
training loop.
Pin ``max_grad_value=0.0`` so the smoke uses the same ``max_grad_norm=
1.0`` clipping it was designed against. Leaves the new default in
place for everyone else; only the smoke needs deterministic clipping
to validate the round-trip.
* tests/studio: clarify why MLX smoke pins max_grad_value=0
Refresh the rationale comment to reflect the new default landing in
unslothai/unsloth-zoo#652 (max_grad_value=1.0, not 5.0). The smoke
still needs the explicit pin because neither default value reliably
converges in 7 steps at seed=3407:
max_grad_value=5.0 -- diverges after step 4 (loss 7.3 -> 8.4)
max_grad_value=1.0 -- stalls (loss ~3.2 plateau across seeds)
max_grad_value=0.5/0.25/0.1 -- noisier still
max_grad_norm=1.0 -- cleanly drops loss to <0.01, emits "Unsloth!"
Mention both the historical 5.0 default and the new 1.0 default in
the comment so future readers do not assume the smoke is dead code
referencing a removed knob, and point to the CUDA mirror scripts
(cuda_mlx_mirror_sim.py + cuda_mlx_clip1_vs_norm1.py) for the
empirical evidence.
No behaviour change; comment-only refresh.
* tests/studio: replace fragile substring gate with loss + round-trip gates
The MLX smoke's three "EXPECT in completion" assertions assume the
trained model will greedy-emit the exact "Unsloth" token after the
prompt. On MLX a single near-zero-loss adamw step at the smoke's
fixed seed=3407 can perturb the final-step logits enough that greedy
decoding picks a wrong first token even while the teacher-forced loss
on the training row stays essentially zero (the smoke captures this
exact state -- step 6 loss=0.049, step 7 grad=36.7, step 7 loss=0.17;
completion goes from "Unsloth!" to "5 lbs!"). Reproduced extensively
on CUDA via scripts/cuda_mlx_step7_*.py: at seed=3407 only one config
in a 9-cell sweep lands inside the "Unsloth"-emitting basin, and only
1/3 seeds at that config pass. This is a property of the assertion,
not of save/reload correctness.
Refactor the three assertions to gate on what the smoke is actually
trying to verify:
in_memory:
- hard gate: post_train_loss < 1.0 (training memorised the row).
- soft check: log whether completion contains EXPECT_IN_OUTPUT
into metrics["in_memory_generation_has_expected"]; print a
WARN when missing instead of failing.
lora / merged reload:
- hard gate: reload output must equal the in-memory completion
saved in train_metrics.json. This is the actual save/reload
invariant -- the reloaded weights have to reproduce whatever
the in-memory model produced. Falls back to the original
gibberish gate if train_metrics.json is unavailable.
gguf reload:
- hard gate: llama.cpp produced usable, non-empty output after
the prompt (>=4 chars). llama.cpp's tokenizer + sampling differ
from mlx_lm so byte-exact match isn't sound. Log
gguf_has_expected for visibility.
Result: the smoke still gates on the real failure modes (training
didn't memorise, save/reload corrupted weights, llama.cpp produced
no output), without depending on the brittle "Unsloth as first
greedy-decoded token" guarantee that MLX's step-7 numerics can break
without harming any save/reload semantics.
Cross-version constraint: no transformers / trl API touched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: gate MLX reload on training-row loss, not greedy text
The strict reload assertion (out == in_mem_out) failed on macOS:
in-memory completion was '5 lbs!' and the reloaded completion was
'_________________________'. Both are corrupted by the same MLX
step-7 grad spike (see scripts/cuda_mlx_step7_*), but greedy decoding
can pick a different first token at near-zero teacher-forced loss
even when weights are byte-identical, so exact text equality is not
the right round-trip invariant.
Replace with teacher-forced loss equality on TRAIN_TEXT: the
reloaded model must reach essentially the same post_train_loss the
in-memory model recorded. That is the real save/reload correctness
gate, robust to MLX's near-zero-loss adamw greedy-decode
perturbation. Falls back to a non-empty-body check when
train_metrics.json is missing.
CUDA mirror at this seed converges cleanly to ~0.006 loss; on MLX
post_train_loss < 1.0 still holds via the existing memorisation
gate. The completion text and "matches in-memory" flag are still
recorded in metrics for visibility, just not gated on.
* ci: retrigger Backend CI after transient pwsh-startup timeout
* ci: retrigger MLX dispatch after pytorch CDN DNS flake
* studio: harden FLA + tilelang installers per reviewer feedback
Addresses bot review on #5434:
* Narrow `_ensure_flash_linear_attention` from `_model_wants_causal_conv1d`
(which also matches Nemotron-H / Falcon-H1 / Granite-H / LFM2) to
`_model_wants_tilelang` (Qwen3.5 / Qwen3.6 / Qwen3-Next only). True
SSM families take the mamba_ssm path and never call FLA's GDN
kernels, so installing FLA there is wasted bandwidth.
* Pin both `flash-linear-attention==0.5.0` and `fla-core==0.5.0` and
install with `--no-deps`. Otherwise pip resolves fla-core's
declared `torch>=2.7.0` requirement and may silently upgrade the
Studio venv's torch on environments running torch 2.4/2.5/2.6.
* Skip both installs on Python <3.10 (FLA, fla-core, and tilelang
all declare `Requires-Python: >=3.10`). On older interpreters the
pip install would fail every launch and leave the worker on the
slow torch fallback while still claiming to have set up the fast
path.
* Skip tilelang install on non-Linux platforms. `tilelang==0.1.8`
only publishes Linux x86_64 / aarch64 and macOS arm64 wheels.
Falling back to its 93MB sdist on a Studio worker is undesirable.
* Detect an existing `apache-tvm-ffi` 0.1.10 / 0.1.11 install and
force a reinstall to 0.1.9 with `--force-reinstall --no-deps`.
Previously the import-only probe returned early and left the
broken version in place, which crashes Triton on sm_100.
* Add a 600s timeout to the tilelang and FLA subprocess.run calls,
matching the existing flash-attn install pattern, so a network
hang cannot block the training subprocess indefinitely.
* 13 new / updated tests covering all six guards plus the
pinned-spec, timeout, and force-reinstall code paths.
Total: 21 passing tests (8 original + 13 new / updated).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: address reviewer.py P1/P2 findings on FLA + tilelang installers
Twelve-reviewer aggregated review on this PR flagged several real
correctness bugs in the first hardening pass. Fixes:
P1:
* Add UNSLOTH_STUDIO_SKIP_FLA_INSTALL escape hatch for symmetry
with UNSLOTH_STUDIO_SKIP_TILELANG_INSTALL and the existing
UNSLOTH_STUDIO_SKIP_FLASHATTN_INSTALL.
* Install einops alongside fla-core. `--no-deps` was suppressing
fla-core's only non-torch runtime dep, so on a clean venv
`import fla.modules` raised ModuleNotFoundError even though pip
exited 0.
* Drop --no-deps from the tilelang force-reinstall path. tilelang
needs z3-solver, ml-dtypes, cloudpickle, etc. at runtime;
--force-reinstall --no-deps left libz3.so missing and
`import tilelang` raised OSError on the next training subprocess.
* Skip FLA install when installed torch is below 2.7.0
(fla-core declares torch>=2.7.0). Otherwise users on Studio's
supported torch 2.4/2.5/2.6 stacks get an incompatible FLA
installed silently.
P2:
* Replace bare `except ImportError` probes with helpers that catch
`Exception` so a broken native package (OSError on missing
.so, RuntimeError in __init__, ...) does not kill the worker
before the fallback path can run.
* Tighten the tilelang platform guard from "any linux" to
"linux + machine in {x86_64, aarch64, ...}" so ppc64le / s390x /
armv7 do not fall through and download the 93 MB tilelang sdist.
* Add --only-binary=:all: to the tilelang install command. The
comment already said we never want the sdist; now the pip
invocation enforces it.
* Verify both FLA and tilelang are importable after pip exits 0;
if not, report and continue on the fallback path.
6 new tests bring the suite to 27 passing (was 21).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: pin packaging + triton with FLA --no-deps install
An end-to-end install simulation in a fresh venv caught a real
regression: `fla/utils.py` does `from packaging import version` and
`import triton` at module load, but fla-core's METADATA only declares
einops + torch. With `--no-deps` the worker would land FLA in any
runtime that lacks packaging (e.g. minimal torch builds) and the
post-install import probe would fall back to the torch GDN loop
silently.
Add `packaging` and `triton` to `_FLA_RUNTIME_DEPS` so the install
spec list always carries them. Tests updated to assert both are now in
the install command.
* studio: hook transformers' fast-path gates for just-in-time FLA + causal-conv1d install
The substring-based detection in this PR (`_model_wants_tilelang` /
`_model_wants_causal_conv1d`) is brittle: it depends on what the user
typed for the model name, not on what the architecture actually needs.
Users typing custom model paths, future Qwen3.7 / non-Qwen GDN
architectures, and any model whose author renamed it would silently
fall back to the torch loop.
The correct signal is the one transformers itself uses to gate the
fast path. `transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py`
does at module import time:
if is_causal_conv1d_available():
from causal_conv1d import causal_conv1d_fn, causal_conv1d_update
if is_flash_linear_attention_available():
from fla.modules import FusedRMSNormGated
from fla.ops.gated_delta_rule import (
chunk_gated_delta_rule, fused_recurrent_gated_delta_rule,
)
Wrap both gates so the first call (always at modeling import, before
any forward pass) installs the matching kernel synchronously and
delegates to the original function. Any model whose architecture
queries those gates auto-triggers the install; models that never
query them (Llama, Gemma, dense Qwen, ...) never pay the cost.
Mechanics:
- Split `_ensure_flash_linear_attention` and `_ensure_tilelang_backend`
into `_unconditional` variants (no substring gate, retains python
/ torch / platform / skip-env guards) plus thin substring wrappers
used by the legacy fallback path.
- New `_install_fast_path_hooks(event_queue)` patches both gates on
`transformers.utils.import_utils` AND sweeps `sys.modules` so any
modeling file that already did `from ... import is_X` sees the
wrapper (the local binding survives a module-level reassignment).
- Wrappers clear the original's `lru_cache` before delegating, install
on False, re-check, and short-circuit on subsequent calls.
- Set `UNSLOTH_STUDIO_SKIP_FAST_PATH_HOOKS=1` to fall back to the
substring path.
Verified end-to-end against `transformers.models.qwen3_5_moe`:
PRE_STATE fla=False tilelang=False causal_conv1d=False
HOOK_INSTALLED
Hook fired for is_causal_conv1d_available; installing kernel...
Installing prebuilt causal-conv1d wheel...
Hook fired for is_flash_linear_attention_available; installing kernel...
Installing flash-linear-attention==0.5.0 (with fla-core==0.5.0) for the fast path...
Installed flash-linear-attention for the FLA fast path
Installing TileLang backend (apache-tvm-ffi==0.1.9, tilelang==0.1.8)...
Installed TileLang backend for FLA fast path
MODELING_IMPORT_OK
FAST_PATH_SYMBOLS {"chunk_gated_delta_rule": true,
"fused_recurrent_gated_delta_rule": true,
"FusedRMSNormGated": true,
"causal_conv1d_fn": true,
"causal_conv1d_update": true}
POST_STATE fla=True tilelang=True causal_conv1d=True
Adds 9 new tests covering: install-on-False, skip-on-True, idempotency,
install-failure handling, env-disable, lru_cache clear, sys.modules
rebind, missing-transformers fallback, substring fallback. Total
test count is now 36 (was 27).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: address reviewer.py n=12 findings on the FLA hook path
Eight issues reproduced by parallel reviewers against 6ce495a; all
fixed and covered by regression tests. 45 pytest cases pass (was 36);
end-to-end Qwen3.5_MoE modeling-import drill still loads all five
fast-path symbols.
P1 fixes:
1. TileLang loses the Qwen-family guard on the normal FLA hook path
(10/12 reviewers, reproduced with allenai/OLMo-Hybrid-1B). The
hook unconditionally installed tilelang for any FLA-using model.
- Threaded `model_name` through `_install_fast_path_hooks(event_queue,
model_name)`.
- `_fla_install` now gates tilelang on
`_model_wants_tilelang(model_name)` AND a successful FLA install.
2. TileLang repair `--force-reinstall` (without `--no-deps`) could
replace `torch==2.12.0+cu130` with `torch==2.12.0`. Split repair
into TWO steps:
step 1: `--force-reinstall --no-deps apache-tvm-ffi==0.1.9`
step 2: regular install of tilelang + apache-tvm-ffi
Step 1 surgically downgrades the broken package; step 2 resolves
missing transitive deps (z3-solver, ml-dtypes) without
--force-reinstall, so it never replaces torch.
3. Hook could return True after the installer's deep import probe
failed: when pip exits 0 but `import fla.modules` raises, the old
wrapper re-called `original()` (transformers' metadata check) and
trusted it. Refactored:
- `_ensure_flash_linear_attention_unconditional(...) -> bool`
- `_ensure_tilelang_backend_unconditional(...) -> bool`
The wrapper now uses the installer's bool directly.
4. SSM models (Nemotron-H, Falcon-H1, Granite-H) use
`lazy_load_kernel("causal-conv1d")` and never call
`is_causal_conv1d_available()`, so the hook never fires for them.
The orchestrator now always runs `_ensure_causal_conv1d_fast_path`
outside the hook-mode if/else.
P2 fixes:
5. `_rebind_in_already_imported_modules` invoked transformers' lazy
module `__getattr__` (hundreds of "Accessing X from .models..."
warnings, ~3.4s overhead). Switched to `module.__dict__.get(...)`
which only sees real module-level bindings.
6. TileLang installed even when FLA was skipped (Torch <2.7) or
failed (timeout, post-install probe failed). Now gated on the
installer's bool return.
7. TileLang repair was skipped when FLA was already True but tilelang
missing or apache-tvm-ffi on the broken list. Added an optional
`post_available_fn` to the wrapper; the FLA hook's
`_fla_post_available` runs `_ensure_tilelang_backend_unconditional`
when (model wants tilelang) AND (tilelang missing OR tvm-ffi broken).
8. `_flash_linear_attention_importable()` only checks deep import,
not version. Added `_flash_linear_attention_current()` that
compares against the pinned `flash-linear-attention==0.5.0` /
`fla-core==0.5.0`; older versions trigger `--force-reinstall
--no-deps` so torch stays untouched.
Helpers extracted to keep the surface tight:
- `_pip_install_cmd(*args)` builds `uv pip install` or
`python -m pip install` depending on uv availability.
- `_run_pip(cmd, event_queue, label)` runs a pip command with
timeout / failure handling and a status emission.
Regression tests added:
- test_hook_does_not_install_tilelang_for_non_qwen_fla_model
- test_hook_does_install_tilelang_for_qwen35
- test_tilelang_repair_does_not_touch_torch_cuda_stack
- test_hook_trusts_installer_bool_not_metadata
- test_rebind_does_not_trigger_module_getattr
- test_hook_skips_tilelang_when_fla_install_is_skipped
- test_hook_runs_tilelang_repair_when_fla_already_true
- test_fla_installer_force_reinstalls_when_older_version_present
- test_run_training_process_eagerly_installs_causal_conv1d_in_normal_mode
Existing tests updated for the new `_install_fast_path_hooks` signature
and the two-step tilelang repair flow.
End-to-end re-verified against transformers.models.qwen3_5_moe:
PRE_STATE fla=False, hook fires for both gates, FLA + tilelang +
causal-conv1d install, all 5 fast-path symbols non-None.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix double-install of tilelang on the FLA hook install path
Backend CI surfaced a test-isolation bug introduced by the
post_available_fn mechanism for finding #7. The wrapper ran
`post_available_fn` in BOTH paths (install ran AND gate already True),
but `_fla_install` already chains tilelang on the install path, so the
post-available step then called tilelang install AGAIN.
This was masked locally because tilelang was installed in the
workspace venv (post_available short-circuited on
`_tilelang_importable()` returning True). CI starts with no tilelang,
so the second call actually fired and the mock recorded two calls.
Fix: only run `post_available_fn` when the install path did NOT run.
That preserves the finding #7 semantics (tilelang repair when FLA
already True but tilelang missing or tvm-ffi broken) without
duplicating the chained install on the gate-was-False path.
Also tightened `test_hook_skips_install_when_gate_already_true` to
monkeypatch `_tilelang_importable=True` and
`_installed_tvm_ffi_version=0.1.9` so it stays a pure "no install at
all" test regardless of the venv's actual state.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: retrigger Mac Studio GGUF after transient HF DNS resolve flake
* studio: skip tilelang on HIP / ROCm torch (Strix Halo crash report)
h34v3nzc0dex tested PR 5434 on Strix Halo (gfx1151, ROCm 7.13,
torch 2.11.0+rocm7.13.0) and hit a hard regression:
File ".../fla/ops/common/backends/tilelang/__init__.py", line 92,
in chunk_bwd_dqkwg
File ".../tilelang/jit/kernel.py", line 137, in __init__
File ".../tilelang/tileop/gemm/__init__.py", line 143,
in _select_gemm_instruction
tvm.error.InternalError: Check failed: (0) is false:
Unsupported target for gemm:
hip -keys=hip,gpu -mcpu=gfx1151 ...
`tilelang==0.1.8` ships no HIP GEMM instruction; `_select_gemm_instruction`
raises at lower-time, not import-time. So:
- pip install succeeds
- `import tilelang` succeeds
- `TileLangBackend.is_available()` returns True
- FLA's dispatcher picks TileLang for `chunk_bwd_dqkwg`
- training subprocess dies at first GDN backward, no graceful fallback
The PR's existing platform gate (`_tilelang_platform_supported`)
checked only `sys.platform == "linux"` and `platform.machine()`, both
of which look identical on a ROCm box.
Fix has two layers:
1. INSTALL GATE: new `_torch_has_hip()` helper checks
`torch.version.hip is not None`. `_tilelang_platform_supported`
now returns False on HIP torch, so the install never fires.
2. RUNTIME GATE: even with the install skipped, a user could have
tilelang already present (e.g. venv carried over from a CUDA box).
`_install_fast_path_hooks` now calls
`os.environ.setdefault("FLA_TILELANG", "0")` when HIP is detected,
which is the env-var FLA's `TileLangBackend` already honors. Users
who know they have a HIP-aware tilelang fork can override by
setting `FLA_TILELANG=1` explicitly.
This costs nothing on CUDA (the gate is a no-op when
`torch.version.hip is None`), and removes the crash for AMD users.
The benchmark numbers in the PR description (1.43x on B200 sm_100)
are not affected.
The other halves of the PR are confirmed working on gfx1151 by the
same report:
- `flash-linear-attention 0.5.0` runs at production scale
(B=1 T=8192 H=16 K=128 V=128 and others) with no patches.
- `causal-conv1d` runs at the shapes the fast-path gate cares
about. (A separate Ubuntu 24.04 `--gcc-install-dir` build
workaround is needed for the source-build path; that mirrors
bbf004c's llama.cpp fix and is out of scope here.)
Tests added:
- test_tilelang_platform_unsupported_on_hip_torch
- test_tilelang_install_skipped_on_hip_torch
- test_install_fast_path_hooks_sets_fla_tilelang_zero_on_hip
- test_install_fast_path_hooks_respects_user_fla_tilelang_override
- test_install_fast_path_hooks_does_not_set_fla_tilelang_on_cuda
Total 50 passing (was 45).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: retrigger Windows Studio UI after transient Playwright tab-lookup flake
* studio: auto-discover FLA-using model types from installed transformers
Drop the hand-maintained `_TILELANG_MODEL_SUBSTRINGS` tuple
(qwen3.5 / qwen3_5 / qwen3.6 / qwen3_6 / qwen3-next / qwen3_next)
and derive the allowlist by scanning the installed
`transformers/models/*/modeling_*.py` for `from fla.` imports.
A model "wants tilelang" iff its modeling file imports an FLA op,
which is the same signal `is_flash_linear_attention_available()` is
the runtime test for. The scan happens once per worker subprocess
and is cached for the process lifetime; an empty result (eg
transformers not importable) means "no tilelang pre-install" --
the FLA runtime hook still drives the install via the gate when
the loaded model actually probes it.
Verified against the live installed transformers, the auto-derived
set is {qwen3_5, qwen3_5_moe, qwen3_next}, with `_model_wants_tilelang`
matching the HF Hub names `unsloth/Qwen3.5-2B`, `Qwen/Qwen3.5-MoE-A3B`,
`mlx-community/qwen3-next-80b`, and correctly rejecting Llama,
Mistral, Nemotron-H, Falcon-H1, etc. Future GDN models (Qwen3.7,
OLMo-Hybrid-FA, ...) are picked up automatically once they ship in
transformers; no further worker edits needed.
Also trim docstrings / comments through the FLA / tilelang / HIP /
hook block: constants get 1-line trailing comments, function
docstrings collapse to 1-3 lines, and the fast-path-hooks banner
shrinks from a 27-line block to 4 lines. The file drops from 2847
to 2630 lines without losing the load-bearing WHY notes
(--no-deps protects torch; `__dict__.get` avoids lazy-module
__getattr__; two-step tvm-ffi repair keeps torch off the dep
graph; HIP setdefault disables FLA's TileLang dispatch even with
tilelang already installed).
7 new tests (50 -> 57 total): discovery returns only FLA-using
model_types; discovery cache reuse; missing transformers handled;
OSError on a modeling file is non-fatal; `_model_wants_tilelang`
matches real HF repo names across separator variants; empty
discovery -> always False; normalization across `-`, `.`, `/`,
space.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: hermetize the non-allowlist hook test against transformers 5.4.0+
transformers 5.4.0 added `olmo_hybrid` as an FLA-using model_type, so
the auto-discovered allowlist now includes it -- and the test's prior
choice of `allenai/OLMo-Hybrid-1B` as a "non-Qwen FLA-only" example
became an allowlist member. CI on Python 3.11 / 3.13 caught this.
Swap to a guaranteed-not-in-allowlist fake model_name AND patch
_discover_fla_model_types to a known {qwen3_5, qwen3_5_moe, qwen3_next}
set so the test stays valid as upstream transformers adds new
FLA-using architectures.
Renames the test to reflect the actual semantic under test:
"outside-allowlist -> no tilelang".
* ci: retrigger Windows Studio API after llama.cpp prebuilt staging WinError 5 flake
* tests: move MLX smoke gate changes to dedicated PR #5537
The seven MLX smoke commits in this PR's history (_on_step grad_norm,
max_grad_value pin, loss + round-trip gates) are unrelated to the
FLA / tilelang work. They now live in #5537 so this PR's diff is
limited to the studio worker installer changes.
Net effect on tests/studio/run_real_mlx_smoke.py vs main: zero.
* studio: friendlier install banners (drop hook / gate-name jargon)
User-visible status text now reads:
Installing flash-linear-attention==<ver> for faster training...
Installing TileLang==<ver> for faster training...
Installing causal-conv1d for faster training...
Installing flash-attn for faster training...
Removed the transient "Hook fired for is_flash_linear_attention_available;
installing kernel..." banner — the install banner that immediately follows
already tells the user what is happening, in plain English.
The internal logger.info messages (server-side log) still carry the
gate names + "Hook fired ..." for debugging.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: add dismissable toasts with corner close button
- Enable Sonner's close button globally on the Toaster, so every toast
(model load progress, model loaded, load failure, etc.) gets an X that
users can click to dismiss without waiting for the auto-dismiss timer.
This matches the Claude desktop notification behavior.
- Drop the per-toast 'closeButton: false' overrides in the model load
runtime so they inherit the global default. The existing 'onDismiss'
handler already flips state to show an inline header status, so the
X on the loading toast hides the toast without canceling the load
(Cancel still aborts).
- Pin the close button to the top-right corner inside the toast box.
Overrides Sonner's left-side default placement, outside-corner
translate, and hardcoded 'top: 0'. Top is set via a small rule in
index.css because Sonner does not expose it as a CSS variable.
- Add a small offset on the Toaster so toasts sit at the chat header
line, shifted left of the parameters and settings buttons on the
right edge instead of stacking on top of them.
- Bump the post-load success and failure durations from 2s and 5s to
8s so users actually have time to read and click the new close X
before the toast auto-dismisses.
* studio: explicit boolean for closeButton prop to satisfy biome
* studio: keep close button X visible in dark mode
Two defensive fixes for the dark-mode close button visibility:
- Use resolvedTheme so sonner's data-sonner-theme always matches the
class next-themes applies to <html>. Passing theme can be 'system',
which makes sonner resolve via its own media query; that can disagree
with next-themes (Tauri webview, hydration races, OS quirks), leaving
CSS vars dark while sonner still applies its light close-button colors
(dark X on dark background).
- Bump the close-icon stroke from sonner's default 1.5 to 2.25 so the X
is readable on a 12x12 svg sitting on dark backgrounds.
---------
Co-authored-by: shimmyshimmer <datta_mike@hotmail.com>
* studio/frontend: make toast and inline error text selectable and copyable
Sonner toasts and the inline model-load error in the chat header were
showing copyable content (backend tracebacks, model-load failures, log
lines) that users could not actually select with the mouse.
Two underlying issues:
1. Sonner's swipe-to-dismiss handler calls `setPointerCapture` in
`onPointerDown`, which preempts the browser's text-selection
gesture. The capture only happens when `dismissible` is true. CSS
alone cannot work around this.
2. The inline model-load error truncated with `text-overflow: ellipsis`
and parked the full string in a native `title=` tooltip, which
browsers render as an OS tooltip that cannot be selected.
Fixes:
- New `@/lib/toast` wrapper that defaults `dismissible: false` on every
toast (callable plus `.success` / `.error` / `.info` / `.warning` /
`.loading` / `.message` / `.custom`). API is identical to sonner's
`toast`, so the 18 call sites just swap their import path. Callers
can opt back into swipe-to-dismiss with `dismissible: true`.
- `<Toaster>` sets `swipeDirections={[]}` to make the intent explicit.
- `index.css` forces `user-select: text` on toast text content and
keeps `user-select: none` on toast buttons.
- New `<CopyableErrorChip>` component replaces the truncated inline
error in the chat header. The chip shows the truncated message
inline and opens a popover with the full, wrap-friendly, selectable
message and a one-click Copy button.
Toasts still auto-dismiss after their `duration`, close buttons and
action buttons still work.
* studio/frontend: tighten code comments in selectable-toast change
* studio/frontend: address PR review on selectable-toast change
Three review-driven fixes:
1. CopyableErrorChip clears the copied->reset setTimeout on unmount via
a useRef + useEffect cleanup so setState cannot fire on an unmounted
component.
2. index.css restricts `cursor: text` to text-bearing toast nodes
(`[data-title]`, `[data-description]`, `p`, `span`). The toast
container keeps its default cursor and no longer pretends to be an
editable surface. `user-select: text` still applies to the full toast
tree so a drag-select starting on padding still works.
3. Toast wrapper now also injects `dismissible: false` into the second
argument of `toast.promise(p, data?)`, covering the loading /
success / error toasts created from a single promise call. Explicit
`dismissible: true` in the data continues to win.
A fourth review point asked us to drop the wrapper and instead pass
`toastOptions={{ dismissible: false }}` to <Toaster>. Sonner v2.0.7's
Toaster only forwards `duration`, `className`, `descriptionClassName`,
`closeButton`, `style`, `unstyled`, `classNames`, `cancelButtonStyle`,
`actionButtonStyle`, and `closeButtonAriaLabel` from `toastOptions`
(see index.mjs lines 1144-1164). `dismissible` is not forwarded, so the
global-option approach is a runtime no-op (verified empirically across
Chromium / Firefox / WebKit). Wrapper is required.
* studio/frontend: drop chip aria-label override so message reads via SR
The CopyableErrorChip trigger set a fixed `aria-label`, which overrides
the visible message in the accessibility tree. Inside the chat header's
`role="status"` region this caused screen readers to announce the
generic label instead of the actual model-load error, a regression
versus the old plain-text status div.
Removed the `ariaLabel` prop and the default override. The button's
visible message text is now its accessible name, so the full
(untruncated) error is announced. Truncation stays purely visual via
CSS. Caller in chat-page.tsx dropped the prop too.
Added a Playwright assertion that the trigger's accessible name
contains the error message across Chromium, Firefox, and WebKit.
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
* studio/frontend: grow chat composer to 16 rows and inset scrollbar
Raise the composer textarea cap from 6 to 16 rows so the input keeps
expanding as you type longer prompts. Also nudge the textarea in with
mt-2 / mr-3 so the internal scrollbar no longer sits flush against
the rounded edges of the chat composer surface.
* studio/frontend: lower composer cap from 16 to 12 rows
Keeps the composer growing past the previous 6-row cap while staying
conservative enough that a fully expanded textarea does not cover the
scroll-to-bottom button or a large slice of recent messages.
* studio/frontend: use symmetric mx-3 inset on composer-input
Replaces mr-3 with mx-3 (and width calc(100%-1.5rem)) so the textarea
sits inset from both edges of the chat composer surface. Keeps the
scrollbar tucked in regardless of writing direction: LTR scrolls on
the right, RTL scrolls on the left, and both edges are now ~16px in
from the surface (4px surface px-1 + 12px mx-3).
The spinner inside ThreadWelcome subscribed to the global
generatingStatus from chat-runtime-store, so any in-flight warmup
(or stale leak from a prior run) surfaced Generating on the empty
Chat with your model surface, even while the user was still typing.
On the normal path the welcome view is gone the moment a message
is submitted, and the assistant bubble already renders its own
per-message GeneratingIndicator. Remove the welcome-screen spinner,
its component, and the now-unused LoaderIcon import.
Co-authored-by: WhiskyAKM <35374730+PTFOPlayer@users.noreply.github.com>
PR #4611 originally proposed a community uninstall.sh for Unsloth
Studio. We folded that idea into the maintainer-authored
uninstall.sh (PR #5497) and uninstall.ps1 (PR #5513) which now ship
in main with safety guards, idempotency, lock-dir / .desktop / .app
cleanup, env-var precedence, tilde expansion, and CI coverage on
real Linux / macOS / Windows runners (PR #5536). Recording this
empty-commit merge so the original contribution from @PTFOPlayer
is attributed in git history.
* Fix num_logits_to_keep on transformers >= 4.51 + compile loss_function
Two follow-ups to the fused-forward work landed in unsloth-zoo PR #665.
1. unsloth_fast_generate (models/llama.py): transformers 4.51 renamed
num_logits_to_keep to logits_to_keep. Previously we unconditionally
set kwargs['num_logits_to_keep'] = 1, which transformers 4.57's
_validate_model_kwargs rejects with:
ValueError: The following `model_kwargs` are not used by the
model: ['num_logits_to_keep']
blocking model.generate() on Llama / Mistral. Now we inspect the
runtime forward signature and use whichever spelling it accepts;
if a caller still passes the legacy name we promote it to the new
spelling instead of stripping it.
2. patch_loss_functions (models/loader.py): the single internal call
site passed torch_compile=False. UnslothForCausalLMLoss is small
(label shift + Triton CE), so torch.compile folds the elementwise
prep into one launch and removes per-step Python overhead. The
< 2.4 fallback inside patch_loss_functions still routes through
torch._disable_dynamo so older torches are unaffected.
Verified:
- Llama 3.2 1B + model.generate() no longer raises; emits a sensible
16-token continuation.
- Gemma3 1B GRPO smoke (max_steps=3) returns bit-identical losses
0.256 / 0.4393 / 0.2031 vs pre-fix; train_runtime 409s (vs 415s
pre-fix, within noise).
- unsloth-zoo test_compiler_rewriter_exhaustive + test_fused_forward_install
pass (96 passed) on this combination.
Related: unslothai/unsloth-zoo PR for the compiler.py single-matmul
backport.
* Revert loader.py loss-compile flip; correct rename-version comment
Drop the patch_loss_functions(torch_compile=True) flip. Tracing the
loss call chain:
UnslothForCausalLMLoss
-> unsloth_fixed_cross_entropy
-> _fast_cross_entropy_loss
-> Fast_CrossEntropyLoss.apply (torch.autograd.Function wrapping Triton)
torch.compile treats custom autograd.Function.apply as an opaque op and
breaks the graph at the boundary. The only Python it can actually
compile in the loss function is the label-shift + ignore-fill prep
(three elementwise ops), and the per-call dynamo guard overhead is in
the same order as that prep. Empirical Gemma3 1B GRPO smoke (max_steps=3)
showed no meaningful runtime delta (415s vs 409s, within noise) and
risked dragging the outer compiled training step into recompiles when
the inner guards drift. Keep torch_compile=False; the Triton kernel is
the work, and it is unchanged either way.
Also: the inline comment in unsloth_fast_generate said the kwarg rename
landed in transformers 4.51. The actual decorator (@deprecate_kwarg)
was tagged version="4.50" and present through 4.51.x, then removed in
4.52+. Correct the comment. No behaviour change.
* studio: add uninstall.ps1 and document it in README for Windows
The previous Windows uninstall guidance was Remove-Item -Recurse -Force on
$HOME\.unsloth\studio, which only deletes the install dir and leaves
behind:
* %LOCALAPPDATA%\Unsloth Studio (data dir)
* Desktop\Unsloth Studio.lnk (Desktop shortcut)
* %APPDATA%\Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk
* Custom UNSLOTH_STUDIO_HOME / STUDIO_HOME roots
* Running unsloth_studio venv processes
* User PATH entry under .unsloth\studio
* HKCU\Software\Unsloth\PathBackup
This script mirrors uninstall.sh for Windows. It stops listening backends
by reading the port from share\studio.port (with a Win32_Process sweep
anchored on \unsloth_studio\ as a fallback), removes the install dir,
data dir, both shortcuts, the Studio PATH entry, and the PathBackup
registry key. Custom roots discovered from env vars or share\studio.conf
are accepted only if they contain a Studio sentinel (share\studio.conf,
unsloth_studio\.unsloth-studio-owned, or bin\unsloth.exe) and are not
on a hard deny list (drive root, %USERPROFILE%, parent of %USERPROFILE%,
or top-level system paths).
README now points Windows users at the script.
* Scope port-file kill and PATH cleanup to known Studio roots for PR #5513
Three findings from the reviewer round:
1. _StopByPortFile killed whatever owned the recorded port without proving
the PID belonged to this Studio install. A stale studio.port pointing
at a port a different local service later bound would force-kill that
service. New _PidUnderKnownRoot checks the listening PID's exe path
against the same $KnownRoots that _StopStudioProcesses already uses.
2. The netstat.exe fallback matched ":$port " anywhere in the line, so a
stale port file with 443 (or any common port) could match an
ESTABLISHED row whose remote endpoint was that port, killing an
unrelated process (browser, IDE). Now requires the row contain
LISTENING, and applies the same _PidUnderKnownRoot ownership check.
3. PATH cleanup removed any entry whose expanded path contained
\unsloth_studio\, which would also clobber an unrelated user virtualenv
that shared the name. Now only removes entries that resolve inside a
known Studio root (default %USERPROFILE%\.unsloth\studio plus any
custom roots discovered from UNSLOTH_STUDIO_HOME / STUDIO_HOME /
share\studio.conf).
* Expand tilde and honor UNSLOTH_STUDIO_HOME precedence for PR #5513
Two findings from the latest review round:
1. install.ps1 (lines 152-154) expands ~ and ~\path to $env:USERPROFILE
before resolving the install root, but uninstall.ps1 was passing the
raw env value to [System.IO.Path]::GetFullPath. That resolved ~\foo
relative to the current directory rather than the user profile, so a
user who installed with UNSLOTH_STUDIO_HOME='~\custom' could not
uninstall through the same variable. New _ExpandTilde helper matches
install.ps1's behavior.
2. Mirror install.ps1's env-var precedence: UNSLOTH_STUDIO_HOME wins,
STUDIO_HOME is ignored when both are set. Otherwise uninstalling
install A could also touch install B if the user has a stale
STUDIO_HOME pointing at B.
---------
Co-authored-by: Daniel Han <info@unsloth.ai>
* studio: register /settings route that opens the settings dialog
Navigating to /settings used to render Not Found because the route
was never registered. The settings dialog only opened via the user
menu, so /settings was a broken deep link if shared. Add a route
that calls useSettingsDialogStore.openDialog() and redirects to the
post-auth landing page so the modal appears on top of the chat.
* studio: harden Connections dialog provider sync and allow manual model IDs
Two related fixes for the Connections panel.
1. Keep localStorage providers when the server returns an empty list.
The dialog used to sync from /api/providers/ on mount and unconditionally
overwrite the Zustand provider store with the server result. When the
server had no enabled configs but the local store had entries (legacy
users, fresh dev installs, or providers created via earlier paths),
opening the dialog silently wiped them. The model picker reads from the
same store, so the chat header reverted from 'gpt-4o . OpenAI' to the
raw 'external::openai-1::gpt-4o' key. Treat the server as authoritative
only when it actually has rows; otherwise keep the local view.
2. Accept manual model IDs alongside the live catalog for remote-mode
providers (DeepSeek, OpenAI, etc.). Previously the only way to save was
to load the available-models catalog via a live API call, which fails
in air-gapped setups, behind 502s, or when the user already knows the
exact model ID. Add a Textarea fallback in the same render block, and
relax the validation to accept manual IDs even when availableModels is
empty. The validation message now points users at the manual path.
* studio: restrict manual model ID entry to openrouter among remote providers
Address review feedback: major remote providers (openai, anthropic,
gemini, mistral, cohere, deepseek, ...) expose large per-model
parameter surfaces that differ across models, so accepting pasted
model IDs leads to mismatched parameter expectations and frustrating
runtime errors. Keep their catalog curated by hiding the manual
textarea and falling back to the prior 'Load available models first'
validation toast for them.
OpenRouter drops unsupported parameters server-side, so manual entry
remains useful there; keep the textarea and the union save path for
it. Custom and curated backends already gated via isCustomProvider /
isCuratedModelList and continue to require manual entry as before.
* studio: shorten code comments in chat-providers-dialog.tsx
Trim three multi-line comment blocks to single lines per review.
* Studio update CI: round-trip install -> update -> uninstall
Adds an "Uninstall and verify clean" step to the three existing
studio-{,-mac-,-windows-}update-smoke.yml workflows so each one ends by
running uninstall.sh / uninstall.ps1 against the install it just
produced, then asserting that the install dir, launcher data dir,
desktop shortcut, CLI shim (and on Mac, the .app bundle) are all gone.
Two trailing reruns confirm idempotency. The uninstall log is added to
the existing artifact bundle.
Catches regressions where install.sh / install.ps1 starts writing to a
new path (registry key, Start Menu entry, %APPDATA% subdir, etc.) and
uninstall.{sh,ps1} has not been updated to match. Safety-guard
scenarios (refuse-\$HOME, refuse-non-Studio, tilde expansion, etc.) are
intentionally NOT exercised here -- those belong in a dedicated fast
smoke job that does not have to wait on a 5-15 min install.
Wall-clock overhead is ~30-45 s on each runner. Path filters extended
to include uninstall.sh / uninstall.ps1 so a pure uninstaller change
also triggers the round-trip check.
* Skip round-trip step when uninstall.{sh,ps1} are not in tree
---------
Co-authored-by: Daniel Han <info@unsloth.ai>
* studio: add uninstall.sh and document it in README
The current uninstall guidance in README.md is `rm -rf ~/.unsloth/studio`,
which leaves behind everything that lives outside that path:
- ~/.local/share/unsloth/ (launcher script, studio.conf, studio.log,
icon assets)
- ~/Applications/Unsloth Studio.app (macOS bundle, orphaned and
pointing nowhere on next reinstall)
- ~/Desktop/Unsloth Studio (broken symlink after the bundle is gone)
- ~/Desktop/unsloth-studio.desktop (Linux)
- ~/.local/share/applications/unsloth-studio.desktop (Linux)
- /tmp/unsloth-studio-launcher-<uid>*.lock (lock dir, possibly stale)
- Launch Services cache entry for ai.unsloth.studio on macOS
- Any running `unsloth studio -p N` processes
Users who follow the documented uninstall and reinstall end up with the
new launcher layered on top of stale state from the previous install,
which has produced concrete bugs (e.g. self-referential symlink inside
the .app bundle after a reinstall over leftover state).
Add uninstall.sh at the repo root that handles all of the above, and
update README.md to point at it as the recommended path. The plain
`rm -rf ~/.unsloth/studio` line is kept as a "partial uninstall, keep
launcher for a later reinstall" alternative. The model cache at
~/.cache/huggingface is intentionally left untouched, with a note in
the script suggesting how to remove it if desired.
Script is POSIX sh, idempotent (every removal is gated on existence
and uses `2>/dev/null || true`), and handles macOS, Linux, and WSL.
Windows is intentionally not covered here; the existing PowerShell
Remove-Item line in README is kept for that.
* studio: trim uninstall.sh header
* studio: address PR review feedback on uninstall.sh
Four findings from automated review, all verified real:
1. pkill pattern only matched `-p N`, not `--port N`. Studio
instances launched with the long option form survived the
uninstall. Fix: run two pkill passes, one for each form, with
`[ =]` covering both space and `=` separators.
2. CLI shim at ~/.local/bin/unsloth (symlink into the venv created
by install.sh:2167) was left behind, becoming a broken symlink
after the venv directory is removed. Fix: add it to the removals.
3. Custom install roots via UNSLOTH_STUDIO_HOME / STUDIO_HOME were
not removed. install.sh records the install location in
~/.local/share/unsloth/studio.conf as UNSLOTH_EXE; parse it,
derive the root as three dirnames up, and remove the root if it
is non-default.
4. On WSL the installer creates 'Unsloth Studio.lnk' on the Windows
Desktop and Start Menu Programs folder via powershell.exe.
Mirror that path on uninstall by invoking powershell.exe to
Remove-Item the same two locations. Best-effort, gated on
powershell.exe being available.
Tests (T2.8b, T2.15, T2.16, T2.17, T2.18, T2.5b) added behind the
scenes; all pass on macOS Darwin 25.3 with `dash -n`, `sh -n`,
shellcheck-clean (SC2016 suppressed on the PowerShell single-quoted
heredoc since the $env: expansions must remain literal to the
shell so PowerShell receives them verbatim).
* studio: harden uninstall.sh against env-mode and shim collisions
- Honor UNSLOTH_STUDIO_HOME / STUDIO_HOME at uninstall time and read
env-mode studio.conf at $<root>/share/studio.conf, not just the
default-mode conf under $HOME/.local/share/unsloth/. Without this,
installs done with a custom STUDIO_HOME leak the install tree even
when the env var is re-exported.
- Guard the custom-root resolver against "/" and empty so a corrupted
studio.conf (UNSLOTH_EXE='/etc/passwd' or similar) or an
UNSLOTH_STUDIO_HOME=/ cannot trick the script into rm -rf'ing root.
- Only remove $HOME/.local/bin/unsloth when it is a symlink resolving
to a Studio venv. pyproject.toml declares unsloth as a console
script, so pip install --user unsloth places a regular file at the
same path; the previous unconditional rm wiped that unrelated CLI.
- When neither env var is set, print a tail hint so users with custom
install roots know to re-run with the variable.
Verified with a sandboxed harness covering 24 scenarios (default and
env-mode installs across macOS / Linux / WSL, idempotency, hostile
lockfile names, path-traversal attempts, malformed conf, pkill long
and short forms, pip-conflict shim, broken-symlink bundle path).
Script remains POSIX (shellcheck -s sh clean, runs under /bin/dash).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Refuse non-Studio uninstall roots and tighten process matching for PR #5497
Three issues found while testing custom-root paths and process cleanup:
1. UNSLOTH_STUDIO_HOME=$HOME sh uninstall.sh rm -rf'd $HOME (same for
STUDIO_HOME and parent-of-$HOME). install.sh accepts any writable
directory for STUDIO_HOME, so the uninstaller must validate ownership
before deletion. _is_studio_root accepts a candidate root only if it
contains share/studio.conf, an unsloth_studio/ directory, or a
bin/unsloth shim pointing into unsloth_studio/bin. _is_unsafe_root is
a defense-in-depth deny list (/, $HOME, $HOME's parent, system paths).
2. pkill -f patterns "unsloth studio.*-p[ =][0-9]" over-matched on argv
substrings. A user running `less notes.md` whose filename contained
"unsloth studio ... -p N" had their less killed. New patterns anchor
on /unsloth_studio/bin/ so only processes whose actual exe lives in a
Studio venv match.
3. pkill missed processes that exec into studio/backend/run.py --port N
(the post-exec form when the unsloth CLI replaces itself). Added a
third pattern for that shape, and prefer PID files written by
install.sh's _spawn_terminal (studio-$port.pid in DATA_DIR) over
argv matching for installs that have them.
* Tighten ownership guards from review round for PR #5497
Three findings from the second reviewer round:
1. _is_studio_root accepted any directory containing an unsloth_studio/
subdir as Studio-owned. A user workspace that happens to contain a
folder named unsloth_studio/ would be deleted. install.sh's env-mode
guard at install.sh:1358-1361 already requires .unsloth-studio-owned
before treating the venv as replaceable. Mirror that: require the
owner marker, share/studio.conf, or the bin/unsloth shim target.
2. The pkill -f fallback patterns were global, so uninstalling install A
would also kill install B's running server. Scope each pattern to the
actual install root being removed by interpolating the root path into
the regex. Also adds a third pattern shape for `unsloth studio` with
no -p / --port flag (the CLI default-port form).
3. Desktop/Unsloth Studio is created by install.sh as a symlink to the
.app bundle. If a user has a regular directory by that name (photos,
notes, etc.), the previous _remove_path call rm -rf'd it. Now we only
remove it when it is a symlink or does not exist.
* Canonicalize env roots and honor UNSLOTH_STUDIO_HOME precedence for PR #5497
Two findings from the latest review round:
1. Canonicalize env-derived roots before the safety check. The deny list
only string-compares against $HOME, so a syntactic variant like
UNSLOTH_STUDIO_HOME=$HOME/../$USER (or trailing slash, or relative
path) bypassed _is_unsafe_root even though it resolves to $HOME. Now
_emit runs CDPATH= cd -P -- + pwd -P first, so all variants normalize
to the same canonical path before the deny check. Also added the same
tilde expansion install.sh's _resolve_studio_destinations does.
2. Mirror install.sh's env-var precedence (install.sh:282-290). When
both UNSLOTH_STUDIO_HOME and STUDIO_HOME are set, install.sh resolves
only UNSLOTH_STUDIO_HOME and ignores STUDIO_HOME. Uninstall was
emitting both, so running uninstall.sh for install A would also
delete install B if the user had a stale STUDIO_HOME pointing at B.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Han <info@unsloth.ai>
* studio/install: fix mac desktop shortcut spawning and lifecycle
The macOS .app generated by install.sh ships a shell-shim wrapper that
is unsigned and has no NSAppleEventsUsageDescription in its Info.plist,
so AppleEvents from the bundle are denied by TCC. The launcher's
`osascript ... tell application "Terminal" to do script ...` call
silently fails and the script falls back to the headless nohup branch,
where the user sees no Terminal window at all. Each click of the Desktop
shortcut then leaks an unattached server (no PID file, no cleanup) and
the launcher times out after 60s without ever opening a browser.
Replace the AppleScript spawn with a `.command` file + `open -a Terminal`.
Terminal handles `.command` natively through Launch Services, no
AppleEvents permission required, works with unsigned bundles.
The new design also decouples the studio server from the Terminal:
- Server is started via nohup, detached from any TTY. Warm relaunches
(server still alive) hit the existing fast path: the launcher's
`_find_healthy_port` returns the running port and the browser opens
in ~80ms with no Terminal involvement.
- The `.command` file is a log viewer (`tail -F` of studio.log), not
the server's parent. It also runs a watcher subshell that polls the
server PID and kills `tail` when the server exits. This means
clicking "Stop server" in the UI causes the Terminal window to drop
to no-running-processes state, so the user can close the window
without the "Do you want to terminate running processes" dialog.
- A trap on HUP/INT/TERM/EXIT in the `.command` file sends SIGTERM
(then SIGKILL at +0.5s) to the server PID, so closing the Terminal
window also stops Studio. Best of both worlds: fast warm relaunch
AND "close terminal == quit Studio".
Also:
- Drop POLL_INTERVAL_SEC from 1 to 0.25. With Python studio startup
at ~2s, the 1s poll added up to 1s of slack between server-ready
and browser-open. 0.25s tightens cold-launch latency at no
meaningful CPU cost.
- Refuse to install the `.app` bundle through a symlink. If a prior
install (e.g. a --tauri build) left $HOME/Applications/Unsloth\\ Studio.app
as a symlink, mkdir -p follows it and writes the new bundle contents
through to the target. Detect and rm the symlink before mkdir -p.
Test plan:
- Existing studio-mac-update-smoke.yml CI runs install.sh end-to-end
on macos-14 and asserts /api/health returns healthy.
- Manual: click Desktop shortcut from cold state, Terminal opens with
logs streaming, browser opens at ~2s. Re-click while Studio still
running, browser opens in <200ms, no new Terminal. Click "Stop
server" in the UI, Terminal closes cleanly with no prompt. Close
Terminal via Cmd+W, server stops within 1s.
* studio/install: trim verbose comments in _spawn_terminal
* studio/install: harden trap quoting in generated .command
The trap bodies in the .command file were written with broken
quoting:
trap "rm -f "$PID_FILE" 2>/dev/null" EXIT
Shell parses this as three concatenated tokens ("rm -f " + unquoted
$PID_FILE + " 2>/dev/null") then runs the trap. With paths that
contain spaces, the unquoted expansion word-splits and the rm
either no-ops or removes the wrong path. Default $HOME has no
spaces so the bug is latent, but it should be space-safe.
Switch both trap bodies to single-quoted form so $WATCHER_PID,
$TAIL_PID, and $PID_FILE expand at signal time inside properly
quoted positions. Shellcheck-clean on the generated .command.
* studio/install: exec studio in nohup wrapper so PID is the server
Without the explicit exec, `nohup sh -c "$_cmd"` runs `_cmd` as a
child of the wrapper shell. Whether sh exec-optimizes that single
command is shell-specific (macOS /bin/sh does, dash does, some bash
configurations do not). When the optimization does not fire, `$!`
records the wrapper PID rather than the studio PID, so:
- the watcher in the generated .command monitors the wrapper, not
the actual studio process; closing the Terminal can leave studio
running if the wrapper exits first
- SIGTERM from shutdown_studio goes to the wrapper rather than the
server
Force the replacement with exec so the recorded PID is always the
studio process regardless of shell version.
Flagged by both gemini-code-assist and codex in PR review; verified
correct.
* Fix orphan-on-spawn-failure, graceful kill, and nested symlink for PR #5496
Three issues found while testing the new macOS spawn path:
1. _spawn_terminal returned 0 even when 'open -a Terminal' failed, so
the nohup'd server was left orphaned with no Terminal owner. Wrap
the .command write + chmod + open chain in 'if {...}; then return 0;
fi', and on failure SIGTERM the orphan (with a 3s grace) before
falling through to the generic terminal-spawn fallback.
2. The generated .command sent SIGKILL only 0.5s after SIGTERM, shorter
than studio/backend/run.py's _graceful_shutdown windows (5s inference
+ 5s export). Wait up to 12s for the server to exit on its own.
3. The .app symlink guard only checked the top-level path. If a prior
corrupted install left Unsloth Studio.app/Contents (or its MacOS or
Resources children) as a symlink, mkdir -p still wrote through them.
Check all four bundle paths, and refuse to continue if the bundle
path exists as a regular file.
---------
Co-authored-by: Daniel Han <info@unsloth.ai>
* Studio: gate image input on a usable mmproj for GGUF vision models
* Improve image gating and model capability sync
Tighten image-handling and model capability syncing across the chat flow. Key changes:
- chat-adapter: Replace per-message current-user image check with a simpler gate that blocks if ANY image is present in the outbound payload when the selected model cannot handle vision. Show the toast reason and flip the per-thread running flag on→off to avoid hanging wait promises before throwing.
- shared-composer: Simplify and correct image-attachment gating for single vs compare modes. Use an attach-time gate that defers to send/ensureModelLoaded in compare mode, introduce attachUnavailableReason, and only block immediately for single-mode. Remove an unused models selector.
- shared-composer: Sync the runtime models[] entry with the response from ensureModelLoaded so UI/send gates read fresh capabilities (isVision, isGguf, isAudio, audioType, hasAudioInput). This addresses catalog lag (e.g., GGUF mmproj arriving after the catalog snapshot).
- UX tweak: the file-picker button no longer outright blocks on image availability; addFiles still filters images per-file and toasts appropriately.
These changes prevent mid-stream server rejections, avoid deadlocks, and ensure model capability checks are accurate when attaching images or audio.
* studio: only pass --mmproj to llama-server when effective_is_vision
When a text-only GGUF (static is_vision=False) was paired with a
family-matching mmproj path, the launcher appended both --mmproj and
--spec-default, leaving llama-server in an inconsistent state while
Studio reported is_vision=False. Gate the --mmproj flag on
effective_is_vision so the launch command tracks the runtime
capability the rest of Studio sees.
* studio: reject image content in streaming /v1/responses for non-vision GGUF
_responses_stream forwards the OpenAI request body directly to
llama-server's /v1/chat/completions, bypassing the image-vs-vision
guard that openai_chat_completions enforces for the wrapped path.
Add the same check at the top of the streaming entry point so an
SDK client that posts an image to a non-vision GGUF receives a
typed 400 instead of an opaque downstream error.
* studio: gate external chat providers in the image input helper
External selections (cohere, deepseek, mistral, openrouter, ...) live
in externalProviders, not in runtime.models[], so activeModel is
undefined for them and the helper short-circuited to allow. Result:
images attached to a non-vision external chat model were dropped
silently downstream instead of rejected up front.
Add providerTypeSupportsVision to external-providers.ts (false for
known text-only providers, true for known vision-capable ones, null
for unknown / custom self-hosted) and thread externalSupportsVision
+ externalModelLabel through the helper. shared-composer.tsx,
runtime-provider.tsx (VisionImageAdapter.add), and chat-adapter.ts
pre-stream gate all resolve the provider type and pass it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/worker): inject --gcc-install-dir for HIP source builds on Ubuntu 24.04
On Ubuntu 24.04 + ROCm clang-20, the HIP source-build fallback in
`_install_package_wheel_first` (causal-conv1d, mamba-ssm source fallback,
flash-attn source fallback) dies at:
/opt/rocm-X.Y/lib/llvm/lib/clang/20/include/__clang_hip_runtime_wrapper.h:112:10:
fatal error: 'cstdlib' file not found
Root cause: clang-20 picks the highest-numbered /usr/lib/gcc/x86_64-linux-gnu/<N>
runtime dir by default. On 24.04 that's gcc-14, whose runtime objects ship in
the gcc-14 package but whose C++ headers (/usr/include/c++/14) come from
libstdc++-14-dev — NOT in the default apt set. libstdc++-13-dev IS in the
default set, so /usr/include/c++/13 exists. clang has no way to discover
that asymmetry and the build fails.
Fix: new `_hipcc_gcc_install_dir()` helper iterates gcc 14 → 11 and returns
the first /usr/lib/gcc/x86_64-linux-gnu/<N> dir where BOTH the runtime AND
/usr/include/c++/<N> exist. The HIP branch of `_install_package_wheel_first`
appends `--gcc-install-dir=<that path>` to HIPCC_COMPILE_FLAGS_APPEND before
invoking pip. Respects an existing `--gcc-install-dir` in the env var
(user-set takes precedence); preserves any other flags the user has set
(appends to the end rather than overwriting). No-op on non-HIP, non-Linux,
non-x86_64.
Mirrors the same fix bbf004c added to studio/setup.sh for the llama.cpp HIP
build branch (#5301), but via env var since pip-driven source builds can't
take CMake flags directly.
Verified on Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151) / Ubuntu 24.04 /
ROCm 7.13 nightly: `_hipcc_gcc_install_dir()` returns
`/usr/lib/gcc/x86_64-linux-gnu/13`, which matches the manual workaround
that already lets `pip install causal-conv1d` succeed on this hardware.
Tests added (8 new in test_training_worker_flash_attn.py):
- test_hipcc_gcc_install_dir_picks_highest_with_headers
- test_hipcc_gcc_install_dir_picks_14_when_headers_exist
- test_hipcc_gcc_install_dir_returns_none_when_no_match
- test_hipcc_gcc_install_dir_returns_none_on_non_linux
- test_hipcc_gcc_install_dir_returns_none_on_non_x86_64
- test_install_injects_gcc_install_dir_on_hip_source_build
- test_install_appends_to_existing_hipcc_compile_flags
- test_install_respects_user_gcc_install_dir
- test_install_does_not_inject_env_on_cuda
Per @danielhanchen's suggestion in
https://github.com/unslothai/unsloth/pull/5434#issuecomment-4469980122
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: apply gemini-code-assist suggestion on _run_kwargs env handling
Use _run_kwargs.get("env", os.environ).copy() + key-mutation instead of
rebuilding env from os.environ directly. Today both forms are equivalent
(no earlier code in _install_package_wheel_first sets _run_kwargs["env"]),
but the .get().copy() pattern survives any future env modification added
upstream of this block without silently throwing it away.
No behavioural change; tests already assert the final HIPCC_COMPILE_FLAGS_APPEND
value, not the env-construction pattern.
Per https://github.com/unslothai/unsloth/pull/5517#discussion_r... (gemini-code-assist[bot])
---------
Co-authored-by: h34v3nzc0dex <h34v3nzc0dex@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Fix ORPO text tokenization with processors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard ORPO tokenizer rewrite anchor
* Resolve processor pad_token_id and preserve preference data collators for ORPO
Two follow-ups so the text-only ORPO + VL processor path works end to end on
top of the build_tokenized_answer and tokenize_row rewrites:
1. Add orpo_trainer_processor_pad_token to rewrite processing_class.pad_token_id
in ORPOTrainer.__init__ to fall back to processing_class.tokenizer.pad_token_id
when the processor itself has no pad_token_id (Qwen3-VL, Gemma-3, etc.).
Without this, DPODataCollatorWithPadding(pad_token_id=processing_class.pad_token_id)
raises AttributeError before training starts.
2. Stop the outer UnslothORPOTrainer.__init__ collator-swap from clobbering
DPODataCollatorWithPadding when the tokenizer is a processor without .pad.
The swap to TransformersDataCollatorForLanguageModeling is now only applied
to LM-style collators, so ORPO/DPO/CPO/KTO keep their own prompt/chosen/
rejected handling. Otherwise the collator can't pad ORPO rows and raises
"You should supply an encoding ... that includes input_ids" at train time.
Verified with Qwen3-VL-2B-Instruct ORPO + text-only data (training completes
to max_steps, no AttributeError, no collator error) and Llama-3.2-1B-Instruct
ORPO (losses and grad-norms bit-exact identical to main, so the change is a
true no-op for plain text tokenizers).
Extends tests/python/test_orpo_processor_text_tokenizer.py with three new
unit tests covering the pad_token_id rewriter.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: extend offline DNS auto-detect to inference parent + training
#5505 fixed the GGUF/llama-server load path. Studio still has two
adjacent code paths that burn ~30-60s of soft-failed timeouts before
the worker subprocess starts when DNS to huggingface.co is dead and
the model is already in the local HF cache.
Inference parent process (routes/inference.py:load_model):
* ModelConfig.from_identifier now runs inside _hf_offline_if_dns_dead
so the LoRA-detect hf_model_info call and the urllib config probes
in utils/transformers_version.py short-circuit when DNS is dead.
* utils/models/model_config.py: extracted the inline HF_HUB_OFFLINE/
TRANSFORMERS_OFFLINE check used by list_gguf_variants and
detect_gguf_model_remote into a shared _env_offline() helper, then
reused it to gate the LoRA-detect hf_model_info call.
* utils/transformers_version.py: _check_tokenizer_config_needs_v5 and
_check_config_needs_550 now early-return False when offline instead
of issuing a 10s urllib.urlopen against huggingface.co/raw/main.
Training worker (core/training/worker.py:run_training_process):
* Add the same 2s DNS probe used by core/inference/worker.py at the
top of the training subprocess. On failure, set HF_HUB_OFFLINE,
TRANSFORMERS_OFFLINE, and HF_DATASETS_OFFLINE before the rest of
the subprocess imports torch/transformers/unsloth, so every
from_pretrained, snapshot_download, and load_dataset call below
resolves from cache. Scope is per-subprocess; the orchestrator
always spawns a fresh worker per training run.
Training trainer (core/training/trainer.py:load_model):
* Skip the proactive hf_model_info gated-repo probe when _env_offline()
is true. The API is unreachable anyway, and a gated model that is
already cached is exactly the scenario the user is trying to train
against. from_pretrained surfaces the real error if access is
actually denied.
Tests (tests/test_offline_inference_parent.py, 7 new cases):
* _env_offline truthy/falsy parsing across HF_HUB_OFFLINE and
TRANSFORMERS_OFFLINE.
* transformers_version urllib short-circuit when offline.
* LoRA detect hf_model_info skip when offline.
Existing tests/test_offline_gguf_cache_fallback.py still passes
(26 cases) because the inline env check was extracted, not changed.
* tests: prefer real httpx over stub in offline-test files
The studio test stub convention only included the 6 httpx exception
names that existed callers needed. Newer huggingface_hub (1.15+)
imports HTTPError, Response, Request, HTTPStatusError, AsyncClient,
and more at module import time. When httpx is truly absent the stub
chase becomes a treadmill.
Use the real package when installed (the CI install list already
includes httpx, so this is the production environment). Fall back to
the stub only when httpx is genuinely missing.
No code under test changes.
* studio: detect cached LoRA adapters offline; tighten test
Two follow-ups from the review pass on #5512:
* ModelConfig.from_identifier no longer skips the remote LoRA-detect
hf_model_info call when _env_offline() is true. huggingface_hub
short-circuits the call via OfflineModeIsEnabled in ~0ms when
HF_HUB_OFFLINE is set, so the original 25s concern was moot once
routes/inference.py wrapped the call in _hf_offline_if_dns_dead.
Skipping the API meant users with a cached LoRA adapter
(adapter_config.json on disk) got is_lora=False and the load
failed. After the API call (which raises fast offline) a new
cache-fallback walks the HF cache snapshot for adapter_config.json
via the existing _iter_hf_cache_snapshots helper.
* test_hf_model_info_not_called_when_offline replaced. The old test
raised AssertionError inside production code that catches Exception,
so it passed even if the call happened. New tests use MagicMock and
assert call_count >= 1, plus a fixture that stages a fake HF cache
with adapter_config.json to verify the offline cache detection.
Test count goes from 7 to 8 in test_offline_inference_parent.py.
Combined with test_offline_gguf_cache_fallback.py: 34 pass in 9.75s.
* Fix/adjust offline training DNS probe per PR #5505 review
Same fix as #5505's _probe_dns_dead refactor: run gethostbyname on a
daemon thread with join timeout so concurrent sockets in the parent
interpreter never inherit a process-wide socket.setdefaulttimeout
mutation. Adds a static-pin regression test that the inference parent
file does not regress on this.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose code comments per review feedback
Shorten the longer explanatory comments added by this PR while keeping
the WHY of each non-obvious branch:
- trainer.py: collapse the 5-line proactive gated-check comment.
- training/worker.py: trim the offline auto-detect preamble and the
"logger isn't configured" note.
- routes/inference.py: shorten the DNS-probe wrap rationale.
- transformers_version.py: collapse the two urllib short-circuit notes.
- model_config.py: shorten the LoRA detect + cache-fallback notes.
- tests/test_offline_inference_parent.py: tighter module docstring,
trim class docstrings, drop multi-line explainer comments inside the
tests; behaviour and coverage unchanged (9/9 tests still pass).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: warn when llama.cpp prebuilt is at least 3 days behind
Layered on #5528. Generalises the MTP-specific staleness warning to
every llama.cpp prebuilt update, not just the ones that add MTP. If
the installed prebuilt is at least 3 days old AND its tag differs
from the latest published tag on the helper release repo (default
unslothai/llama.cpp), Studio nudges the user to run
"unsloth studio update".
How it works
Reads the install marker UNSLOTH_PREBUILT_INFO.json that
install_llama_prebuilt.py already writes to install_dir. The marker
carries the installed tag, the helper repo, and an installed_at_utc
timestamp. Studio compares those against the latest published tag
from the GitHub releases API for the helper repo.
GitHub fetch is cached at two levels:
- Process-level memo for /status hot path.
- Disk-level cache (24h TTL) at ~/.unsloth/studio/cache/llama_cpp_freshness/
so cold-start Studio launches do not always hit the API.
On a transient fetch failure (offline, rate-limited) we keep the
last-good disk value alive rather than poisoning the cache with None.
The check fails open: if anything is missing (marker, timestamp,
GitHub response), stale stays False so users never see a misleading
banner.
Surfaced in two places
1. Startup banner (logs + stderr) in main.py:lifespan(), alongside the
MTP capability probe added in #5528. Single line, e.g.:
WARNING: llama.cpp prebuilt is 5 days behind: installed b9190,
latest b9300. Run "unsloth studio update" to refresh.
2. /api/inference/status now returns:
llama_cpp_prebuilt_stale: bool
llama_cpp_installed_tag: str | None
llama_cpp_latest_tag: str | None
so the frontend can render a banner / popup with the actual tag
delta the user is missing.
3-day threshold
Mirrors the typical Unsloth llama.cpp release cadence. Anything
shorter would nag users who restart Studio at the wrong moment;
longer leaves real bugs sitting on the user's machine. Configurable
via the threshold_days kwarg if a future call site wants a different
window.
Tests
17 new cases in tests/test_llama_cpp_freshness.py cover marker
discovery in both cmake and root install layouts, missing / invalid
marker, GitHub fetch caching across process restarts (disk cache hit
after the in-memory cache is reset), the stale / not-stale decision
matrix (tag mismatch + age threshold), fail-open behaviour when
GitHub is unreachable, custom threshold, singular/plural day in the
warning string, and unparseable installed_at_utc. The broader
205-test inference regression suite still passes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: warn when llama.cpp prebuilt is too old for MTP
Layered on #5527. Adds a one-shot llama-server --help capability probe
so users get a clear signal when their prebuilt is missing MTP support,
plus a graceful fallback if they load an MTP GGUF against an outdated
binary.
What's surfaced:
1. Startup log + stderr line in main.py:lifespan() if MTP isn't
advertised:
WARNING: llama.cpp prebuilt is missing MTP support
(--spec-type mtp / draft-mtp). Run `unsloth studio update` to
refresh it. MTP GGUFs will load without speculative decoding.
2. Load-time graceful fallback in load_model's spec block: skip the
auto-emit and log a clear warning instead of letting llama-server
fail with an unknown-flag error.
3. /api/inference/status now returns llama_cpp_supports_mtp: bool so
the frontend can show a banner / popup.
Probe internals:
- Class-level cache keyed on (binary_path, mtime). One subprocess call
the first time, instant thereafter. Touching the binary (e.g. via
`unsloth studio update`) invalidates the cache automatically because
the mtime changes, so the new build is picked up without restarting
the server.
- Recognises both upstream naming forms: the original draft-mtp from
llama.cpp PR #22673 and the renamed mtp variant in later commits.
- Spec block uses whichever token the binary accepts so we emit the
right value regardless of which release the user has.
Tests:
- 6 new cases in test_llama_cpp_mtp_detection.py covering each probe
variant (draft-mtp, renamed mtp, pre-MTP build, missing binary,
mtime-based cache invalidation).
- Existing 38 MTP detection cases still pass; broader 188-test
regression suite (server args, reload inheritance, gguf metadata,
load progress, context fit, model validation) still green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
danielhanchen