The header docstring advertises UNSLOTH_GPUS values like "0" and "0,1"
but Docker reads a bare integer for --gpus as a COUNT, not an INDEX.
UNSLOTH_GPUS=0 was therefore exposing zero GPUs, and the entrypoint's
GPU check refused to start. Wrap bare-int and comma-list inputs as
"device=$GPUS" so the documented values do what they say; "all" and
already-quoted device= selectors pass through unchanged.
Previously, UNSLOTH_REF was pinned to the triggering tag (e.g. v2026.5.8)
but UNSLOTH_ZOO_REF was hardcoded to main. That made release-tag images
ship a zoo from whatever was on main at build time rather than the zoo
release cut alongside that unsloth tag, so a 2026.5.8 tag image could
install a zoo from days later. Mirror the tag branch of UNSLOTH_REF.
SHA-based branch pushes still fall through to main because the unsloth
SHA does not exist in the unsloth-zoo repo. workflow_dispatch still
honours the unsloth_zoo_ref input.
* tests: unblock three stale assertions broken on main
MLX CI on Mac M1 + Backend CI (both Repo tests CPU and Python 3.10/11/12/13)
have been red on every push to main for days. None of the underlying code
is wrong; three test files have stale anchors / assertions left behind by
PR #5537 (max_steps bump) and PR #5775 (composer + provision-desktop-auth).
1. tests/studio/run_real_mlx_smoke.py:393
PR #5537 bumped max_steps from 7 to 30 for seed-robust convergence but
left `assert len(losses_per_step) == 7`. With logging_steps=1 the
callback fires once per step; 30 entries, not 7. Track config.max_steps
so the gate auto-follows future bumps.
2. tests/studio/test_composer_rtl_bidi_attribute.py:29
PR #5775 changed the composer aria-label from the literal
`aria-label="Message input"` to a JSX ternary
`aria-label={overlay ? "Image edit instructions" : "Message input"}`.
Anchor on the inner string literal `"Message input"` instead.
3. studio/backend/tests/test_desktop_auth.py:487
The guarded_import in test_provision_desktop_auth_writes_secret_and_creates_db_without_backend_deps
blocks any import whose name == "utils", including the relative
`from .utils import echo` inside typer._click.decorators (typer 0.25+).
Gate the block on level == 0 so only absolute imports of `utils` /
`auth` / `fastapi` / `structlog` are rejected; relative imports
inside third-party packages pass through.
All three tests pass locally; the MLX one is a mechanical 7->config.max_steps
swap and will be exercised by MLX CI on this PR.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: expose --parallel / -np on `unsloth studio run`
The CLI was hardcoding `llama_parallel_slots=4` in `run_kwargs` at
`unsloth_cli/commands/studio.py`, leaving users unable to tune the
concurrent decode slot count even though the engine, KV-cache math,
and `studio.backend.run.run_server(llama_parallel_slots=...)`
plumbing all already accepted any N. This change adds a `--parallel`
/ `--n-parallel` / `-np` typer option (default 4 -- matches the
previous hardcoded value), forwards it into `run_kwargs`, and pins
the new surface with 4 unit tests.
Per-request state in `routes/inference.py` is already isolated
(`cancel_event` and `prev_text` are per-request locals in every
streaming handler; the `_lock` / `_serial_load_lock` only wrap
load/unload, not chat completions), so no concurrency refactor is
needed alongside this -- the engine layer already handles N
concurrent requests on one loaded model when llama-server is told
to.
Range guards: 1 <= N <= 64. With higher N each slot gets ctx/N KV
cache; users tuning this should be aware that per-call context
shrinks proportionally.
`unsloth studio` (the bare default command, no subcommand) still
defaults to llama_parallel_slots=1 via `run_server`'s own default;
this PR does not change that path -- it only exposes the knob on the
one-liner `studio run` command that already silently used 4.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Forward --parallel through venv re-exec and drop colliding short aliases
`unsloth studio run` re-execs into the Studio venv when invoked from
outside it (the common path). The arg-builder forwards every typer
option but the new --parallel, so the child re-execs at the default 4
and any user value is silently dropped. Worse: pre-PR users who
already pass `-np N` as a pass-through extra (where llama.cpp's
last-wins parsing made it stick) silently lose N after this PR lands.
Forward --parallel explicitly in the re-exec arg list.
While auditing the re-exec path, also drop the colliding 1-char
short aliases -m (--model) and -f (--frontend) plus the redundant
-hfr. Click's short-option clustering had been silently mis-parsing
~11 llama-server short flags via the pass-through path: -fa as
`-f a`, -mg 0 as `-m g` + stray 0, -fitt 1024 as `-f itt` + stray
1024, -hff path as `-f f` + stray `-h path`, -cmoe / -cram / -sm /
-ncmoe etc. The docstring promise ("any flag this command does not
recognize is forwarded verbatim") was silently violated.
-hf (2-char) is kept because Click treats multi-char shorts atomically
(no clustering of -hff / -hfv / -hffv / -hft) and -hf is documented
in basics/api/README.md. --model / --hf-repo / --frontend long forms
all unchanged. studio_default keeps -f because it has no pass-through.
Tests:
- test_studio_run_parallel_flag.py: 8 new re-exec coverage cases
(all 3 aliases, 3 platforms via sys.platform mock, pre-PR `-np`
regression, mixed with pass-through extras).
- test_studio_run_short_alias_clashes.py (new): surface checks that
the removed shorts cannot reappear, plus 11 parametrized cases
proving each previously-broken llama-server short flag now passes
through verbatim, plus a happy-path test that documented -hf still
works for `org/repo:variant` syntax.
All 27 tests pass. Negative test (revert either fix) shows the new
tests catch the regression.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stale studio run docstring describing rejected llama-server flags
The pre-PR docstring listed --port, -c / --ctx-size, --api-key, -ngl,
--jinja, --flash-attn, --no-context-shift as "rejected with HTTP 400",
but only --port and --api-key (plus other networking / auth / model
identity / single-model UI flags) are actually in
studio/backend/core/inference/llama_server_args.py's denylist. -c /
-ngl / --jinja / --flash-attn / --no-context-shift are pass-through
and last-wins-override Studio's auto-set value.
Rewrite the docstring to match the real denylist groups and point at
the canonical source. Also add --parallel to one of the examples now
that it is a first-class flag.
* ci: broaden Linux + narrow Windows llama.cpp runtime patterns + trim #5741 comments (#5746)
* ci: broaden Linux llama.cpp runtime pattern to lib*.so*
#5741 patched the explicit Linux pattern list to add
``libllama-*-impl.so*`` after ggml-org/llama.cpp#23462 (between
b9279 and b9283) split each binary's entry code into a paired
``lib<binary>-impl.so`` shared library. Same class of upstream
repackaging will hit us again whenever a new shared lib is added.
Mirror what macOS already does and replace the per-lib list with a
single ``lib*.so*`` glob. ``copy_globs`` (line 3614) unions
patterns, so the per-variant ``libggml-cuda.so*`` / ``libggml-hip.so*``
entries were never filtering anything; the spec lives in
``runtime_payload_health_groups`` (line 5209) which keeps the
explicit minimum-required list per variant.
Dry-run against b9296-bin-ubuntu-x64.tar.gz: 40 files copied (all
ggml, llama, mtmd, impl variants + the two binaries we ship), 22
skipped (other CLIs, rpc-server, LICENSE). Functionally equal to
the post-#5741 set.
* cleanup: trim #5741 comments on the pydantic split
Comments added in #5741 explained the original bug in full each
time. They are mostly redundant with the commit message and the PR.
Trim them to one short paragraph per site.
No behavior change.
* ci: narrow Windows runtime pattern to llama-server.exe + llama-quantize.exe
Studio only invokes llama-server and llama-quantize. Mac and Linux
already filter to those two binaries; Windows was the odd one out
with ``*.exe`` copying every CLI upstream ships (llama-cli,
llama-bench, llama-mtmd-cli, ...).
Dry-run on b9296 (win cpu-x64, cpu-arm64, cuda-13.1, hip-radeon):
20 unused EXEs skipped per variant, all DLLs (incl. the new
llama-*-impl.dll family) still copied via ``*.dll``.
``existing_install_matches_choice`` already checks llama-server.exe
exists explicitly (line 5297), so the health gate is unchanged.
* Lower default weight_decay in RL config from 0.01 to 0.001 (#5747)
In full FT, AdamW weight decay shrinks the parameter directly so the
implicit prior is W -> 0. In LoRA the trained parameters are A and B
while the effective weight is W = W_init + (alpha/r) * B @ A; decaying
A and B separately drives BA -> 0, hence W -> W_init rather than 0.
The previous default of 0.01 inherited from full-FT recipes adds a
measurable pull on the merged adapter back toward the base model over
a few thousand steps. 0.001 keeps a small Frobenius-norm prior on
||A||^2 + ||B||^2 for numerical stability without meaningfully biasing
the merged weight toward init, and aligns with the value used across
the unsloth notebook templates.
* Studio: strip orphan tool_call XML leaking into visible content (#5735)
* Studio: strip orphan tool_call XML from streamed visible content
The speculative-buffer state machine in
`studio/backend/core/inference/llama_cpp.py` can slice a tool_call XML
block between the silent DRAINING path and the user-visible
content_accum, depending on when in the model's emission the BUFFERING
-> STREAMING -> DRAINING transitions fire. Three leak shapes were
observed in a 2026-05-22 sweep of 900 Qwen3.5 / Qwen3.6 GGUF runs:
Pre-fix XML leak rate: 20/900 (2.22%), concentrated 6.7% on the
larger Q8 / MTP configs:
Qwen3.6-35B-A3B Q8_0 4/60 (6.7%)
Qwen3.6-35B-A3B-MTP Q4 4/60 (6.7%)
Qwen3.5-35B-A3B Q8_0 3/60 (5.0%)
Qwen3.6-27B Q8_0 3/60 (5.0%)
The existing `_TOOL_XML_RE` only matched well-formed
`<tool_call>...</tool_call>` and `<function=...></function>` pairs, so
unterminated openings (close was DRAINED) and orphan closes (opening
was DRAINED) survived the strip and reached the user.
Fix relaxes the regex to also strip:
1. Orphan opening up to end-of-string: `(?:</tool_call>|\Z)`
2. Orphan closing tag: bare `</tool_call>` / `</function>`
Verified on the full sweep: 20/900 -> 0/900 (100% of detected leaks
eliminated). 16 unit tests in `test_tool_xml_strip.py` pin all three
leak shapes plus the well-formed cases, plus parametrised checks on
the 5 actual real-world leak samples from the sweep data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip tail-only </parameter> orphan + tighten regex
The 2026-05-22 gdpval sweep surfaced a 4th XML-leak shape not caught
by the earlier regex: a bare `</parameter>\n\n` at end-of-buffer (7
of 192 trials, all Qwen3.5-27B + a few Qwen3.6-27B). The model emits
the full `<tool_call><function=...><parameter=...>...content...
</parameter></function></tool_call>` envelope, the speculative buffer
DRAINS the opening tags as intended, but EOS (max_tokens cutoff)
truncates the outer `</function></tool_call>` close, leaving just
`</parameter>` as the visible tail.
We strip this ONLY when end-anchored (`\s*\Z`) so legitimate
mid-text uses (user code samples, documentation discussing the
Qwen tool-call XML shape) survive. Verified on the 192-trial
gdpval corpus: before=7, after=0.
While at it, fold the five top-level alternations into three by
sharing tag-name and prefix subgroups:
<tool_call>... + <function=\w+>... + --> <(?:tool_call|function=\w+)>...
</tool_call> | </function> --> </(?:tool_call|function)>
Semantically identical (verified by replay over the 192-trial
corpus + adversarial inputs, 0 diffs) and 1.34x faster on real
workloads. Backtracking-safety pinned by two new perf guards
(256KB '<' spam, 1000x orphan opens).
Tests: 16 -> 28 (6 new functional + 4 well-formed-vs-orphan +
2 perf guards).
* Tighten comments in XML-strip regex and tests
Code says what it does; comments were repeating it. Strip the verbose
explanations down to the WHY-only bits (engine quirk, tail-anchor
rationale, real-world source of each test sample). No code changes.
inference.py: 21 -> 12 lines around _TOOL_XML_RE
test_tool_xml_strip.py: 343 -> 259 lines (-84)
Tests: 28/28 still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Address review: deny pass-through --parallel, preserve legacy short aliases, fix test harness
Round 1 review fixes for #5737:
1. Deny --parallel / --n-parallel / -np in the pass-through validator.
Without this, `unsloth studio run --model X --parallel 8 -- --parallel
999` would last-win-override the running llama-server slot count while
Studio's app.state.llama_parallel_slots and KV-cache fitting stay at
the typer value (8), so the resource plan and the running process
disagree. Also bypasses the typer 1..64 range guard. Reject so the
only path is the first-class typer flag.
2. Backwards-compat shim for -m / -hfr / -f. Dropping the short aliases
from typer broke any script using `unsloth studio run -m X` or
`-hfr Y` or `-f dist`. Add _consume_legacy_short_aliases which pops
EXACT whole-token matches (or `-x=value` inline form) from ctx.args
into the corresponding typer parameter. Clustered tokens (`-fa`,
`-mg`, `-fitt`, ...) are left in the pass-through tail unchanged.
--model becomes Optional with an explicit missing-required check
after the preprocessor so legacy `-m X` still satisfies the
"must specify a model" requirement.
3. Drop mix_stderr from CliRunner. Typer 0.25.1 / Click 8.4.1 removed
the kwarg; the test harness raised TypeError before exercising the
PR behaviour. Tests run cleanly on current and older Typer/Click.
4. Correct the -np regression test docstring. Pre-PR `-np 8` was
clustered by Click as `-p 8` (port=8) + stray `-n`, silently
breaking the port binding -- not "passed through as 8 slots". The
post-PR assertion (child gets --parallel 8) is unchanged.
5. Update studio run docstring listing rejected flags so it now
correctly includes --parallel / -np / --n-parallel.
New tests:
- test_llama_server_args.py: parametrized denylist coverage for
--parallel / --n-parallel / -np including equals-form, including
out-of-range bypass attempts (999, 0). is_managed_flag flips True.
- test_studio_run_short_alias_clashes.py: legacy -m / -hfr / -f
promote to typer params; --model X + -m Y conflict errors; clustered
-mg / -fa / -fitt still pass through (the original bug fix holds).
132 tests pass (98 backend + 34 cli).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Extend legacy-alias shim tests for repo:variant, inline value form, and missing model
Three additional edge cases for the -m / -hfr / -f preprocessor:
- `-m unsloth/foo:UD-Q4_K_XL` round-trips through both the preprocessor
and _split_repo_variant so the child sees --model + --gguf-variant.
- `-m=foo` inline value form is promoted just like `-m foo`.
- Missing --model after the preprocessor raises typer.Exit(2) cleanly
(replacing typer's pre-PR required-flag enforcement now that --model
is Optional to allow the legacy promotion path).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scrub .github/workflows for staging push (matches staging base)
* Fix studio CLI argv handling and pass-through docstring drift
- studio/backend/core/inference/llama_server_args.py: drop the stale
``-np``/``--parallel`` entry from the docstring's pass-through tunable
list. These flags moved into _DENYLIST_GROUPS so the docstring now
contradicts the validator and would mislead future maintainers
debugging the ValueError from validate_extra_args(["--parallel","8"]).
The deleted wording was introduced by dbea77e34 ("Studio: forward
llama-server args from `unsloth studio run`, activate `unsloth run`,
and allow passing model:quant to load models") when --parallel was
still a documented pass-through; the same commit's "quant" reference
is about the model:quant syntax, unrelated to the parallel slot
wording being deleted here.
- unsloth_cli/commands/studio.py: add _expand_attached_np_short next to
_consume_legacy_short_aliases. Both work around Click's short-option
clustering for this command -- the legacy preprocessor for `-m` / `-f`
/ `-hfr` and this one for the attached `-np<N>` form. Click clusters
`-np8` as `-n -p 8` because `-p` is the typer short for `--port`,
silently setting port=8 and dropping the parallel value; rewriting the
attached form into separated `-np <N>` in sys.argv before Click
parses preserves the user's value. Space/equals forms (`-np 8`,
`-np=8`) already work and are left alone.
- unsloth_cli/__init__.py: import _expand_attached_np_short from the
studio command and run it only when argv[0] looks like the unsloth
console-script or workspace cli.py, so importing this module from a
notebook or pytest run does not mutate the caller's argv.
* Tighten the -np canonicaliser comments
Drop the helper's co-location sentence (location is self-evident from
grep) and shorten the entry-gate rationale to one short sentence
covering the why.
* Sync .github/workflows with upstream author branch
* Sync .github/workflows with upstream author branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bump install.sh / install.ps1 pin to unsloth>=2026.5.7 (#5753)
PyPI release unsloth 2026.5.7 is now live. Bumps the pinned floor in
install.sh and install.ps1 from unsloth>=2026.5.6 to unsloth>=2026.5.7
so fresh installs resolve to the new wheel.
Tagged on main as v0.1.416-beta.
* Catch attached `-np<N>` form in backend pass-through validator
The CLI-side `_expand_attached_np_short` rewrites `-np8` to `-np 8`
before Click parses, but HTTP /load `llama_extra_args=["-np8"]` goes
straight to `validate_extra_args` which only matched the exact token.
Reproducer: `validate_extra_args(["-np8"])` previously returned
`["-np8"]` instead of raising; once forwarded to llama-server it
last-win-overrode Studio's slot count while
`app.state.llama_parallel_slots` stayed at the typer value.
Normalise `-np<digits>` to `-np` in `_flag_name` so the denylist
catches the attached form alongside `-np`, `-np=8`, `--parallel`,
`--parallel=8`, and `--n-parallel`. Tests parametrize the new form
including out-of-range values.
* Restore _consume_legacy_short_aliases unit tests + _expand_attached_np_short tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore .github/workflows from origin/main
Earlier merge from claude_review's staging-scrub commits accidentally
deleted production CI workflows. Restore them to main's state.
* Scrub .github/workflows for staging push (matches staging base)
* Sync .github/workflows with upstream author branch
* Round 5+6: broaden -np gate to exact basenames + runtime parallel test
Reviewer-flagged improvements squashed into one commit so the auto-push
review bot doesn't keep stomping the branch:
- unsloth_cli/__init__.py: exact-basename match instead of
endswith('cli.py'). Covers unsloth, unsloth.exe, unsloth-cli,
unsloth-cli.exe, cli.py, unsloth-cli.py. A third-party mycli.py that
happens to import unsloth_cli no longer has its argv mutated.
- unsloth_cli/tests/test_studio_run_parallel_flag.py: parametrised
runtime test (N in {1, 4, 8, 64}) that fakes the in-venv path and
asserts run_server is invoked with llama_parallel_slots=N.
Complements the existing source-text check so refactors that preserve
runtime semantics don't trip a false failure.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 7: respect '--' end-of-options and reject flag-as-value
Round 7 reviewer flagged three legitimate edge cases:
- _expand_attached_np_short rewrote post-'--' tokens. Convention: '--'
ends option processing; payload after it is raw. Stop the loop there.
- _consume_legacy_short_aliases promoted post-'--' legacy aliases for
the same reason. Treat post-'--' tail as raw.
- Legacy '-m -fa' silently consumed '-fa' as the model name, hiding
the real CLI shape error. Reject any next-token that starts with '-'
(except the lone '-' stdin/path sentinel) with a clear BadParameter.
Also expanded the missing-model error string to mention the still-
supported legacy '-m' / '-hfr' aliases so users hitting that diagnostic
on legacy scripts get the right migration hint.
Added four regression tests covering each new behaviour.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 8: soften flag-as-value to long-form only + normalise is_managed_flag
Round 8 reviewer flagged two cleanups:
- _consume_legacy_short_aliases rejected any next token starting with
'-' as a flag, which would break legitimate values like '-foo'
(path or model name with leading dash). Narrow the rejection to
'--long' tokens only; '-x' short forms still pass through.
- is_managed_flag did raw _DENYLIST membership while validate_extra_args
goes through _flag_name first, so '-np8' / '--parallel=8' /
'--port=9000' classified as not-managed by the helper but rejected
by the validator. Route is_managed_flag through _flag_name so the
two helpers agree on every form callers might use.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 9: also catch -np-1 / -np+1 signed attached forms in denylist
Round 9 reviewer noticed _flag_name normalised -np<digits> but missed
signed variants -np-1 and -np+1, so validate_extra_args waved them
through while rejecting --parallel -1. llama.cpp would error out on
negative slot counts anyway, but the validator should classify every
form of the managed flag identically so the boundary is consistent.
* Round 10: signed -np in CLI canonicaliser + reject empty inline aliases
Round 10 reviewer flagged two real issues:
- _expand_attached_np_short rewrote only -np<digits>; signed forms
-np-1 / -np+1 fell through. Backend _flag_name already classifies
them as managed, so the CLI rewriter must too -- otherwise Click
clusters -np-1 into -n -p -1 (port=-1) and never reaches the
backend validator at all.
- -m= / -hfr= / -f= empty inline forms were accepted and produced
--model '' / --frontend '' (then Path('') silently became '.') on
re-exec. Reject empty inline values at the preprocessor with a
clear BadParameter so the malformed input fails fast.
Both behaviours pinned with parametrised regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Expose --parallel on plain `unsloth studio` for API-path parity
The PR added --parallel to `unsloth studio run` but the plain
`unsloth studio` callback (used for API-only / bare-server launches)
still hardcoded llama_parallel_slots to its run_server default. With
--parallel now denied as a llama_extra_args pass-through, that flow
had no first-class way to raise concurrency.
- unsloth_cli/commands/studio.py: add --parallel / --n-parallel typer
Option (default 4, range 1..64) to studio_default, forward through
the venv re-exec, and pass llama_parallel_slots= to run_server in
the in-venv path.
- studio/backend/run.py: argparse --parallel / --n-parallel with the
same range guard so the spawned child accepts the forwarded flag.
- unsloth_cli/tests/test_studio_run_parallel_flag.py: test pins the
new option presence, aliases, default and range guards.
* Round 12: narrow entry-point gate, preserve pre-PR plain-studio default, drop brittle source-text test
Three Opus subagent reviewers (security / backcompat / code-quality)
flagged the same handful of real issues. Consensus fixes:
- unsloth_cli/__init__.py: narrow the -np canonicaliser gate to just
{unsloth, unsloth.exe} (the only pyproject-declared console_script).
The previous cli.py / unsloth-cli.py entries would silently rewrite
sys.argv for any third-party myproj/cli.py that happens to import
unsloth_cli. Dev users running python cli.py ... -np N still work
via the space form, which parses without the rewrite.
- unsloth_cli/commands/studio.py + studio/backend/run.py: restore the
pre-PR llama_parallel_slots default of 1 on plain unsloth studio and
python studio/backend/run.py. unsloth studio run keeps its
hardcoded-pre-PR default of 4. Without this, my earlier API-path
parity commit silently dropped per-call context to ctx/4 for the
plain-studio flow.
- unsloth_cli/tests/test_studio_run_parallel_flag.py: drop the brittle
source-text grep test (test_run_kwargs_use_parallel_value). The
parametrised runtime test test_in_venv_path_passes_parallel_to_run_server
already pins the same intent against actual behaviour.
- unsloth_cli/tests/test_studio_run_short_alias_clashes.py: pin the
narrow entry-point gate with a parametrised negative test covering
seven third-party argv[0] basenames (cli.py, /path/myproj/cli.py,
pytest, unsloth-cli, etc.). Re-broadening the gate now trips a
test instead of silently mutating an unrelated CLI's argv.
* Round 13: shared parallel constants, denylist invariant test, defence-in-depth
Three Opus subagent reviewers (adversarial-user / maintenance /
cross-file consistency) flagged a consistent set of cleanups; folded
into one commit to avoid the pre-commit.ci force-push race.
unsloth_cli/commands/studio.py:
- Extract _PARALLEL_MIN / _PARALLEL_MAX / _PARALLEL_DEFAULT_RUN /
_PARALLEL_DEFAULT_PLAIN module-level constants and use them in both
typer Options (plain studio_default = 1, studio run = 4).
- _expand_attached_np_short now rewrites -np<junk> when the suffix
starts with a digit (or signed digit) so '-np8x' surfaces as a
clean '-np takes an int' typer error instead of a baffling
'--port invalid' complaint after Click clusters '-n -p 8x'.
- Re-exec forwarding emits --load-in-4bit / --no-load-in-4bit
explicitly in both directions; previously the True default relied
on both layers sharing the same default forever.
- run() docstring now explicitly says --parallel / -np pass-through
via llama_extra_args is denied (use the typer flag above).
studio/backend/run.py:
- Mirror the parallel constants and route the argparse default,
range check, and error message through them. Help text mentions
the asymmetry with 'unsloth studio run' so direct-launch dev users
aren't confused by Default 1 in isolation.
studio/backend/core/inference/llama_server_args.py:
- _flag_name strips surrounding whitespace before denylist lookup so
a caller can't slip a managed flag past the boundary with a
trailing space (the trimmed form is what downstream parsers see).
Tests:
- New typer-aliases-subset-of-denylist invariant: every alias the
typer Option claims as --parallel on run() MUST be in the backend
parallel denylist group. Catches the failure mode where someone
adds a new alias and forgets the boundary.
- Extended denylist parametrize to cover ~14 previously untested
aliases (-mu, -dr, -hfv/-hfrv/-hffv family, -mmu, full --ui group,
--models-preset / --models-autoload / --no-models-autoload).
- Whitespace-padded denylist rejection (' --parallel', '-np ', etc).
- --load-in-4bit re-exec test pinning both polarities + default.
- -np<junk> argv rewriter regression tests.
- Cross-reference headers between the two test files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: repair mlx studio base export save_method (#5727)
* Round 14: align backend -np recogniser with CLI rewriter + reject parent --parallel
Round 14 (reviewer.py --parallel 20 with gpt-5.3-codex-spark) flagged
two real P1s and a stale-rebase warning. All three addressed.
- studio/backend/core/inference/llama_server_args.py: widen
_flag_name so -np<digit-prefix> with trailing junk (-np8x,
-np-1foo, -np+1bar, -np9zzz) classifies as managed flag -np,
matching the CLI _expand_attached_np_short rewriter. Without this,
POST /api/inference/load with llama_extra_args=['-np8x'] slipped
past the boundary while the CLI canonicalised the same form. The
two sides now agree on every digit-prefix form.
- unsloth_cli/commands/studio.py: reject --parallel on the
studio group when a subcommand is invoked. Pre-PR the studio
callback had no --parallel; my Round 12 addition made
'unsloth studio --parallel 8 run ...' silently drop the 8
because typer doesn't propagate parent options into subcommand
kwargs. Now errors with exit 2 and a message pointing the
operator at the correct invocation
('unsloth studio run --parallel 8 ...').
- Picked up origin/main via merge (parent commit 0caf0526): the
pre-flight stale-rebase detector found 2 lines on main in
studio/backend/core/export/export.py missing from PR HEAD.
Merged cleanly with no conflicts.
Tests:
- Parametrised denylist coverage for -np<digit-prefix>+junk forms.
- New runtime test confirms exit 2 + helpful error when the group
--parallel is supplied alongside an invoked subcommand.
- Test that the default group --parallel value still lets a
subcommand resolve (no false-positive rejection).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten code comments across --parallel PR
Comment-only pass over the seven PR-touched files; trim verbose
docstrings, collapse multi-line section dividers, and drop
redundant prose that the code already conveys. No behaviour change.
* Studio: trim remaining verbose docstrings missed in last pass
Shorten the test_studio_run_parallel_flag.py module docstring and
the `Re-exec arg-builder coverage` block. No behaviour change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: second comment-tightening pass across PR-touched code
Trim docstrings and inline comments in studio.py, run.py,
llama_server_args.py, and unsloth_cli/__init__.py. No behaviour change;
all 215 tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: deny --embedding / --rerank / --tools pass-through
`--embedding` and `--rerank` flip llama-server into single-endpoint
mode, which breaks Studio's /v1/chat/completions hop. llama-server's
own `--tools` flag silently stacks on top of Studio's tool policy
resolved by `--enable-tools` / `--disable-tools`.
Add all three (plus the `--embeddings` / `--reranking` plural aliases)
to the boundary denylist so HTTP /load and pass-through extras both
reject them cleanly instead of silently desyncing the server surface.
Test added to the existing `test_denylist_rejects_all_aliases`
parametrize. 220 tests pass.
* Studio: make PR-touched tests robust to minimal envs + Windows
Two cross-OS CI findings:
1. `test_typer_parallel_aliases_are_subset_of_backend_denylist` was
doing `from core.inference.llama_server_args import _DENYLIST_GROUPS`
which triggers `core/inference/__init__.py` and pulls in the full
backend chain (fastapi / structlog / loggers / utils.hardware).
The invariant only needs the constants tuple, so load the module
directly via `importlib.util.spec_from_file_location` -- the test
now runs with just typer + pytest installed.
2. `test_legacy_frontend_alias_still_promotes_to_frontend` asserted
the literal string `"/tmp/dist"` after the value round-trips through
`Path()`. On Windows `str(Path("/tmp/dist"))` is `"\tmp\dist"`, so
the assertion tripped on the same logical path. Compare via
`Path(x) == Path("/tmp/dist")` so the test passes on every OS.
Both surfaced by the staging-4 cross-OS CI; no production-code change.
220 tests still pass locally.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: load llama_server_args.py directly in its unit tests
Same fix as the previous CLI-test commit: import the module via
`importlib.util.spec_from_file_location` instead of
`from core.inference.llama_server_args import ...`, so the test no
longer needs the full backend chain (fastapi / structlog / loggers /
utils.hardware) installed via `core/inference/__init__.py`.
The boundary validator is intentionally dependency-free; its unit
tests should reflect that.
* Fix test_main_composer_has_dir_auto anchor after PR #5784
PR #5784 ("Improve image generation UI") rewrote the message-input
textarea's static `aria-label="Message input"` into a JSX conditional
`aria-label={overlay ? "Image edit instructions" : "Message input"}`
but did not update the RTL bidi-attribute regression test, leaving
the literal-string `find('aria-label="Message input"')` anchor with
no match. The `Repo tests (CPU)` job has been red on main since.
Anchor on the inner `"Message input"` string literal instead -- it
survives both spellings and still pins the same textarea element so
the `dir="auto"` assertion has the right block to inspect.
Verified by re-running the exact CI command:
954 passed, 3 skipped, 23 deselected (was 948 passed, 1 failed).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Long Yixing <longyixing331@gmail.com>
PyPI release unsloth 2026.5.8 is now live. Bumps the pinned floor in
install.sh and install.ps1 from unsloth>=2026.5.7 to unsloth>=2026.5.8
so fresh installs resolve to the new wheel.
* Fix unsloth studio update silently downgrading on macOS arm64
Root cause: studio/install_python_stack.py's "Updating base packages"
step passes `--upgrade-package unsloth -r base.txt -c constraints.txt`
with base.txt's `unsloth` and `unsloth-zoo` entries unpinned. On macOS
arm64 the resolver silently backtracks to an older unsloth (2026.5.2 or
even 2025.7.2) whenever a transitive constraint (the most common one is
bitsandbytes wheel availability: 0.49.0+ ships macosx_14_0_arm64 wheels,
older versions do not) makes the unpinned requirement satisfiable by an
older release. install.sh already maintains an explicit `unsloth>=N.N.N`
floor for the same reason, but the floor was missing from the in-venv
update path.
Reproduced on macos-14 across 2026.3.18 / 2026.4.8 / 2026.5.2 / 2026.5.6
starting states. All four ended on unsloth==2026.5.2 after a clean
`unsloth studio update` invocation (2026.5.6 was a true downgrade,
others were stale or partial advances).
Fix mirrors install.sh: query PyPI at runtime for the current latest
version of unsloth and unsloth-zoo, then pass `unsloth>=<latest>` and
`unsloth-zoo>=<latest>` as extra positional pins alongside the existing
`--upgrade-package` flags. Network failures fall back to the historical
unpinned behaviour so offline installs continue to work. Applied to all
three upgrade branches (standard update, local-repo overlay, no-torch).
Also fix the cosmetic `Hardware detected: MLX -- Apple Silicon (i386)`
banner. platform.processor() reads `uname -p` which returns "i386" on
many universal2-shaped Python builds even on a native arm64 interpreter;
platform.machine() is the reliable source ("arm64" once is_apple_silicon
has gated us).
* Dedup floor-pin call sites + LRU cache PyPI lookup
Three upgrade branches each rebuilt the same conditional `unsloth>=` /
`unsloth-zoo>=` arg list with two PyPI round-trips per branch -- six
round-trips per `unsloth studio update` invocation. Extract a
`_pin_floor_args(*, include_unsloth=True)` helper and wrap
`_resolve_latest_pypi_version` in `functools.lru_cache` so the three
branches share a single PyPI request per package.
Functionally equivalent; pure cleanup on top of the previous commit.
* Warn when PyPI is unreachable so the silent fallback is visible
If `_resolve_latest_pypi_version` returns None for either lookup the
floor args are silently dropped, which restores the pre-fix resolver
behaviour. Print a single cyan `warning` line in `_pin_floor_args` when
that happens so users behind a proxy / captive portal / firewalled
PyPI mirror know the upgrade has degraded -- and can supply network
egress or a `--index-url` mirror and retry.
* Soft floor with unpinned-fallback for hosts where floor is unsatisfiable
Reviewer found that the unconditional unsloth-zoo>=LATEST floor turns
a previously-resolvable macOS 13 arm64 update into a hard resolver
failure: unsloth-zoo 2026.5.4 requires mlx-vlm>=0.4.4 -> mlx>=0.30.0,
and mlx 0.30+ only publishes macosx_14_0_arm64 wheels. The pre-fix
behaviour backtracked to an older unsloth instead of erroring. We
should not turn "stale" into "fail".
Add pip_install_with_floor_fallback: first try the install with the
floor appended; if the resolver cannot satisfy it (subprocess exit
code != 0), retry the install without the floor and print a clear
warning. The fall-through preserves the legacy "succeed-but-stale"
contract on hosts where wheel availability is the bottleneck.
Also extend pip_install_try with a req= kwarg so the floor attempt
can pass `-r base.txt` like pip_install does, and add an
UNSLOTH_NO_PYPI_FLOOR=1 opt-out for air-gapped CI / corporate PyPI
mirrors that intentionally do not expose pypi.org directly.
All three upgrade branches (standard, local-repo, no-torch) now go
through the helper so the fallback behaviour is consistent.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add second fallback level: floor without constraints
macOS arm64 floored attempt with -c constraints.txt fails because the
single-env constraint `transformers==4.57.6` conflicts with the new
unsloth-zoo 2026.5.4 -> mlx-vlm 0.4.4+ -> transformers>=5.1.0 chain.
First fallback level retries the floored install without constraints
(transformers freely resolves to a mlx-vlm-compatible version);
downstream pip_install calls still apply constraints.txt to anything
that doesn't transitively conflict.
If THAT still fails (wheel availability rather than constraint
conflict), drop the floor and fall back unpinned as before.
Verified locally with uv pip compile against aarch64-apple-darwin
python-3.13: strict-constrained floor errors, no-constraint floor
resolves cleanly to unsloth==2026.5.7 + unsloth-zoo==2026.5.4 +
transformers==5.5.0 + mlx-vlm==0.5.0.
* setup.sh/.ps1: also gate fast-path on unsloth-zoo being up to date
The version-check fast-path in setup.sh / setup.ps1 only looked at
unsloth itself. If unsloth was at the PyPI latest but unsloth-zoo was
stale, the gate set _SKIP_PYTHON_DEPS=true and install_python_stack.py
never ran -- so the new floor pin from PR #5767 had no effect for the
exact "unsloth at latest, zoo behind" state several reviewers flagged.
Probe both packages' installed-vs-latest versions and only skip the
deps step when BOTH match. When either is behind, fall through to
install_python_stack.py so the new resolver fix gets a chance to run.
Verified setup.sh with `bash -n`; the setup.ps1 change uses PowerShell
if-expressions for the null-default pattern rather than bash-style
${var:-default} which is not valid PowerShell.
* Skip unsloth-zoo floor too for custom no-torch test packages
Reviewer found the asymmetric guard: the no-torch branch was already
gating the unsloth floor on package_name == "unsloth" (test side
packages may not publish to PyPI), but the unsloth-zoo floor was
still added unconditionally. A custom no-torch update that ships its
own forked zoo metadata could now hit a public PyPI floor that does
not match the fork's published version.
Add a symmetric `include_zoo` parameter to `_pin_floor_args` and
gate both pins on the same `package_name == "unsloth"` check.
* Address review feedback: simpler except clause + private-index note
Gemini flagged TimeoutError in the PyPI fetch exception list. OSError already
covers socket timeouts and the 3.11+ TimeoutError subclass on every supported
Python, so drop the redundant entry and explain what each remaining exception
catches.
Codex flagged that floor lookups against pypi.org could break installs behind
a lagging private mirror. Step 3 of pip_install_with_floor_fallback already
recovers transparently in that case; expand the docstring so the behavior is
discoverable without reading the body.
* extras-no-deps: skip transformers==4.57.6 on macOS arm64
Reviewer flagged that the resolver-selected transformers from the
no-constraints base step on macOS arm64 (transformers 5.x for mlx-vlm
0.4.4+) gets silently downgraded back to 4.57.6 by extras-no-deps.txt
during the very next step, breaking mlx-vlm imports at runtime even
though unsloth itself reports as latest.
Add a PEP 508 platform marker so the pin only applies off macOS arm64.
constraints.txt still enforces 4.57.6 everywhere else; mlx-vlm only
publishes wheels for darwin arm64, so other platforms are unaffected.
* setup.sh/.ps1: gate fast-path zoo probe on _PKG_NAME == unsloth
Reviewer found the asymmetric custom-package regression: the new
zoo-aware fast-path probes public unsloth-zoo unconditionally, but a
custom STUDIO_PACKAGE_NAME side build may ship its own zoo fork via
dependency metadata and not install public unsloth-zoo at all. The
previous behaviour (skip Python deps if the custom package itself is at
its declared latest) is preserved by only running the zoo probe when
the managed package literally IS unsloth.
Matches the include_zoo gate already in _pin_floor_args() at
install_python_stack.py.
* install_python_stack: all-or-nothing floor + uv-to-pip retry
Two reviewer findings on the floor-pin helpers:
1. _pin_floor_args() previously kept a half-floor if one PyPI lookup
succeeded and the other failed. With unsloth at latest but the zoo
lookup down, the resolver could still backtrack zoo while we
required unsloth at latest, defeating the pin. Return [] on any
lookup failure so the unpinned legacy path runs cleanly.
2. pip_install_try() ran ONLY uv when USE_UV was true; a uv-specific
failure short-circuited to False even when pip itself could have
applied the floor. Mirror pip_install()'s uv-to-pip fallback: try
uv, fall through to pip on non-zero exit, and only then give up.
* extras-no-deps: rewrite marker without `not` for PEP 508 parsers
pip's vendored packaging rejects `not (...)` in PEP 508 markers; the
grammar only specifies `and` / `or` between boolean atoms. The staging
macos-14 matrix failed every job at "Installing extras (no-deps)" with
`Expected a marker variable or quoted string`. Apply De Morgan's law
so the marker uses `or` between two `!=` checks, which both pip and
uv parse cleanly. Behaviour identical: skip the 4.57.6 pin only on
darwin arm64; pin everywhere else.
* constraints: skip transformers==4.57.6 pin on macOS arm64 too
Marker-gating the extras-no-deps.txt pin was not sufficient. Every
subsequent pip_install in the update pipeline passes
-c single-env/constraints.txt, and constraints.txt itself pinned
transformers==4.57.6 unconditionally. The latest staging-2 run shows
the base step's no-constraints fallback installed transformers 5.5.0
correctly, but a later constrained step (extras / studio / data-designer
deps) silently downgraded it back to 4.57.6, leaving mlx-vlm 0.5.0
in the venv with an unsatisfied transformers>=5.5.0 requirement.
Apply the same `sys_platform != "darwin" or platform_machine != "arm64"`
marker to the constraints.txt entry so it is inert on darwin arm64.
Other platforms still pin 4.57.6 because mlx-vlm only publishes wheels
for darwin arm64; no other platform is affected.
* constraints: carve out darwin arm64 from every == pin
Marker-gating only transformers was not enough; staging-2 still failed
with the same `transformers==4.57.6 in venv after the update` outcome
because the resolver hit a `huggingface-hub==0.36.2` (and adjacent)
conflict with mlx-vlm's `huggingface-hub>=1.5.0` requirement, then
fell back to a stale stack even after my no-constraints level fired
on the base step.
Apply the same `sys_platform != "darwin" or platform_machine != "arm64"`
marker to every == pin in constraints.txt. Range pins (mcp, fastmcp,
websockets) stay active everywhere because they do not conflict with
the mlx-vlm chain. mlx-vlm only publishes wheels for darwin arm64, so
no other platform is affected.
* install_python_stack: also --upgrade-package transformers and mlx-vlm
Staging-2 showed that even after the constraints.txt carve-out for
darwin arm64, the venv still ended up with the OLD `transformers==4.57.6`
paired with a NEW `mlx-vlm==0.5.0` from unsloth-zoo's transitive
upgrade. The resolver's --upgrade-package flag only freshens the named
packages and their newly-pulled transitive deps; transformers was
already installed at a version that satisfied unsloth-zoo's range
(`>=4.51.3,<=5.5.0` with exclusions), so the resolver did not upgrade
it -- even though mlx-vlm 0.5.0 requires `transformers>=5.5.0`.
Add `--upgrade-package transformers` and `--upgrade-package mlx-vlm`
to all three base-step branches. Both are no-ops when the package is
absent (mlx-vlm only ships wheels on darwin arm64); on darwin arm64
this is what nudges the resolver to upgrade both together so the
final venv is internally consistent. On Linux/Windows, transformers
stays at 4.57.6 because constraints.txt still pins it there and
mlx-vlm never enters the resolution.
* install_python_stack: explicit mlx-vlm + transformers realign on macOS arm64
Even with --upgrade-package hints, uv leaves the venv with the
already-installed transformers (4.57.6 inherited from the OLD venv's
constrained install) when that version still happens to satisfy
unsloth's own metadata range -- but it does not also re-resolve
mlx-vlm's stricter `transformers>=5.5.0` requirement, so the venv
ends up with mlx-vlm 0.5.0 paired with transformers 4.57.6 and
mlx-vlm imports break at runtime.
After the base step, on darwin arm64 only, run an explicit
`pip install --upgrade mlx-vlm transformers` with constrain=False.
This forces both packages through the resolver again as direct
top-level requirements, so transformers is pulled up to whatever
mlx-vlm's metadata requires (5.5.0 today). No effect on any other
platform because mlx-vlm has no wheels off darwin arm64 and the
branch is gated on IS_MAC_ARM.
* requirements: marker-gate every == pin that conflicts with mlx-vlm chain
Staging-2 kept ending up with transformers==4.57.6 even after the
realign step, because studio.txt unconditionally pins
huggingface-hub==0.36.2 (and datasets==4.3.0). Installing studio.txt
with constraints active pulls the resolver back to a huggingface-hub
that only recent transformers (4.x) supports, which silently downgrades
the realigned 5.5.0 to 4.57.6 -- exactly the inconsistency we tried to
prevent.
Also extras-no-deps.txt still pinned trl==0.23.1 unconditionally; the
0.23.1 wheel transitively requires huggingface-hub<1, same coupling.
Marker-gate all three. The carve-out is identical to constraints.txt's:
inactive on darwin arm64 (where the mlx-vlm chain dictates newer
versions), active everywhere else (where Linux/Windows users rely on
the single-env pins). mlx-vlm only publishes wheels for darwin arm64
so no other platform is affected.
* realign: --force-reinstall mlx-vlm + transformers + huggingface_hub
Plain --upgrade does not force uv to re-resolve mlx-vlm's transformers
requirement when the already-installed transformers happens to satisfy
unsloth's own range. Switch to --force-reinstall on the three packages
so the resolver tears them down and brings them back together with
consistent versions. Include huggingface_hub because transformers 5.x
requires hf-hub>=1.5.0 and the resolver would not touch it otherwise.
* realign: pin transformers via mlx-vlm's own metadata spec
`pip install --force-reinstall mlx-vlm transformers` still resolved to
an already-installed transformers 4.57.6 because uv treats it as
satisfying unsloth's transformers range without re-checking mlx-vlm's
stricter requirement. Pull mlx-vlm's actual transformers specifier
from its installed metadata at runtime and pass it as an explicit
version requirement (e.g. `transformers>=5.5.0` for mlx-vlm 0.5.0).
That removes the resolver's wiggle room: it MUST pick a transformers
satisfying mlx-vlm AND unsloth, which on darwin arm64 with the latest
unsloth-zoo means transformers==5.5.0. Falls back to unpinned
`transformers` if metadata read fails, so this never errors.
* realign: uninstall-then-install to bypass uv's incumbent bias
Every flag-based approach failed: --upgrade, --upgrade-package,
--force-reinstall, and even an explicit `transformers>=5.5.0`
requirement all left the venv with transformers==4.57.6 because uv
treats the already-installed version as satisfying unsloth-zoo's
range and refuses to disturb it, even when it does not satisfy
mlx-vlm's stricter requirement.
Replace the realign step with an explicit uninstall of the conflicting
trio (transformers / mlx-vlm / huggingface_hub) followed by a fresh
install. With no transformers in the venv, the resolver MUST pick a
version satisfying every installed package's metadata, which on
darwin arm64 with the latest unsloth-zoo is uniquely 5.5.0.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose comments across PR #5767 changes
* Simplify mac-arm64 fix: install MLX stack with --no-deps
The previous approach (PyPI floor pin + 3-level fallback + macOS arm64
realign step + marker carve-outs on every == pin) was fighting symptoms.
The root cause is that unsloth-zoo declares mlx-vlm>=0.4.4 as a darwin
arm64 dep, and mlx-vlm 0.5.0's metadata pulls in transformers>=5.5.0,
which conflicts with the main venv's transformers==4.57.6 pin and forces
the resolver to backtrack unsloth.
Severing that chain at its source: install mlx + mlx-metal + mlx-lm +
mlx-vlm with --no-deps BEFORE unsloth-zoo. The resolver sees mlx-vlm
already installed (>=0.4.4) and never inspects its transformers metadata.
Per-model transformers version routing is already handled at runtime by
the side-car venvs in utils/transformers_version.py (.venv_t5_530 for
Ministral/GLM/Qwen3 MoE, .venv_t5_550 for Gemma 4).
Net change: -224 / +71 lines across install.sh, install_python_stack.py
and the three requirements files.
Reverted:
- _resolve_latest_pypi_version + _pin_floor_args + pip_install_with_floor_fallback
- macOS arm64 realign step (pip uninstall + reinstall)
- --upgrade-package transformers --upgrade-package mlx-vlm in base steps
- All ; sys_platform != "darwin" or platform_machine != "arm64" markers
in constraints.txt, studio.txt, extras-no-deps.txt
- pip_install_try restored to its pre-PR signature
Added:
- install.sh: Apple Silicon MLX --no-deps install before unsloth (both
fresh and migrated branches)
- install_python_stack.py: same step gated on IS_MAC_ARM and not skip_base
Kept (independent bugs):
- setup.sh / setup.ps1 dual-package zoo version check
- platform.processor() -> platform.machine() hardware-detect fix
* Minimise PR to mac-arm64-specific changes only
Revert setup.sh and setup.ps1 to main -- the dual-package zoo check was
defensive and not strictly needed once mlx-vlm is installed --no-deps
(the resolver-backtrack scenario that produced stale zoo no longer happens).
Tighten remaining comments in install.sh and install_python_stack.py.
Final PR-attributable changes:
install.sh +24/-5 (MLX --no-deps in 2 places)
studio/install_python_stack.py +19 (MLX --no-deps + IS_MAC_ARM)
studio/backend/utils/hardware/hardware.py +6/-6 (processor() -> machine())
studio/backend/requirements/*.txt unchanged
* Revert "Minimise PR to mac-arm64-specific changes only"
This reverts commit 9470daa855.
* Revert "Simplify mac-arm64 fix: install MLX stack with --no-deps"
This reverts commit f8a43b87e8.
* Revert "Trim verbose comments across PR #5767 changes"
This reverts commit c3f293a10f.
* Simplify mac-arm64 fix: --no-deps MLX + METADATA patch
Root cause: unsloth-zoo declares mlx-vlm>=0.4.4 as a darwin-arm64 dep, and
mlx-vlm 0.5.0's published metadata declares transformers>=5.5.0. Every
subsequent resolver run with constraints.txt's transformers==4.57.6 sees
the conflict and backtracks unsloth to escape it (user-reported downgrade).
The aggressive pin doesn't reflect what mlx-vlm actually requires at
top-level import time -- the symbols it loads (AutoProcessor, AutoTokenizer,
ProcessorMixin, BatchFeature) are stable across transformers 4.51+. Model-
specific submodules that genuinely need 5.x APIs are only loaded once the
3-tier transformers dispatcher (utils/transformers_version.py) has activated
the matching .venv_t5_530 / .venv_t5_550 side-car at runtime.
Fix: on Apple Silicon, install the MLX stack with --no-deps then rewrite
mlx-vlm/mlx-lm's installed METADATA to declare transformers>=4.51.3. Now
the resolver sees mlx-vlm 0.5.0 as compatible with the main venv's
transformers==4.57.6 and there's nothing to backtrack.
Reverts the previous heavy machinery:
- _resolve_latest_pypi_version, _pin_floor_args, pip_install_with_floor_fallback
- macOS arm64 realign step (pip uninstall + reinstall)
- --upgrade-package transformers --upgrade-package mlx-vlm in base steps
- All ; sys_platform != "darwin" or platform_machine != "arm64" markers
in constraints.txt / studio.txt / extras-no-deps.txt
- setup.sh / setup.ps1 dual-package zoo check (Windows never had the bug;
with this fix in place stale zoo no longer happens on macOS either)
- pip_install_try restored to pre-PR signature
Kept:
- install.sh: MLX --no-deps install in fresh + migrated branches
- install_python_stack.py: same step gated on IS_MAC_ARM and not skip_base
- _relax_mlx_metadata() helper, called immediately after each MLX install
- studio/backend/utils/hardware/hardware.py: platform.processor() ->
platform.machine() cosmetic fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use UV_OVERRIDE to relax mlx-vlm transformers pin
uv supports --overrides / UV_OVERRIDE which globally overrides any package's
stated dependency requirement. mlx-vlm 0.5.0 declares transformers>=5.5.0
and mlx-lm 0.31.3 declares transformers>=5.0.0; neither is true at top-level
import time (their imports use AutoProcessor / AutoTokenizer / ProcessorMixin /
BatchFeature which are stable across transformers 4.51+). Per-model 5.x
routing is handled at runtime via the .venv_t5_530 / .venv_t5_550 side-cars.
Override file (overrides-darwin-arm64.txt) declares transformers>=4.51.3 ;
exported via UV_OVERRIDE env var on Apple Silicon by both install.sh and
install_python_stack.py. uv then resolves mlx-vlm as compatible with the main
venv's transformers==4.57.6 (constraints.txt) and unsloth advances cleanly to
LATEST.
Drops, vs. the previous attempts:
- _resolve_latest_pypi_version + _pin_floor_args + pip_install_with_floor_fallback
(floor-pin machinery -- replaced by single UV_OVERRIDE line)
- macOS arm64 realign step (pip uninstall + reinstall)
- --upgrade-package transformers --upgrade-package mlx-vlm in base steps
- All ; sys_platform != "darwin" or platform_machine != "arm64" markers
- _relax_mlx_metadata() helper + sed METADATA patch (uv reads from index, not
dist-info, so dist-info patches were ineffective)
Kept:
- install.sh / install_python_stack.py: MLX latest install on Apple Silicon
(now without --no-deps, the override lets the resolver pick a consistent set)
- studio/backend/utils/hardware/hardware.py: platform.machine() cosmetic fix
* Trim UV_OVERRIDE comments; bump override floor to 4.57.6
Match the main venv's constraints.txt pin exactly so the override file
reads as the actual installed version rather than mlx-vlm's API floor.
Comments collapsed to one-liners where possible.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: auto-recover when shadowed 'unsloth' on PATH hides the frontend dist
The CLI launcher derives `_PACKAGE_ROOT` from where `unsloth_cli` imports
from, and `studio/backend/run.py` derives its default `frontend_path` from
`Path(__file__).resolve().parent.parent / "frontend" / "dist"`. When
another `unsloth` (a separate venv with `pip install unsloth`, a system
install, an older venv earlier on PATH) wins `which unsloth`, both
resolve into a site-packages tree that ships frontend source files but no
vite-built `dist/`. The backend warned `[WARNING] Frontend not found at
...` and then happily served 200 on every `/api/*` route while returning
`{"detail":"Not Found"}` on `/`. The 404 was silent to users -- the
process was healthy, the log line scrolled by, and the only symptom was a
blank browser tab.
This is a real situation: many devboxes carry a workspace venv with
`unsloth` installed years before the user runs `curl|sh` to install
Studio. The installer-managed binary at `~/.local/bin/unsloth` exists
but loses to the older venv on PATH order.
Three layers of fix, additive:
Layer C -- runtime auto-discovery (unsloth_cli + run.py)
The CLI now resolves `--frontend` explicitly before spawning `run.py`,
probing in order: package-local default, installer venv site-packages
(`$STUDIO_HOME/unsloth_studio/lib/python*/site-packages/...` and the
Windows `Lib/site-packages/...` equivalent), and editable-install source
roots read from `__editable___*_finder.py` MAPPING dicts in the installer
venv. `run.py` does the same probe as a backstop for direct `python
run.py` invocations.
Layer E -- loud structured error
The silent `[WARNING]` is replaced with a `SystemExit` that names every
candidate path tried and lists the four one-line fixes (run the absolute
path, pass `--frontend`, pass `--api-only`, reinstall). Suppressed only
in `--api-only` mode where no UI is served by design.
Layer F -- installer self-check (install.sh + install.ps1)
At the tail of install, both installers compare `command -v unsloth`
(POSIX) / `Get-Command unsloth` (PowerShell) against the just-installed
binary. If a different path wins, a yellow `warning` block names the
shadowing binary and prints the alias / absolute-path / PATH-reorder
fixes. install.sh uses the venv Python for path canonicalization so it
also works on macOS (BSD `readlink` has no `-f`).
Cross-platform notes:
- Glob patterns probe both `lib/python*/site-packages` (POSIX) and
`Lib/site-packages` (Windows).
- Canonical-binary path branches on `sys.platform == "win32"` to pick
`unsloth.exe` over `unsloth`.
- install.sh fixed for macOS; install.ps1 is the Windows analog.
Tests: `studio/backend/tests/test_frontend_resolution.py` covers five
cases via AST-load of the helpers (no uvicorn / FastAPI import needed,
matching `test_host_defaults.py`'s style):
1. Resolver returns None when nothing exists anywhere.
2. Resolver picks the first existing candidate when the default works.
3. Fallback to `$UNSLOTH_STUDIO_HOME` site-packages dist when the default
is missing.
4. Fallback to an editable-install source root via MAPPING parsing.
5. Resolver tolerates a non-existent `$UNSLOTH_STUDIO_HOME`.
All 5 new + 2 existing host-default tests pass.
* Studio: address review feedback on PR 5782 (Windows hardlink, Win path hint, broader tests)
Four parallel platform reviews (Windows, Linux, macOS, general) on the
initial commit surfaced a small batch of correctness items, all addressed
here:
Windows install.ps1 (medium severity, false positive on every install):
The user-facing shim at $StudioHome\bin\unsloth.exe is a hardlink to
$VenvDir\Scripts\unsloth.exe (created at line 1582). Resolve-Path does not
de-duplicate hardlinks, so the previous string compare always saw the two
paths as different and the new "another 'unsloth' wins on PATH" warning
would fire on every fresh Windows install. Switched to content-hash
equality via Get-FileHash, which collapses hardlinks, symlinks, and
identical copies to a single identity. Also restricted the probe to
Get-Command -CommandType Application so PowerShell aliases / functions /
scripts named "unsloth" don't false-trigger.
Windows run.py SystemExit hint (medium severity, defeats the recovery UX):
The structured error printed Path(STUDIO_HOME)/"unsloth_studio"/"bin"/
"unsloth.exe" on every platform, but on Windows the installer places the
shim at $STUDIO_HOME/bin/unsloth.exe (no unsloth_studio segment) and the
venv binary at $STUDIO_HOME/unsloth_studio/Scripts/unsloth.exe. The hint
pointed at a non-existent path on Windows. Branch on sys.platform ==
"win32" to emit the real shim location; Linux / macOS keep the unsloth_
studio/bin/unsloth layout.
MAPPING regex robustness (low):
[^\n]* silently failed if a future setuptools / black reformat wrapped
the MAPPING dict across multiple lines. Tightened to [^}]* + re.DOTALL,
which still rejects nested dicts (setuptools never emits those for
editable installs) but tolerates either single- or multi-line literals.
install.sh broken-venv edge case (low, macOS reviewer):
Previously _canon fell back to echoing the raw input when the venv python
failed, which would make two symlinked-but-identical paths look different
and false-trigger the warning. Now _canon returns empty on failure and
the caller skips the whole comparison if either side is unresolvable.
argparse default + log readability (nits):
run.py's argparse --frontend default now reuses the module-level
_DEFAULT_FRONTEND_PATH constant so it stays in lockstep with run_server's
default. The [OK] log message resolves the chosen path so support output
is always absolute.
Tests grow from 5 to 8 in studio/backend/tests/test_frontend_resolution.
py (10/10 with the existing host-default tests):
- Windows-layout fallback: Lib/site-packages with capital L.
- Multi-line MAPPING dict: locks in the [^}]* + re.DOTALL behaviour.
- SystemExit message contract: every actionable fix string and the
attempted-paths list must appear; pins the user-facing recovery
message so a future refactor doesn't drop a bullet.
End-to-end re-verified on this box: shadowing workspace_22/bin/unsloth
still serves 200 on / through the editable-finder fallback, with the
follow-up resolve-then-log change yielding [OK] Frontend loaded from
/mnt/disks/unslothai/ubuntu/unsloth/studio/frontend/dist.
Out of scope (called out by reviewers but deferred):
- _resolve_frontend_path candidate ordering still tries _PACKAGE_ROOT
first. For the rare case where a shadowing install carries an older
built dist, this serves the stale UI instead of the fresh one. Fix is
non-trivial (the --local workflow intentionally wants _PACKAGE_ROOT to
win when the cloned repo is the source of truth), so leaving it for a
follow-up.
- studio/backend/colab.py still bails out on missing frontend instead of
routing through the new resolver. Pre-existing behaviour, separate PR.
- _resolve_frontend_path is duplicated across run.py and unsloth_cli/
commands/studio.py. Minor maintenance concern; consolidation is
natural in a later refactor.
* Studio: guard ast.literal_eval result with isinstance(dict)
Addresses gemini-code-assist[bot] high-priority inline review on PR 5782
flagging that `mapping.get('studio')` could raise AttributeError if the
MAPPING regex matched a brace-delimited literal that ast.literal_eval
parsed as a non-dict (set, list, None). The regex `\{[^}]*\}` happily
matches `{1, 2, 3}` and literal_eval returns a set; the previous code
then crashed on .get().
Setuptools's editable-install template only emits dict literals so this
is defensive rather than a live bug, but the guard is one line per call
site and prevents a future template change from taking out backend
startup or CLI invocation.
Both call sites (studio/backend/run.py:558 and
unsloth_cli/commands/studio.py:234) now bail out on the finder file when
isinstance(mapping, dict) is False; the resolver keeps probing the
remaining finders, so a malformed entry in one finder cannot poison the
discovery of a good one elsewhere.
Adds test_resolver_does_not_crash_on_non_dict_mapping_literal to
test_frontend_resolution.py, which writes one bad finder (MAPPING is a
set literal) alongside one good finder (MAPPING is a real dict) and
asserts the resolver returns the good finder's dist path. Without the
guard this test crashes with AttributeError; with the guard it passes.
11/11 tests green.
* Studio: per-card web_search result + shell_call output fallback (OpenAI)
Two empty-output bugs in the OpenAI Responses tool-result rendering that
showed up clearly when a single prompt invoked 9 web_search + 4
code_execution + 1 image_generation in one turn. Reproduction shape in
the SQLite-stored chat history:
- 8 of 9 web_search tool-call records had result == "" (the cards
rendered as empty cards in the thread)
- 4 of 4 code_execution (shell_call) records were missing the result
key entirely (NoneType), so the cards that showed "Ran cat ..." style
commands displayed the command line but no output panel at all
- image_generation worked, as did the very last web_search of the run
Root causes in studio/backend/core/inference/external_provider.py:
1. web_search_call's tool_end emitted result: "" by design, with the
intent of overwriting only the LAST call at response.completed with
the full citation list (the source-pill extractor on the frontend
flatMaps across every web_search result, so a single non-empty
result is enough for the trailing source pills). Side effect: every
intermediate card renders empty in the thread. Fix: seed each call's
own tool_end result with "Searching: <query>" so the per-card text
is never empty, then keep the last-call overwrite path so the
source-pill extractor still works. Falls back to empty when the
model emits an action with no query, so the existing last-call path
stays unchanged for that edge.
2. shell_call's tool_start was emitted from
response.output_item.done for the call item, but tool_end lived in
the separate response.output_item.done handler for shell_call_output.
When OpenAI's Responses stream bundles the output array onto the
shell_call item's own done event (no separate shell_call_output
item), the previous handler emitted tool_start with no following
tool_end. The card spun on "running" indefinitely and stored as
NoneType in the thread DB. Fix: when the shell_call's done event
carries an embedded output list, emit tool_end immediately from
that. Track tool_end_emitted on the shell_calls map so a subsequent
shell_call_output event (some streams ship both) is skipped instead
of double-completing the card. A final flush at response.completed
emits tool_end for any orphan shell_call that received neither
bundled output nor a separate output event, so cards always finalise.
Tests (studio/backend/tests/test_openai_tool_result_fallbacks.py, 6
new):
- web_search: three calls, each card's result is its own Searching:
query (no empties)
- web_search: last call still gets the aggregated citation block when
url_citations arrive (pins the overwrite path)
- web_search: empty action.query falls back to result == "" (no junk
Searching: placeholder)
- shell_call: bundled output on done emits a single tool_end with that
output as the result text
- shell_call: bundled-then-separate output does not double-emit
tool_end (subsequent shell_call_output is skipped)
- shell_call: orphan call with neither bundled nor separate output is
flushed at response.completed so the card finalises
15/15 tests green when combined with the existing 9 in
test_openai_code_execution.py. Pre-commit + ruff format clean.
Scope: OpenAI Responses-API code path only. The Anthropic native
Messages-API path (_stream_anthropic) is untouched, as is the local
llama-server path. Local-model behaviour cannot regress because the
edited handlers only fire inside the OpenAI cloud branch.
* Studio: per-model external max_tokens cap + clamp on model switch
Two related external-provider issues that surfaced from the same
investigation as the per-card web_search / shell_call result bugs in
the previous commit:
A. Slider cap was a one-size-fits-all 32768 for every external model.
provider-capabilities.ts kept a single EXTERNAL_MAX_OUTPUT_TOKENS
constant (32k), well below what most providers actually accept. The
docstring even called out the right per-provider numbers (Anthropic
Opus 128k, GPT-5.x ~128k, Gemini 2.5 ~65k, DeepSeek 8k) but the
code picked the lowest as a conservative floor. Effect: long
generations from gpt-5.5 / claude-opus-4-7 silently truncated at
32k even though the API would have served up to 128k.
Fix: introduce getExternalMaxOutputTokens(providerType, modelId)
returning the documented per-model cap. Patterns are checked
longest-first so e.g. gpt-5.5-pro matches before gpt-5.5. Unknown
provider/model combinations fall back to the existing 32k floor so
no surprise increases for ids we don't know about.
Per-model caps from the official docs:
- OpenAI gpt-5.5 / gpt-5.5-pro: 128000
- OpenAI gpt-5.4 / gpt-5.4-pro: 65536
- OpenAI gpt-5.3: 16384
- Anthropic claude-opus-4-7: 128000
- Anthropic claude-opus-4-6 / sonnet-4-6 / opus-4-5 / sonnet-4-5 /
haiku-4-5: 64000
- Gemini 3.x family: 65535
- DeepSeek: 8192
- OpenRouter: strip provider/ prefix from the id and re-resolve
The slider in chat-settings-sheet.tsx and the send-time clamp in
chat-adapter.ts both call the new function so the slider's max=
matches what the wire layer will accept.
B. Slider value lied after switching from a local model to external.
When Studio auto-loads the helper Gemma-4-E2B-it on first chat,
chat-adapter sets params.maxTokens to Gemma's context_length
(262144 for Gemma 4). Switching the model picker to gpt-5.5 then
flips the slider's max prop to the external cap, but the stored
params.maxTokens is never reset. The numeric value next to the
slider would render 262144 against a track that ended at the
external cap. The send-time clamp brought the outbound max_tokens
back down to the cap, so the API call was safe, but the displayed
number had no relationship to what was actually being sent.
Fix: chat-runtime-store.setCheckpoint now clamps params.maxTokens
to getExternalMaxOutputTokens(...) on transitions into an external
model. Looks up the provider via useExternalProvidersStore so we
can derive providerType from the parsed external model id. No-op
when the stored maxTokens is already at or below the new cap, so
user-tuned values within range survive the switch.
Scope: pure frontend changes scoped to external-provider code paths.
Local model behaviour is untouched -- the ggufContextLength branch of
the slider's max= is unchanged, and setCheckpoint only mutates
maxTokens when isExternalModelId(modelId) is true. The send-time
clamp continues to be the safety net for any in-flight request that
crosses a model switch before the store-level clamp has applied.
Typecheck (tsc -b) clean; bun run build succeeds (2.13s).
Co-changes with the previous commit (7fe1adbf, per-card web_search +
shell_call output fallback) form a single PR: every empty-output and
silent-truncation issue surfaced from the same animal-popularity
prompt reproduction is now addressed in one branch.
* Studio: correct external max_tokens caps for Gemini and DeepSeek
Per-doc corrections to the per-model cap table added in 95da8d52:
- Gemini 3.x family: 65535 -> 65536, per
https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview
(the published max_output_tokens is exactly 64K = 65536). The earlier
65535 was an off-by-one rough cap.
- DeepSeek (deepseek-chat / deepseek-reasoner aliases): 8192 -> 384000,
per https://api-docs.deepseek.com/quick_start/pricing. DeepSeek V4
Flash / Pro both list MAX OUTPUT = 384K; the chat / reasoner ids are
deprecated aliases for V4 Flash non-thinking / thinking modes. The
8192 value was carried over from V3 and silently truncated V4 traffic
at 2% of its actual ceiling.
Affects only the slider max and the send-time clamp for these provider
types. Other providers' caps unchanged. tsc -b clean.
* Studio: also flush orphan shell_calls on response.incomplete
Addresses gemini-code-assist[bot] high-priority inline review on PR
5785: the orphan-shell_call final flush added in 7fe1adbf landed only
in the response.completed branch. Truncated OpenAI Responses streams
emit response.incomplete instead (for example when the request hits
max_output_tokens), which left in-flight shell_call cards spinning
indefinitely in the UI.
Mirror the same flush block in the response.incomplete handler so the
truncated-stream path finalizes every pending tool card. The
tool_end_emitted guard keeps the path idempotent: if a shell_call
already completed via bundled output on its done event, the incomplete
flush is a no-op for it.
Two new tests in test_openai_tool_result_fallbacks.py:
- test_shell_call_flushed_on_response_incomplete_truncation pins the
bug repro: an in-flight shell_call followed by response.incomplete
must emit tool_end so the card finalizes.
- test_shell_call_incomplete_does_not_double_emit pins idempotency:
a shell_call that completed via bundled output and is then followed
by response.incomplete emits exactly one tool_end with the bundled
result text.
17/17 tests green (8 fallback tests + 9 existing code-execution). Pre-
commit + ruff format clean.
* Studio: trim verbose comments across PR 5785 edits
Compress the in-code commentary added across this branch to one or two
lines per block; the verbose prose was easier as a PR description than
as inline noise. No behavioural changes: 17/17 tests still green, tsc -b
still clean.
* feat(recipes): round-trip local model variants
* feat(recipes): add local model selector
* feat(recipes): wire selector into model editors
* fix(recipes): clear stale model state on relink
* feat(recipes): load selected local models for jobs
* chore(frontend): simplify biome scripts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(recipes): handle local selector edge cases
* fix(recipes): polish local model selector behavior
* fix(recipes): delay local model restore until terminal runs
* fix(recipes): accept resolved default gguf variants
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: longest-prefix pricing match + accept chat-style usage keys
Two P1 / High follow-ups from PR 5690 review feedback:
1. Pricing prefix lookup returned the first key it iterated, so
dated snapshots like ``gpt-5.4-mini-2026-04-23`` collided with
the shorter ``gpt-5.4`` entry and overbilled by 3x+. Sort the
table keys longest-first so the most specific entry wins.
2. ``calculate_cost`` only read ``input_tokens`` / ``output_tokens``,
but Studio's OpenAI-Chat-style usage envelope re-emits
``prompt_tokens`` / ``completion_tokens`` (the OpenAI Chat
Completions vocabulary). Callers handing in the chat-style
shape silently got a zeroed bill. Accept either pair so the
calculator works against both raw upstream usage and the
Studio-translated envelope.
Tests (4 new in test_pricing.py): dated mini/pro snapshots inherit
the right rate; chat-style usage keys price correctly; raw key wins
when both shapes are present.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: dedupe cache buckets when costing chat-style Anthropic usage
When the caller hands in Studio's chat-style envelope (``prompt_tokens``
emitted by ``_build_usage_chunk``) for Anthropic, that value already
folds ``cache_creation_input_tokens`` + ``cache_read_input_tokens`` into
the total. The previous follow-up accepted the chat-style key but then
re-added both cache buckets in ``billable_input_tokens`` and ``input_usd``,
double-counting cache tokens on every Anthropic chat-style call.
Detect which envelope landed (``input_tokens`` present = raw upstream;
absent + ``prompt_tokens`` present = Studio chat-style) and peel the
cache buckets off for Anthropic before the downstream math so both
envelopes produce identical costs.
OpenAI: ``input_tokens`` and Studio's ``prompt_tokens`` both already
include ``cache_read`` and exclude any notional ``cache_creation``, so
the OpenAI path stays a straight passthrough.
Tests (2 new): both envelopes match for Anthropic on a triple
(uncached + cache_creation + cache_read); OpenAI envelopes match on a
cached-tokens fixture.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: prefer raw output_tokens over chat-style completion_tokens
Codex flagged that the previous fallback chain
'usage.get("output_tokens") or usage.get("completion_tokens")'
treats an explicit 0 as missing -- a mixed-envelope payload where
'output_tokens' is 0 but 'completion_tokens' is non-zero (or
stale) bills the wrong amount. Mirror the has_input_tokens
precedence pattern: when the raw key is present we use it even at
0; otherwise fall back to completion_tokens.
* Studio: read OpenAI cached tokens from prompt_tokens_details too
Codex flagged that the chat-style OpenAI envelope Studio re-emits
via _build_usage_chunk surfaces cached prompt tokens under
prompt_tokens_details.cached_tokens, not input_tokens_details. The
OpenAI branch only checked input_tokens_details, so a cache-heavy
chat-style turn billed every cached token at the full input rate
instead of the 0.1x cache_read discount.
Walk both keys when discovering the cached count. New regression
test pins that the two envelopes price identically for a turn with
80k of 100k tokens cached.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten pricing prefix match + clamp corrupt usage
Three follow-ups on the longest-prefix pricing match landed in this PR:
- Prefix match now requires a dash boundary or end-of-string. The
longest-key sort alone still falsely landed "claude-opus-4-15" on
the "claude-opus-4-1" row, and "gpt-5.5-prod" on the "gpt-5.5-pro"
row (a 6x overcharge). Demanding the next character be "-" rules
out the lookalikes while keeping dated snapshots
("gpt-5.4-mini-2026-04-23", "claude-opus-4-7-20260414") landing on
their canonical row.
- Clamp every token count to >= 0. A corrupted upstream payload
(negative cached count, off-by-one in a fixture) could previously
produce a negative bill that masked real spend in the session
total tooltip.
- Tolerate a non-dict "cache_creation" (e.g. an upstream proxy
folded the field down to a single int). The current code raised
AttributeError mid-turn; now it falls back to the 5m-default
bucket so the rest of the cost calculation still runs.
Adds tests/test_pricing_edge.py with 20 adversarial cases covering
the boundary check, negative / None / zero token values across both
envelopes, cache_read > prompt corruption, the OpenAI long-context
threshold crossover on cache-inflated billable input, malformed
sub-objects, and unknown-provider degradation. Combined suite is
51 tests, all green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Surface Anthropic cache-read fallback and forward 1h breakdown
Two correctness gaps surfaced on the chat-style usage envelope:
1) Anthropic cache_read fell through to "uncached input" pricing when
the envelope arrived without the native ``cache_read_input_tokens``
key (e.g. via a proxy that only emits the mirrored
``prompt_tokens_details.cached_tokens`` block). Studio's canonical
``_build_usage_chunk`` always sets both so production traffic was
never affected, but the calculator should accept either as a
defense-in-depth measure. Add a fallback to read the mirrored
field when the native one is missing or zero; the native key still
wins when both are present so the math stays deterministic.
2) ``_build_usage_chunk`` dropped the ``cache_creation`` 5m / 1h
breakdown. Downstream ``calculate_cost`` then could not apply the
2x 1h premium and silently fell back to the 5m default,
underbilling 1h cache writes by 2x on chat-style traffic. Forward
the breakdown verbatim when the upstream usage carries it.
Tests grow by 4 (20 -> 24): two for the prompt_tokens_details
fallback (with native-precedence pin), one for the chunk shape, one
for the end-to-end pricing parity check at 1h.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add Anthropic fast_mode pricing multiplier
PR 5715 wires the fast-mode-2026-02-01 beta header + speed:"fast"
field through to Anthropic, but the cost calculator never learnt
about the matching 6x premium documented at
https://platform.claude.com/docs/en/build-with-claude/fast-mode
(Opus 4.7 standard $5/$25 per MTok, fast $30/$150).
This adds:
- ANTHROPIC_FAST_MODE_MULT = 6.0 constant.
- calculate_cost(..., fast_mode=True) applies the 6x to base input
AND output rates before any cache multipliers (cache mults stack
on top of fast per Anthropic docs).
- Provider+model gate: silently no-op on every model that is not
claude-opus-4-6 / claude-opus-4-7 so a stray fast_mode=True on
Sonnet/Haiku can never over-charge.
- model_priced label tagged "(fast)" so the cost tooltip can
surface which rate fired.
- pricing_snapshot now exposes fast_mode_mult so the frontend cost
panel doesn't have to hard-code 6.
7 new edge tests pin the math; existing 55 still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor explicit zero cache_read_input_tokens on Anthropic envelopes
The previous follow-up fell back to ``prompt_tokens_details.cached_tokens``
whenever the native ``cache_read_input_tokens`` was missing OR equal to 0,
even though the commit message stated the native key always wins when
present. A proxy that forwards a stale ``prompt_tokens_details`` block
alongside an authoritative ``cache_read_input_tokens: 0`` would then
inflate cache_read past the real native count, posting a false cache_read
line and bumping billable_input_tokens. Switch the gate to native-key
presence so an explicit zero stays authoritative; the mirror only kicks
in when the native key is absent. Add a regression test pinning the
explicit-zero precedence.
* Move fast_mode pricing back to #5715
The fast_mode 6x multiplier landed in two places at once -- here
(f66df7ba) and on #5715 (4f1afdb5) -- since both audits ran in
parallel. Drop the duplicate from this branch so the change lives
in its natural home (#5715, which introduces fast_mode itself);
this PR stays focused on the cache-read fallback + 1h breakdown.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten pricing comments for PR #5722
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface Anthropic document citations inline + in Sources panel
Anthropic's Messages API streams ``citations_delta`` events on
``content_block_delta`` when the request enables
``citations: {enabled: true}`` on document blocks. Each event carries
one citation pointing at the source document; previously they were
silently dropped, so reader-visible references never reached the chat
UI even when the model was citing properly.
The proxy now:
- dedupes by the type-specific anchor (char_location / page_location /
content_block_location / search_result_location) so re-cites of the
same span collapse onto a single footnote;
- injects ``[N]`` inline right after the matching text run;
- forwards the full list as a synthetic ``document_citations``
tool_event at ``message_stop`` so the Sources panel can render
per-document footnotes next to web_search / web_fetch citations.
Streams that never emit ``citations_delta`` stay byte-identical.
References:
- https://platform.claude.com/docs/en/build-with-claude/citations
- https://platform.claude.com/docs/en/build-with-claude/search-results
Tests (5 in test_anthropic_citations.py): passthrough, single
char_location, dedup of repeat citations, distinct sources get
distinct numbers, search_result_location supported.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Anthropic document_citations in the Sources panel
The PR added a backend _toolEvent.type='document_citations' on
message_stop and an inline [N] marker in the assistant text, but the
chat-adapter only handles container_*/tool_*/sources from
web_search and web_fetch tool calls. Reviewers flagged that the
inline [N] markers had no matching footnote entries in the Sources
panel.
Capture the new event into a documentCitationParts buffer, convert
each citation dict into a Sources-panel source entry (using
document_title or search-result source URL plus cited_text as the
snippet), dedupe by id, and append to the final yield alongside
the existing web_search/web_fetch sourceParts.
* Studio: dedupe search_result_location citations by search_result_index
Anthropic's documented search_result_location citation shape carries
search_result_index, source, title, and start/end_block_index --
NOT document_index/document_title. The previous key keyed on
document_index + document_title + source + start_block_index, so
two distinct search results from the same source collapsed onto the
same footnote and the second [N] marker was lost.
Switch the search_result_location branch to key on the documented
fields, and pin the behaviour with a regression test asserting that
two citations sharing source/title but with different
search_result_index get distinct [1] [2] markers.
* Studio: keep each citation distinct across the end-anchor
Codex follow-ups on the citations PR:
* Backend _anthropic_citation_key now includes the end anchor for
every variant (end_char_index, end_page_number,
end_block_index). Anthropic ranges are start-AND-end pairs, so
a same-start / different-end pair is two distinct citations
that previously collapsed onto one footnote.
* Frontend documentCitationToSource ids include the position
fields (search_result_index, start/end char/page/block) instead
of being keyed on URL alone. Two citations from the same
document or two search_result_locations with the same source
now produce distinct Sources-panel entries, matching the
inline [N] numbering.
* Studio: key Sources list by per-citation id instead of url
Codex flagged that the Sources renderer keys badges on source.url,
so two Anthropic document citations sharing the same source URL
collide as React keys and one badge gets dropped (or duplicated).
The chat-adapter already mints a per-citation id that folds the
position fields (search_result_index, start/end char/page/block)
into the URL, so the two citations have distinct ids even when
their URL matches. Plumb that id through SourceData and use it as
the React key for both the measurement badges and the visible
SourceBadge list. Falls back to the URL when no id is supplied
(web_search and web_fetch source parts).
* Studio: enable Anthropic doc citations on input_document blocks
Plumb citations: {enabled: true} onto the translated Anthropic document
block (both base64 and URL source branches) so the upstream actually
emits citations_delta events. Without this opt-in the inline [N] +
Sources panel plumbing added in this PR is a no-op for real user
PDF / doc uploads.
Refs https://platform.claude.com/docs/en/build-with-claude/citations
Also add edge-case coverage for the citations_delta path:
malformed citations, mixed types per document, reversed indices,
missing document_index, non-int block indices, unknown citation
type, internal _key never leaking, footnote numbering across
content blocks, and the input_document wire-through itself.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reject unsafe citation sources, bound cited_text payload
Three follow-ups on top of #5718 surfaced by a deeper review pass:
1) javascript: / data: / vbscript: in citation source is XSS-able.
``documentCitationToSource`` was assigning ``cit.source`` straight
into ``Source.url`` and rendering it as an <a href>. A hostile
model emitting ``cit.source = "javascript:alert(document.domain)"``
would execute on click (openLink only intercepts URLs that contain
"://" or start with "mailto:", which both miss the javascript:
scheme). Restrict the navigable path to http(s):// only; anything
else falls back to the existing #anthropic-doc anchor and the
source title still renders the raw identifier for context. Also
reject CR/LF inside the URL string.
2) Frontend sources collapse distinct backend footnotes when the
citation type differs but positions match. char_location(0,5) and
page_location(0,5) over the same source previously deduped into
one entry because the id only carried position. Fold citation
type into the id anchor so the 1:1 mapping with inline [N]
markers is preserved across every citation shape.
3) ``cited_text`` was forwarded unbounded inside the synthetic
document_citations tool_event. The Sources panel trims to 240
chars for display anyway; for large RAG / search_result spans
(~10kB cited_text is plausible) this inflates SSE bytes 40x
for no UI benefit. Truncate server-side at 512 chars with an
ellipsis so the description-trim downstream still has room to
work and the wire stays bounded.
Tests grow from 21 to 22; existing 7 + edge 15 still green. Frontend
typecheck clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply http(s) URL guard to all Sources-panel link sources
The previous round only filtered ``cit.source`` inside
``documentCitationToSource``. Two parallel code paths still copied
provider/tool-controlled ``URL:`` text directly into clickable
``<a href>`` Sources-panel links:
* ``parseSourcesFromResult`` in chat-adapter.ts (legacy web_search /
web_fetch tool result parser)
* ``parseSearchResults`` in tool-ui-web-search.tsx (inline tool card)
A hostile tool response like ``URL: javascript:alert(1)`` or
``URL: data:text/html,...`` was therefore still rendered as a
navigable badge in the Sources panel.
Centralise the safe-URL test (``isSafeNavigableSourceUrl``,
``isSafeHttpUrl``) using ``new URL()`` + protocol allowlist + CR/LF
rejection, and apply it to both parsers. Unsafe blocks are dropped
rather than rewritten to a hash anchor because the web_search /
web_fetch parsers have no document-index fallback.
Citation conversion now uses the same helper so the in-place
http(s) regex and CR/LF check stay in one place.
* Shorten citation comments for PR #5718
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface Anthropic web_fetch as a standalone Fetch pill
web_fetch used to be silently bundled with the Search pill on the
assumption that "search returns URLs, fetch reads them" is the
typical workflow. Two problems with that:
- Anthropic bills each web_fetch invocation separately from
web_search hits, so combining them made the per-message cost
surface ambiguous.
- It blocked "just fetch this one URL" workflows where the user
already knows the page they want read and does not want a search
round-trip.
Adds:
- `webFetchToolsEnabled` to the chat-runtime-store, persisted to
localStorage under `unsloth_chat_web_fetch_tools_enabled`, with a
matching `supportsBuiltinWebFetch` capability flag and a
`setWebFetchToolsEnabled` setter.
- A new Fetch pill in the chat composer, rendered next to Images and
only when the active provider returns true from
`providerSupportsBuiltinWebFetch` (Anthropic today). The pill
defaults off so per-fetch billing is always a deliberate opt-in.
- chat-page bootstraps `webFetchToolsEnabled` from the same stored-
preference fallback the other pills use.
- chat-adapter reads `webFetchToolsEnabled` directly when deciding
whether to append "web_fetch" to `enabled_tools`, decoupling it
from `toolsEnabled` (Search).
Backend translation is unchanged: when `enabled_tools` already
contains "web_fetch", `_stream_anthropic` appends the
`web_fetch_20250910` / `web_fetch_20260209` tool exactly as before
(test_anthropic_web_fetch.py pins the standalone-only path at
`test_web_fetch_tool_appended_to_request_body` and the combined
path at `test_web_fetch_combined_with_web_search_and_code_execution`).
Frontend tsc passes.
* ci: re-trigger after transient GitHub API HTTP flake (checkout + ggml-org release fetch)
* Studio: include web_fetch in the disabled-tool guard axis
Reviewer P1 / High on PR #5742 (codex + gemini): after introducing
the standalone Fetch pill, `disabledToolGuard` still only branched on
`webSearchEnabledForThisTurn`. With Fetch ON and Search OFF the
system prompt would tell Claude "you do not have web search or web
fetch tools in this conversation", which contradicts the actual tool
schema being sent and suppresses `web_fetch` tool calls, defeating
the standalone-fetch workflow this PR adds.
Treat search and fetch as a single "any web tool enabled" axis. The
guard only needs to warn the model when no web tool is wired in for
this turn; once either pill is on the model can pick the right one
from the tool schema. The existing `webLabel` already covers both
names, so the user-visible guard text stays accurate in every
combination.
tsc clean.
* ci: re-trigger after transient infra flake on Windows prebuilt / actions/checkout
* Studio: route web_fetch through per-model version dispatch
The web_fetch tool body in `_stream_anthropic` hardcoded
`web_fetch_20250910` instead of calling `_anthropic_web_fetch_version`,
so Opus 4.6 / 4.7 and Sonnet 4.6 missed the `web_fetch_20260209`
dynamic-filtering variant. The picker, the unit tests for it, and a
deliberate "follow-up" note in `test_anthropic_web_fetch.py` already
existed; this just threads it through the emission site.
Mirrors how web_search and code_execution are dispatched per model.
Old models still resolve to `web_fetch_20250910` and continue to work.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten web_fetch comments for PR #5742
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: rewrite OpenAI Responses citation markers to markdown links
OpenAI's /v1/responses stream interleaves text deltas with inline
citation markers built from private-use codepoints (U+E200 / U+E201 /
U+E202) shaped like `citeSOURCE_ID`. The codepoints render
as garbled "E202" glyphs or empty boxes in most fonts, and the
markdown layer further strips them, leaving run-on text like
"citeturn1view0turn1view1turn3view0...". The url list still arrived in
the Sources panel via url_citation annotations, but the inline cite
hand-off into the prose was unreadable.
Rewrite each marker into `[N](URL)` when the matching url_citation
has already been recorded on this stream, and drop the marker
silently otherwise. The lookup uses a new `source_id` field captured
on `_record_url_citation` (accepts source_id / id / locator across
Responses API revisions). Annotations are now applied BEFORE the
delta text is rewritten so that markers and their resolving
annotation arriving in the same SSE event still resolve.
Reference: https://developers.openai.com/api/docs/guides/citation-formatting
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve every source_id alias for a deduplicated url_citation
OpenAI's Responses stream cites the same URL under multiple
source_id markers when the model references different spans of the
same page. The previous dedup-by-URL kept only the first alias and
dropped the rest, so subsequent markers for the same URL never
resolved and got stripped from the prose. Switch the citation
record to a ``source_ids`` list and append new aliases on every
duplicate. The rewriter resolves any alias back to the same
citation number so the inline markers all collapse onto one footnote
rather than fanning out into bogus repeats.
Also collapse the two passes over ``all_url_citations`` in
``_record_url_citation`` into a single loop for clarity. Adds two
regression tests covering the alias-collision and mixed-shape cases.
* ci: re-trigger after flake in Studio GGUF Tool calling (rebased on main #5741 already)
* ci: re-run after transient CodeQL Python checkout auth flake
* Fix split-marker buffer + multi-source ids for PR #5713
The original rewriter only handles markers that arrive whole inside a
single response.output_text.delta event. OpenAI's stream chunks text
on byte-buffer boundaries with no awareness of the marker grammar,
so a marker can straddle two deltas (delta-1 ends with
"citetu", delta-2 starts with "rn0view0"). Each delta
was rewritten in isolation, so the half-marker leaked as garbled
"E200/E202" glyphs in the rendered prose.
Buffer the unterminated tail across deltas and concatenate it onto
the front of the next one so the rewriter sees a complete marker.
Flush the held-over tail on response.completed / response.incomplete /
[DONE], stripping any leftover private-use bytes so a never-closed
marker (truncated stream, missing annotation) never leaks.
Also handle the multi-source marker shape from the OpenAI docs --
citeid1id2 should expand to one bracket
link per resolvable id. The previous regex captured only the first
source id and silently dropped id2/id3.
Reference: https://developers.openai.com/api/docs/guides/citation-formatting
Tests: 21 new cases covering multi-source, locator suffix, marker
split across two and three deltas, unterminated marker on truncation,
late annotation resolving a buffered marker, idempotency, and the
head/tail split helper directly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Defer citation segments until url_citation annotation arrives
The split-marker buffer already concatenates a marker that straddles
two response.output_text.delta events. But when the annotation event
for a url_citation arrives AFTER the delta that contains its inline
marker (the typical OpenAI Responses ordering), the rewriter still
saw an empty lookup table at delta time and silently stripped the
marker. The URL kept showing up in the sources panel but the inline
link reference was permanently gone.
Add _rewrite_citation_markers_partial which leaves an unresolved
marker verbatim and reports has_unresolved=True. The streaming loop
buffers any closed segment that contains an unresolved marker into a
pending_citation_segments FIFO and drains the queue on every later
annotation event, on response.completed, on response.incomplete, and
on the [DONE] sentinel. Drain order is preserved so later clean text
does not leapfrog an earlier deferred segment. End-of-stream forces a
strip so no codepoint leaks if the annotation never arrived.
Add six regression tests covering single-pass resolution, the late-
annotation two-pass case, multi-source markers with partial
resolution, mixed known and pending markers in one segment, and
idempotency on marker-free input.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop unterminated citation tail to prevent cite-prefix plain-text leak
`_flush_pending_marker_tail` stripped the three private-use citation
codepoints from the held-over buffer, but left the literal ``cite``
keyword plus the source id behind as plain text. A stream ending
mid-marker therefore emitted user-visible garbage like
``Some text citeturn0view0`` instead of the intended clean prose.
``pending_marker_tail`` is by construction the suffix that starts at
an unclosed ``\\ue200`` opener -- the split helper guarantees there is
no closing ``\\ue201`` byte. Without that close the marker is
meaningless: the source id cannot be resolved to a URL and the user
prose before the opener was already emitted as ``head`` on the
originating delta. Bail out before the strip step and return the
empty string. As a belt-and-braces measure also drop any orphan
``cite<sid>`` literal at the head of the buffer in case a future
caller passes a partially-terminated tail.
Update the matching ``_simulate_delta_stream`` harness in the edge
tests so it mirrors the new flush logic, and add four regression
tests covering unterminated marker with surrounding prose, marker-
only inputs, prefix-only outputs, and the split-then-close path that
still must resolve to a link.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Defer multi-source markers until all ids resolve for PR #5713
`_rewrite_citation_markers_partial` previously treated a marker as
resolved when even one token in a multi-source marker resolved,
dropping any still-pending source ids. In streamed Responses events
the annotations for a multi-source marker can arrive across separate
`annotation.added` chunks, so the caller no longer buffered that
segment for retry and the late source id was lost from the inline
citation entirely.
Flag the marker unresolved whenever any token misses the lookup so
the streamer keeps the segment pending. End-of-stream force flush
still drops unresolved tokens through `_replace_openai_citation_markers`
so locator-style suffixes (which look like unresolved ids at the token
level but only appear at end-of-stream) render cleanly.
Updated the multi-source test to assert the new pending-then-flush
behavior; locator output now lands at force-flush rather than mid
stream.
* Shorten citation marker comments for PR #5713
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Anthropic fast_mode toggle + surface streaming refusals
Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7
generate output tokens up to 2.5x faster at 6x standard Opus
pricing. The toggle lives in Configuration → Provider when the
selected Anthropic model is Opus 4.6 or 4.7 and is otherwise
hidden. Backend gates the same prefixes a second time so a stale
frontend cannot make Anthropic 400 the request, and the
`fast-mode-2026-02-01` beta header is merged onto whatever other
betas the request already needed (code-execution, compaction).
Streaming refusals (`message_delta.delta.stop_reason="refusal"` on
Claude 4 models) now surface a short user-facing notice in the
assistant message before the translated OpenAI chunk emits the
existing `finish_reason="content_filter"`. Previously the chat
bubble truncated silently because the SSE stopped mid-stream with
no visible explanation. Per the upstream docs the conversation
must be reset before continuing, so the notice tells the user
exactly that.
Reference:
- https://platform.claude.com/docs/en/build-with-claude/fast-mode
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals
Tests:
- studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases
pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet /
Haiku / older Opus / None / False, and the refusal notice + finish
reason on a synthetic refusal stream).
* Studio: drop refused Anthropic turns from the next request
Anthropic's streaming-refusal guidance says the refused assistant
turn must be removed or updated before the next call -- otherwise
the safety classifier keeps refusing. The PR only added a
user-visible notice; the partial assistant output (plus the notice
itself) still rode the next request via toOpenAIMessage.
Tag the refusal turn with an HTML-comment sentinel emitted alongside
the notice. The chat-adapter checks for that sentinel in
toOpenAIMessage and returns null, so the refused turn is excluded
from outboundMessages. The notice still renders in the transcript
(HTML comments don't display), so users keep the explanation.
* Studio: filter None finish_reason entries in test helper
test_refusal_maps_to_content_filter expects only ['content_filter']
in the finish_reasons list, but the post-PR refusal path emits a
user-visible content notice chunk first. Every _content_chunk
carries 'finish_reason: None' by construction; the helper was
appending those, so the assertion saw [None, 'content_filter']
instead of ['content_filter'].
None is not a finish reason -- it's just mid-stream delta noise.
Skip None values in _finish_reasons so the helper reflects what
the test names actually claim to check. Same fix applies cleanly
to the other helper usages (pause_turn test expects [] and the
sibling stop test expects ['stop'], both unaffected).
* Studio: cover Anthropic fast-mode edge cases
Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal.
The base file pins the happy path; this file fills in the cliffs:
* Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and
claude-opus-4-6-2026-02-01 still gate fast_mode through, while
claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not.
* Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT
auto-enable fast_mode -- the prefix tuple must be bumped explicitly
when a new family is whitelisted upstream.
* Beta-header merge: fast_mode coexists with code-execution-2025-08-25
and compact-2026-01-12 in one comma-separated anthropic-beta header
with no duplicates and no truncation. Pins the value to the exact
fast-mode-2026-02-01 docs token so a typo would fail CI.
* Non-destruction: fast_mode=None produces byte-identical outbound
body and headers to the version that omits the argument entirely.
Same for fast_mode=False. Guarantees the upgrade path is
non-breaking on existing Anthropic streams.
* Refusal stream ordering: the user-visible notice precedes the
finish_reason chunk so a streaming UI paints text before flipping
to content_filter. Refusal sentinel emitted exactly once. Notice
rides a normal content delta chunk with finish_reason still null.
Partial assistant deltas survive before the notice.
* Provider-side refusal coverage: a refusal on Sonnet (not just Opus)
still emits the notice + sentinel + content_filter mapping, since
refusal handling is not gated on fast-mode capability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Persist fastMode, drop refused user message on retry
Two follow-ups on #5715:
1) sanitizeInferenceParams stripped fastMode. fastMode is in
PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept
numeric fields plus systemPrompt and trustRemoteCode, so the new
toggle was silently dropped on reload and on the
/api/chat/settings round-trip. Save it the same way trustRemoteCode
is saved.
2) Refusal recovery now also drops the triggering user turn.
Returning null from toOpenAIMessage on the assistant side left the
user prompt that caused the refusal in the outbound history, so
the very next request would re-trigger the same classifier.
Anthropic's refusal-handling guidance is explicit on this: remove
the refused turn AND the user message that triggered it before
the next call. Implemented via a pre-pass that pops the trailing
user message when an assistant carries the refusal sentinel.
Typecheck clean.
* Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes
The text sentinel for the Anthropic refusal drop signal was spoofable:
any assistant message containing the literal
<!--studio:anthropic-refusal--> would prune the prior user + assistant
pair on the next request. Move the signal onto a separate _toolEvent
chunk that the chat adapter latches into
assistant.metadata.custom.anthropicRefusal; assistant text can no
longer control the pruner.
Tighten the fast-mode model gate (backend + frontend) to require a "-"
family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do
not get speed: "fast" on a naive startswith match.
Use survivingMessages for the image / audio attachment scan so a
refused user turn does not gate or mis-attribute the next non-refused
turn.
Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and
apply the documented 6x fast-mode multiplier in the cost calculator
(stacks with prompt-cache multipliers per the docs); expose the new
multiplier on the pricing snapshot for the UI tooltip.
Tests cover the tool-event chunk shape, the prefix-collision rejects,
usage.speed propagation, the 6x pricing math, and that the visible
refusal text carries no embedded sentinel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten fast-mode and refusal comments for PR #5715
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface external-provider cache hits and writes in context bar
The Anthropic / OpenAI Responses streaming paths already emit an
include_usage-style SSE chunk carrying prompt_tokens_details.cached_tokens
and cache_creation_input_tokens / cache_read_input_tokens (see
_build_usage_chunk in external_provider.py), but the chat-adapter only
read the local llama-server timings.cache_n field. As a result, the
context-usage tooltip never showed cache hits or writes for external
providers, even though the backend was computing them.
Read the external usage envelope as a fallback when timings.cache_n is
absent, and surface Anthropic cache_creation_input_tokens as a separate
"Cache writes" line in the tooltip so users can tell a cache miss from a
cache hit on a turn that both reads and writes the cache.
- ServerUsage gains optional prompt_tokens_details.cached_tokens,
cache_creation_input_tokens, cache_read_input_tokens.
- contextUsage store entry gains optional cacheWriteTokens.
- ContextUsageBar gains optional cacheWrites tooltip line.
- chat-page wires both fields through to the bar.
* Studio: render cache stats for external providers too
Reviewer round on the original PR caught three asymmetric-fix sites
where the producer side surfaced external prompt-cache stats but the
consumer side still gated on ggufContextLength (which is only ever set
for the local llama-server runtime). Result: the entire cache-stats
PR shipped invisible for Anthropic / OpenAI Responses / Gemini, which
is exactly the set of providers it was added for.
- chat-page.tsx: drop the ggufContextLength precondition on the
ContextUsageBar mount. The bar already tracks usage; let it decide
what to render based on what it knows.
- context-usage-bar.tsx: make `total` optional. When absent, drop the
"/ total" ratio + percentage progress bar + "approaching limit"
helper, and just show per-turn counters + cache stats. Bootstrap
guard tightened so an all-zero, all-undefined state still renders
nothing.
- runtime-provider.tsx: external-provider rehydration was rejected by
the `store.ggufContextLength` check. Keep the "fits inside window"
sanity check when a local context window IS known, drop it when
it isn't.
- message-timing.tsx: the per-message timing popover used a separate
"Cache hits" code path that only read llama-server's timings.cache_n.
Fall through to custom.contextUsage for external providers, and add
a parallel "Cache writes" line for Anthropic cache_creation events.
* Studio: tighten cache-stats comments
* Scope contextUsage to active checkpoint
Three follow-ups on #5736 so the relaxed external-provider render
gate does not show stale token / cache stats from a different model:
1) setCheckpoint now clears contextUsage on a real checkpoint
change. setActiveThreadId and clearCheckpoint already did this;
the most-traveled transition path (the user switching models from
the picker) leaked the prior turn's counts because they were never
cleared.
2) The external-selection branch in chat-page.tsx now also clears
contextUsage at the same time it nulls ggufContextLength /
activeNativePathToken. Without this an in-session switch from a
local model to an external provider would visibly carry the
previous local turn's counters into the new provider's bar.
3) exitCompare's rehydration is now scoped: restore the saved
usage only when the message's modelId matches the active
checkpoint AND, for local turns where a context window is known,
when the saved total fits inside that window. Without this the
bar could render a stale local-model usage on top of an external
provider, or an oversized usage object that exceeds the now-
active window.
Typecheck clean.
* Plug remaining stale-contextUsage paths
Follow-up to 042e0ac4 that catches four asymmetric-fix sites the
checkpoint-scoping pass missed:
1) setParams now also clears contextUsage on a real checkpoint
change. The local model load path in use-chat-model-runtime calls
setParams(mergeBackendRecommendedInference(...)) which mutates
params.checkpoint before refresh() eventually fires setCheckpoint;
the intermediate window rendered the previous model's counters
under the new checkpoint.
2) chat-adapter.ts setContextUsage on stream completion now gates on
the captured params.checkpoint still being active. A late
completion from provider A used to clobber the context bar after
the user switched to provider B mid-stream.
3) chat-page.tsx exitCompare rehydration no longer accepts a saved
modelId-stamped usage when the active checkpoint is empty. A user
who entered compare, cleared the model, and exited compare would
otherwise see the cleared model's stats reappear.
4) runtime-provider.tsx thread-load no longer restores legacy
unscoped usage (no modelId) unless a local context window is
known. With the relaxed external-provider render gate, old
pre-PR persisted messages without a modelId stamp could attach
their counts to an unrelated active provider.
Also switches message-timing.tsx cache-hit fallback from || to ??
so an explicit cache_n=0 is not replaced by a stale cachedTokens.
Typecheck clean.
* Shorten cache-stats comments for PR #5736
* Studio: stop leaking seeded admin pw to cross-origin callers
The "/" SPA fallback serves index.html with an inline
``window.__UNSLOTH_BOOTSTRAP__`` script containing the seeded admin
password while a password change is pending. Default web mode runs
``CORSMiddleware`` with ``allow_origins=["*"]`` + ``allow_credentials=
True``, which reflects an attacker-controlled ``Origin`` back on every
request and sets Access-Control-Allow-Credentials true. The combination
let any cross-origin page ``fetch('/')`` with credentials and read the
bootstrap admin password out of the HTML body. The API smoke
``CORS: GET / leaks bootstrap pw to cross-origin caller`` audit already
tracked this (tests/studio/studio_api_smoke.py:224) but did not gate CI.
Gate ``_inject_bootstrap`` on a same-origin check: legitimate top-level
navigations omit ``Origin`` on most engines, so the absence of the
header is treated as same-origin; when the header IS present and does
not match ``request.url.scheme://request.url.netloc`` exactly, we now
skip injecting the bootstrap tag. ``Vary: Origin`` is added so an
intermediary cache cannot serve a same-origin response (with bootstrap)
to a later cross-origin caller (and vice versa).
Coverage: ``test_index_bootstrap_origin.py`` exercises the helper with
missing / matching / evil / scheme-mismatch / port-mismatch origins.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments on bootstrap cross-origin helper
* Studio: canonicalise Origin before same-origin gate
A plain string-compare between the Origin header and request.url.netloc
misclassifies legitimate same-origin requests as cross-origin in three
scenarios:
- Browser strips the default port from Origin (https://example.com)
but Starlette's netloc keeps it (example.com:443). Per RFC 6454 the
default port is dropped on the wire, so the strings will not match
even though the requests share an origin.
- Host case differs (Origin: http://Example.com vs netloc:
example.com). Per RFC 3986 host comparison is case-insensitive.
- Scheme case differs (HTTP:// vs http://). Per RFC 3986 the scheme
is also case-insensitive.
These are usability degradations rather than security gaps (legitimate
user denied the bootstrap injection, no attacker gain), but worth
shipping so non-default Studio deployments keep the change-password
auto-fill.
Adds _canonical_origin(scheme, netloc) -> (scheme, host, port) and
compares the canonical tuples. Default-port lookup covers
http/https/ws/wss; userinfo (user:pass@) is stripped per RFC 3986
since Origin never carries credentials. Origin: "null" (sandboxed
iframes, file:// pages) and unparseable values collapse to cross-
origin so the bootstrap pw is never leaked through those paths either.
Tests: 14 cases (was 5). Covers the original same/missing/evil/
scheme/port matrix plus default-port stripping in both directions,
host + scheme case folding, Origin: null, garbage values, and
userinfo-in-netloc.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix IPv6 netloc parsing for PR #5739
The canonical-origin helper used ``netloc.partition(":")`` which
mis-parses bracketed IPv6 hosts (``[::1]:8902`` -> host=``[``,
port-str=``:1]:8902``). The int() then raises and the canonicaliser
returns None, so every IPv6 same-origin request is misclassified as
cross-origin and Studio refuses to inject the bootstrap pw on a
legitimate top-level nav when launched with ``unsloth studio -H ::1``.
Bracket-aware split per RFC 3986 §3.2.2, plus extra regression tests
for IPv6, opaque (data:/blob:/file:), comma-joined multi-Origin and
localhost-vs-127 cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard urlparse ValueError in same-origin gate
urlparse raises ValueError on malformed bracketed Origin values
(unclosed [, invalid IPv6 hex, text after ]) and on a few NFKC
edge cases since Py 3.8. Without a guard, a request carrying
Origin: http://[malformed surfaced as HTTP 500 from the SPA
handler rather than being treated as cross-origin per the
docstring's safer-default rule. Wrap both urlparse calls in
try/except ValueError and return False on parse failure.
Also distinguish a missing Origin header (top-level same-document
GET, treat as same-origin) from an explicit empty string (not a
valid serialised origin per RFC 6454 §6.1, treat as cross-origin).
Four new regression tests pinned down by the PR audit: malformed
IPv6 bracket, invalid IPv6 hex, bracket with trailing garbage,
and the empty Origin header.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten origin-gate comments for PR #5739
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- docker-publish.yml: add `concurrency: docker-publish-${{ github.ref }}`
(cancel-in-progress: false) so two pushes to main never race the
`:latest` retag. Don't cancel in-progress runs -- the build is
expensive and a half-built image left around is worse than a stale
:latest for a few minutes.
- Dockerfile: soften the requirements.lock.txt comment. `pip freeze`
captures versions but not wheel hashes, and several deps resolve
from VCS / nightly indexes that float, so the file is not actually
byte-reproducible. Reword as an "informational pin record".
1. Stop leaking secrets via docker run -e VAR=VALUE argv (run.sh, test_locally.sh)
`docker run ... -e HF_TOKEN=hf_xxx ...` puts the literal token in
the docker CLI's argv, which is visible to any user on the host
via `ps auxe` / `/proc/<pid>/cmdline` for the lifetime of the
process. Switch to the dash-only form `-e HF_TOKEN`, which tells
docker to read the value from the parent shell's env and never
appears in argv. Same fix for WANDB_API_KEY and UNSLOTH_LICENSE in
run.sh and HF_TOKEN in test_locally.sh.
2. Stop stripping numpy/tests/ in the runtime layer (Dockerfile)
The Dockerfile explicitly upgrades numpy >= 2.4 because numpy 2.2.6
shipped a stripped wheel where `from numpy._core.tests._natype
import pd_NA` fails. Numpy 2.4 restores `numpy/_core/tests/`, then
the existing `find ${VENV} -name tests -exec rm -rf {} +` deleted
it again -- re-introducing the same broken-import state on the
deployed image (the build-time verification at line 220 runs
BEFORE the strip so it passed). Whitelist numpy's tests directories
from the strip; keep stripping the rest.
3. Align :latest tag gate between merge and smoke-test jobs
(.github/workflows/docker-publish.yml)
merge job: enable = is-default-branch AND unsloth_ref == ''
smoke-test job: enable = is_default_branch only
On `workflow_dispatch`, `github.event.inputs.unsloth_ref` defaults to
"main" (not ""), so the merge step skipped `:latest` but the smoke
step still emitted `:latest` as tags[0]. The smoke step then
`docker pull`-ed a prior `:latest` from Docker Hub instead of the
image just merged -- so the smoke test verified the OLD image, not
the new one. Copy the merge step's exact `enable=` expression into
the smoke-test step so the two stay byte-identical and a workflow_
dispatch run validates whatever was actually merged.
* fix(chat_templates): check find() return value before slicing on placeholders
Two places in `construct_chat_template()` use `str.find()` for sentinel
placeholders (`{INPUT}` / `{OUTPUT}`) without checking the -1 return:
1. The `except:` fallback (around line 2464) computes
`chat_template[chat_template.find("{OUTPUT}") + len("{OUTPUT}"):]`.
If the template has no `{OUTPUT}` marker, `find()` returns -1 and the
slice starts at offset 7 (`-1 + len("{OUTPUT}")`), producing garbage
that's then `re.escape`-d and fed back into the template-recovery
regex. The user sees a confusing `IndexError` on
`response_part = response_part[0]` instead of the real problem.
2. The final trim before returning (`input_part[:input_part.find("{INPUT}")]`
and the matching `{OUTPUT}` line) silently drops the last character
when the placeholder is missing — `find()` returns -1, and `[:-1]`
slices everything except the last character, returning a corrupted
template prefix to the caller.
Replace both with an explicit `-1` check that raises a clear
`RuntimeError` naming the missing placeholder, matching the existing
guard pattern from #4923 (`try_fix_tokenizer`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(chat_templates): also guard {INPUT} and fallback regex/separator paths
Builds on the {OUTPUT} / final-trim guards in this branch by closing
the three remaining ways the except-block fallback in
construct_chat_template() can still raise a confusing IndexError or
AttributeError on malformed templates:
1. Validate both {INPUT} and {OUTPUT} before deriving `ending`. The
regex two lines later (`{INPUT} + ending + ...`) still produced an
empty list and crashed on `response_part[0]` if {INPUT} was missing.
2. Guard the regex no-match case. Some templates contain both
placeholders but not in a recoverable two-example shape, in which
case `re.findall` returns an empty list and `[0]` raises.
3. Initialize `found = None` before the separator-search loop and
raise if the loop never sets it. Previously, if the first
iteration's `re.finditer` was empty the loop broke without binding
`found`, and `found.group(1)` raised AttributeError on the stale
int left over from the outer rfind loop.
Rephrase the final-trim error messages from internal variable names
("input_part") to user-facing wording ("instruction section") and
include a bounded (200-char) excerpt of the offending content so the
error is debuggable without being unbounded.
Add tests/python/test_construct_chat_template_validation.py covering
each failure mode with a fake tokenizer (no HF_TOKEN, no model
download, CPU-only).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
PR #5537 bumped max_steps from 7 to 30 but the post-train assertion
still hardcoded the old count, so every fresh run that reaches the
post-train phase fails on `expected 7 logged steps, got [30 floats]`.
Derive the expected count from `config.max_steps` and add a
`train_result["train_steps"]` cross-check so the gate self-updates
with future sweep changes.
reviewer.py drops the 12 raw per-persona review markdown files under
individual_reviews/ in the current working tree. Drop them here and add
the directory to .gitignore so it cannot recur.
Round-2 of the 12-persona reviewer.py pass found 17 issues. Address the
P1s + the regression-class P2s in this commit; the remaining nits are
left for a follow-up cleanup pass.
1. unsloth/_gpu_init.py: the `NVIDIA_VISIBLE_DEVICES in os.environ` check
triggered for every NVIDIA-runtime container including `--gpus all`
(NVIDIA_VISIBLE_DEVICES=all is the default). Gate strictly on a
non-special device list. Also drop the precondition that the env var
was absent: if the user already pinned TORCHINDUCTOR_COMPILE_THREADS=1
we should still plant the UNSLOTH_FORCE_SINGLE_COMPILE_WORKER sentinel
so the zoo-side patch knows to preserve the forcing.
2. unsloth/_gpu_init.py: after the post-`import unsloth_zoo` reassertion,
monkey-patch `unsloth_zoo.temporary_patches.common.determine_compile_threads`
to return 1, so any later `torch.compile` call that rebuilds the
options dict still sees the single-worker forcing even if a downstream
patch_torch_compile pops the env var again.
3. docker/Dockerfile: torchaudio==2.11.0 mismatched the torch==2.10.0
release pairing; pin to 2.10.0 so the ABI is correct and the audio
stack matches torch/cu128.
4. docker/Dockerfile: drop `12.1+PTX` from TORCH_CUDA_ARCH_LIST. The
cu128 toolkit compiler does not know about compute_121; the trailing
PTX entry forced nvcc to emit a `sm_121` gencode that breaks any
in-container source builds.
5. docker/smoke_test.py: the device-capability floor said `cap[0] < 8`,
rejecting Turing (sm_75) while the Dockerfile + entrypoint advertise
sm_75 as supported. Lower the smoke floor to sm_75 and print a hint
that bf16 is not available on Turing.
6. docker/run.sh: `-it` is unconditional; CI / non-TTY invocations died
with "the input device is not a TTY". Probe `[ -t 0 ] && [ -t 1 ]`
first. Also remove `set -x` which echoed the forwarded HF_TOKEN /
WANDB_API_KEY / UNSLOTH_LICENSE values to stdout.
7. docker/test_locally.sh: `-e HF_TOKEN="${HF_TOKEN:-}"` either pasted
the secret verbatim into the process arg list or shadowed any
in-container value with an empty string. Forward conditionally.
8. .github/workflows/docker-publish.yml: gate `latest` on default branch
AND on `unsloth_ref` not being overridden via workflow_dispatch.
Otherwise a maintainer testing a feature SHA from main could overwrite
`:latest` with non-main source.
9. docker/Dockerfile.studio: add an `UNSLOTH_STUDIO_REF` build-arg so
the Studio companion image is pinned to a known unsloth ref instead
of cloning `main` whenever it builds.
Two vision-notebook deps that ship by reference rather than via unsloth
extras: transformers' Gemma3N + TimmWrapperModel needs `timm`, and
DeepSeek-OCR's dynamic modeling file requires `addict`. Both are tiny
(~30MB combined). Including them in the base unified resolve avoids
hitting `ImportError: TimmWrapperModel requires the timm library`
or `ImportError: This modeling file requires the following packages
that were not found in your environment: addict` after the user has
already downloaded the model.
Repros: nb/Gemma3N_(4B)-Vision.ipynb (timm), nb/Deepseek_OCR_(3B).ipynb
(addict).
PyPI release unsloth 2026.5.7 is now live. Bumps the pinned floor in
install.sh and install.ps1 from unsloth>=2026.5.6 to unsloth>=2026.5.7
so fresh installs resolve to the new wheel.
Tagged on main as v0.1.416-beta.
install.sh --local installs unsloth into the Studio venv as an editable
package keyed to the just-cloned source tree. We were rm-rf'ing that
tree in the same RUN; the resulting `unsloth_cli` import then failed at
container start with `ModuleNotFoundError: No module named 'unsloth_cli'`.
Clone the source directly under UNSLOTH_STUDIO_HOME/src so it persists
in the image layer, and strip only .git to save ~120MB.
The previous commit pulled in 51 transient async-task-output markdown files
from the local workspace's reviewer.py runs. Drop them and add the directory
to .gitignore so it cannot recur.
Round-trip with the reviewer.py 12-persona pass surfaced four real
issues. Fix all four in this PR so the new Docker release path is
self-consistent.
1. docker/smoke_test.py used `import xformers` unconditionally, which
guarantees a failure on arm64 (built with `[huggingface]` extras to
skip xformers since it has no aarch64 cu128 wheel). Wrap the import
in try/except so the same smoke script validates both arches.
2. unsloth/_gpu_init.py forced `TORCHINDUCTOR_COMPILE_THREADS=1` before
`import unsloth_zoo`, but `patch_torch_compile` in unsloth_zoo main
pops that env var in non-debug mode. After unsloth_zoo init the
guard was effectively undone, so cgroup-pinned `docker --gpus
'"device=N"'` containers still spawned the Inductor subprocess pool
that cannot enumerate the GPU. Set `torch._inductor.config.
compile_threads = 1` directly post-import-torch and re-populate the
env var so `determine_compile_threads()` in the zoo options dict
also returns 1, regardless of whether the zoo-side fix from PR #694
has shipped yet.
3. docker-publish.yml UNSLOTH_REF build-arg defaulted to `'main'` for
tag pushes and scheduled runs, so a `v1.2.3` release image would
contain whatever `main` happened to be at build time, not v1.2.3.
Pick the tag's `github.ref_name` for tag events and `github.sha`
for branch/schedule events.
4. The smoke-test job pulled `:latest` regardless of which tag the
merge job had just published, so tag/schedule/sha publishes were
never actually validated. Re-run docker/metadata-action with the
same config the merge job used, then smoke-test the first tag from
its output.
All four changes are gated and backwards-compatible.
The base unsloth-blackwell image ships the `unsloth` CLI but refuses to
start `unsloth studio` until the dedicated Studio venv is laid down under
UNSLOTH_STUDIO_HOME by install.sh. Build it once and commit the result as
an opt-in companion tag (`:studio`) instead of bloating the base image.
Build:
docker buildx build --build-arg BASE_TAG=test \
-f docker/Dockerfile.studio -t unsloth-blackwell:studio docker/
Run:
docker run --rm --gpus '"device=0"' -p 8888:8888 unsloth-blackwell:studio
Open http://localhost:8888. Inference (llama.cpp CPU + GPU) and training
are both available. First-boot admin password lands in container logs
and at /opt/unsloth-studio/auth/.bootstrap_password.
1. Inductor subprocess GPU invisibility in `--gpus '"device=N"'` containers.
The NVIDIA container runtime sets NVIDIA_VISIBLE_DEVICES but not
CUDA_VISIBLE_DEVICES; Inductor's compile worker subprocess pool then
cannot enumerate the cgroup-pinned device and raises `Could not find
an active GPU backend` from triton_helpers.set_driver_to_gpu. Force
a single in-process compile thread on that exact fingerprint; opt
out via UNSLOTH_FORCE_SINGLE_COMPILE_WORKER=0.
Repros: nb/Mistral_v0.3_(7B)-CPT.ipynb, nb/gpt-oss-(20B)-Fine-tuning.ipynb.
2. unsloth_base_fast_generate injects `logits_to_keep` before transformers
`_validate_model_kwargs` runs. transformers 5.0 auto-injects the same
kwarg inside `GenerationMixin.generate` (utils.py:2527) AFTER the
validator, so PEFT-wrapped GRPO models raise
ValueError: The following `model_kwargs` are not used by the model: ['logits_to_keep']
Gate the legacy injection on transformers < 5.0 and defensively pop
any leaked kwarg on 5.x. transformers 4.57.6 behaviour is preserved.
Repros: nb/gpt-oss-(20B)-GRPO.ipynb, nb/gpt_oss_(20B)_RL_2048.ipynb.
Both patches are gated and backwards-compatible (transformers 4.57.6 + 5.x,
TRL 0.22.2 + 0.27.1 + 1.x, PEFT 0.19.x).
vLLM 0.19.1 pulls numpy down to 2.2.6 whose wheel ships numpy/_core/
without the tests/ subdir, but numpy/testing/_private/utils.py imports
`from numpy._core.tests._natype import pd_NA`. Anything that hits
`from numpy import *` (scipy._lib.array_api_compat does) then crashes.
unsloth_zoo's gemma patch does `from transformers.processing_utils import
Unpack` which touches that path, so `import unsloth` blew up on every
GRPO notebook in the vLLM image.
Bump numpy to >=2.4 right after the vllm install; vllm still imports
fine on numpy 2.4.6 (verified locally).
The case-arm `auto) [ "${TARGETARCH}" = "amd64" ] && WANT_VLLM=1` exits 1
on arm64 (the [ test ] is false and nothing follows ||), which with
`set -e` aborts the entire RUN. Replace with an explicit if/then/fi so
each arch's auto branch returns 0.
Caught by ubuntu-24.04-arm CI on the staging fork.
Unsloth's GRPO notebooks (Qwen3_4B-GRPO.ipynb, Qwen3_8B_FP8_GRPO.ipynb,
Llama_FP8_GRPO.ipynb, etc.) set `fast_inference=True` which requires
vLLM to be importable in the same venv. Install vllm pre-release wheels
from https://wheels.vllm.ai/nightly alongside the cu128 pytorch index,
holding torch==2.10.0 fixed so uv refuses any vLLM build that would
yank torch out from under unsloth.
amd64 only -- vLLM does not publish aarch64 wheels yet
(vllm-project/vllm#31128 is open). On arm64 the GRPO notebooks that
need fast_inference will fail to import vllm; non-GRPO and
fast_inference=False paths are unaffected.
Gated by ARG INSTALL_VLLM=auto so the install can be disabled for
contributors who want a smaller image or are blocked by vllm/torch
resolve conflicts during iteration.
The nvidia/cuda base image already registers the CUDA apt repo with its own
Signed-By keyring. Installing cuda-keyring_1.1-1_all.deb on top adds a
duplicate sources entry with a different Signed-By value, which makes
`apt-get update` refuse the entire repo:
E: Conflicting values set for option Signed-By regarding source
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/
The repo URL is monolithic (every CUDA version is served from the same
path), so we can install cuda-nvrtc-13-0 + cuda-nvcc-13-0 directly without
touching the keyring. Empirically reproduced on the ubuntu-24.04-arm
GitHub Actions runner (staging-fork CI run 26360461375); fix verified via
the same staging-fork after force-push.
* Studio: strip orphan tool_call XML from streamed visible content
The speculative-buffer state machine in
`studio/backend/core/inference/llama_cpp.py` can slice a tool_call XML
block between the silent DRAINING path and the user-visible
content_accum, depending on when in the model's emission the BUFFERING
-> STREAMING -> DRAINING transitions fire. Three leak shapes were
observed in a 2026-05-22 sweep of 900 Qwen3.5 / Qwen3.6 GGUF runs:
Pre-fix XML leak rate: 20/900 (2.22%), concentrated 6.7% on the
larger Q8 / MTP configs:
Qwen3.6-35B-A3B Q8_0 4/60 (6.7%)
Qwen3.6-35B-A3B-MTP Q4 4/60 (6.7%)
Qwen3.5-35B-A3B Q8_0 3/60 (5.0%)
Qwen3.6-27B Q8_0 3/60 (5.0%)
The existing `_TOOL_XML_RE` only matched well-formed
`<tool_call>...</tool_call>` and `<function=...></function>` pairs, so
unterminated openings (close was DRAINED) and orphan closes (opening
was DRAINED) survived the strip and reached the user.
Fix relaxes the regex to also strip:
1. Orphan opening up to end-of-string: `(?:</tool_call>|\Z)`
2. Orphan closing tag: bare `</tool_call>` / `</function>`
Verified on the full sweep: 20/900 -> 0/900 (100% of detected leaks
eliminated). 16 unit tests in `test_tool_xml_strip.py` pin all three
leak shapes plus the well-formed cases, plus parametrised checks on
the 5 actual real-world leak samples from the sweep data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip tail-only </parameter> orphan + tighten regex
The 2026-05-22 gdpval sweep surfaced a 4th XML-leak shape not caught
by the earlier regex: a bare `</parameter>\n\n` at end-of-buffer (7
of 192 trials, all Qwen3.5-27B + a few Qwen3.6-27B). The model emits
the full `<tool_call><function=...><parameter=...>...content...
</parameter></function></tool_call>` envelope, the speculative buffer
DRAINS the opening tags as intended, but EOS (max_tokens cutoff)
truncates the outer `</function></tool_call>` close, leaving just
`</parameter>` as the visible tail.
We strip this ONLY when end-anchored (`\s*\Z`) so legitimate
mid-text uses (user code samples, documentation discussing the
Qwen tool-call XML shape) survive. Verified on the 192-trial
gdpval corpus: before=7, after=0.
While at it, fold the five top-level alternations into three by
sharing tag-name and prefix subgroups:
<tool_call>... + <function=\w+>... + --> <(?:tool_call|function=\w+)>...
</tool_call> | </function> --> </(?:tool_call|function)>
Semantically identical (verified by replay over the 192-trial
corpus + adversarial inputs, 0 diffs) and 1.34x faster on real
workloads. Backtracking-safety pinned by two new perf guards
(256KB '<' spam, 1000x orphan opens).
Tests: 16 -> 28 (6 new functional + 4 well-formed-vs-orphan +
2 perf guards).
* Tighten comments in XML-strip regex and tests
Code says what it does; comments were repeating it. Strip the verbose
explanations down to the WHY-only bits (engine quirk, tail-anchor
rationale, real-world source of each test sample). No code changes.
inference.py: 21 -> 12 lines around _TOOL_XML_RE
test_tool_xml_strip.py: 343 -> 259 lines (-84)
Tests: 28/28 still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The earlier message had per-arch driver minimums (525/535/555/570) that
came from when each chip first got driver support. That's not how CUDA
toolkit floors work -- cu128 imposes 570.26+ on EVERY GPU regardless of
arch. Only B300 (sm_103) and DGX Spark (sm_121) need a newer driver
(580+), and they ship factory with those drivers anyway.
External HF README has the same correction applied in temp/hf_readme.md
(updated separately when published).
torch wheels ship their own cuDNN/cuBLAS/cuSPARSE/cuRAND/cuSOLVER/cuFFT/
NCCL/cuSparseLt inside torch/lib/, and libtorch_cuda.so's RPATH
($ORIGIN/../../nvidia/cudnn/lib:$ORIGIN/../../nvidia/cublas/lib:...)
points at those wheel-bundled copies. The dynamic loader resolves through
the wheel, never the system, so the libcudnn/libcublas in the system
cudnn-runtime layer are unreachable code on every pull.
Verified empirically via `readelf -d torch/lib/libtorch_cuda.so` and
confirmed bitsandbytes' NEEDED list resolves against torch's bundled
libcudart/libcublas/libcublasLt/libcusparse/libnvJitLink before bnb
loads. Triton's .so files have zero CUDA NEEDED entries -- they dlopen
through the host driver.
Compressed image saving: ~2.7 GB (cudnn-runtime base 2.86 GB -> base
0.10 GB, on amd64; arm64 similar). Uncompressed: ~5 GB. Zero functional
impact.
Source: Fork 5 image-size audit, May 2026.
Empirically (cu128 wheel SASS list `sm_80;90;90a;100;100a;120;120a` on
aarch64) the cu128 wheel covers DGX Spark sm_121 via sm_120 binary
forward-compat. BUT two CPU-side compilers shipped at cu12.8 do not know
sm_121 and need a cu13 swap:
(1) torch's bundled libnvrtc.so.12 from CUDA 12.8 rejects sm_121 as a
--gpu-architecture. Symlinks libnvrtc.so.13 over it.
(2) Triton's nvidia backend runs ptxas. Wheels older than 3.6.0 bundled
cu12.8 ptxas which silently downgrades sm_121 to sm_80 (see
triton-lang/triton#8335). Bump pin triton>=3.6.0 (3.6 bundles cu13
ptxas) AND install cuda-nvcc-13-0 so the entrypoint can point
TRITON_PTXAS_PATH at it as defense in depth.
Both fixes are arm64-only (gated on TARGETARCH, ~400 MB on the arm64
image; amd64 is untouched, no sm_121 hardware exists on x86_64). Neither
component talks to libcuda, so this does NOT bump the toolkit driver
floor away from cu128's 570+.
TRITON_PTXAS_PATH is set from the entrypoint (only when the cu13 ptxas
actually exists in the image) rather than via a Dockerfile ENV, because
ENV is unconditional and Triton errors out if TRITON_PTXAS_PATH points
at a nonexistent file.
Sources: martimramos/dgx-spark-ml-guide Challenge 14; triton-lang/triton
issue #8335; ptrblck PyTorch forum thread on sm_121 fwd-compat from
sm_120.
Daniel Han