Round-6 sonnet-panel review surfaced seven more concrete bypass
classes. All seven are now closed (629 tests passing, 60 new R6
regression tests):
1. Path traversal under home prefix. `~/../etc/shadow`,
`~root/../etc/shadow`, `~ubuntu/../../etc/shadow`, and
`/home/u/../u/.aws/credentials` all slipped because
`_normalize_path_separators` re-attached the home prefix even
when `..` had escaped it. Now when the tail of a home-prefixed
path begins with `..`, the projection is treated as absolute
(mirroring the runtime resolve when HOME is a single-segment path
like `/root`). POSIX `~<user>/` tilde-user expansion is
handled the same way.
2. Pandas / numpy reader keyword arguments. `pd.read_csv(filepath_or_buffer='/etc/shadow')`,
`pd.read_excel(io=...)`, `np.fromfile(fname=...)` used the actual
API parameter names that the previous gate's `{"file", "path"}`
kwarg list missed. The kwarg set is expanded to cover
`filepath_or_buffer`, `path_or_buf`, `fname`, `filename`, `io`,
`buf`, `source`, `src` and a few common variants.
3. Bash directory exfil verbs. `cp -r ~/.ssh /tmp/out`, `mv ~/.aws /tmp`,
`tar czf out.tar.gz ~/.ssh`, `rsync -av ~/.aws/ /tmp/`,
`zip -r out.zip ~/.ssh` previously slipped because
`_find_sensitive_paths` only flagged named files, leaving the
bash side asymmetric to the Python shutil dir-exfil gate. A new
`_BASH_DIR_EXFIL_RE` matches dir-copy verbs (`cp`, `mv`, `rsync`,
`tar`, `zip`, `7z`, `scp`, `sftp`, `xz`) followed by a sensitive
directory. `ls ~/.ssh` and `find ~/.aws -type f` stay allowed.
4. Inner-tree alias walk for eval / exec. `exec("import shutil as sh\nsh.copytree('~/.ssh', dst)")`
slipped because the inner AST visit did not re-run the
alias-tracking pre-pass that built `shutil_module_aliases`.
Extracted both pre-pass loops into helpers (`_run_alias_prepass`,
already had `_run_string_binding_prepass`) and call both on each
literal eval / exec payload before the visitor recurses.
5. Chained assignment. `a = b = '/etc/shadow'; open(a).read()`
slipped because the binding pre-pass only handled
`len(targets) == 1`. Multi-target Assign nodes now bind every
Name target to the resolved value.
6. Annotated assignment. `path: str = '/etc/shadow'; open(path)`
slipped because the binding pre-pass walked `ast.Assign` but
not `ast.AnnAssign`. Same-shape handler for single Name target.
7. Brace-bomb empty-alt bypass. `cat ~/{,x0,...,x341}/{.ssh/id_rsa,other}`
exhausts the expansion cap before the empty alt's second-brace
projection reaches `~/.ssh/id_rsa`. A defensive
`_SENSITIVE_IN_BRACE_RE` catches sensitive-name fragments inside
an unexpanded brace group attached to a sensitive root,
regardless of whether the brace expansion completed. Anchored
with `(?<=[,{/])` lookbehind and `(?=,|\}|/)` lookahead so
project-local lookalikes (`./workspace/home/u/{a,b}/...`) stay
allowed via the `_PATH_TOKEN_START` boundary.
NetworkAndIoVisitor inner-tree pre-pass. The visitor eval / exec
recursion now mirrors SignalEscapeVisitor's call to both
`_run_alias_prepass` and `_run_string_binding_prepass` so it is
independently correct regardless of visitor execution order.
Cumulative bypass closures across rounds 1 through 6: 24 distinct
classes, 629 regression tests, three-OS green.
Two more from the follow-up list closed (569 tests passing):
1. ``ast.Subscript`` resolution. ``open(['/etc/shadow'][0])`` and
``open({'k': '/etc/shadow'}['k'])`` previously slipped because
``_extract_string_from_node`` had no Subscript handler. List /
tuple / dict subscripts are now resolved: when the index is a
static constant we return the indexed value; otherwise any
sensitive entry in the container surfaces so the gate fires.
Indexes outside the container's static range fall back to
sensitive-scan + first-resolvable so adversarial patterns like
``open(['safe.txt', '/etc/shadow'][i])`` are still blocked.
2. UDP / ``connect_ex`` metadata destination. The connect-only
``NetworkAndIoVisitor`` gate missed ``s.sendto(data, address)`` /
``s.sendmsg(buffers, ancdata, flags, address)`` (the destination
tuple is positional but not at index 0) and ``s.connect_ex(addr)``
(non-raising connect variant). The visitor now matches the full
``{connect, connect_ex, sendto, sendmsg}`` set and scans every
positional arg for a ``(host, port)`` tuple shape; the first
resolved host wins.
17 new regression tests cover the Subscript class (8 blocked, 3
allowed) and the UDP / connect_ex class (4 blocked, 2 allowed).
After this commit, ``bypass_hunt.py`` reports zero NEW bypasses;
the only remaining ALLOWs are the documented follow-up list
(``getattr(__builtins__, ...)``, ``vars(__builtins__)[...]``,
``base64.b64decode`` of paths, ``chr()`` / ``str.join`` concat,
trusted-host upload-shape evasion).
Two final bypass classes from the round-5 follow-up list are now
closed (552 tests passing):
1. Bash glob under a sensitive root. ``cat /etc/sha*ow``, ``cat
/etc/sh?dow``, ``cat /etc/*``, ``cat ~/.ssh/id_*`` -- the shell
expands ``*`` / ``?`` against the filesystem at runtime, so the
literal-path scan never sees ``/etc/shadow``. A new
``_SENSITIVE_ROOT_WITH_GLOB_RE`` mirrors the existing
``_SENSITIVE_ROOT_WITH_EXPANSION_RE`` (which gates ``$(...)`` /
backtick substitutions) for the ``*`` / ``?`` family. The
``[^\s'\";&|`$]*`` literal-text-only constraint keeps the match
attached to the sensitive root token, so ``find /etc/ -name
'*.conf'`` (whitespace between root and glob) and project-local
globs like ``./src/*.py`` stay allowed.
2. Ternary ``IfExp`` branches. ``open('/etc/shadow' if cond else
'data.txt')`` previously slipped because
``_extract_string_from_node`` had no ``ast.IfExp`` handler.
Either branch can execute at runtime; the gate now resolves both
branches and prefers the sensitive one (via the same
``_looks_sensitive`` check that backs the binding-bias) so the
downstream check fires. Falls back to whichever branch resolved
when neither is sensitive.
24 new regression tests cover the glob class (10 blocked, 6 allowed)
and ternary (6 blocked, 2 allowed).
_find_sensitive_paths does not match /etc/passwd (it lives in the
open-call gate's _SENSITIVE_FILE_PREFIXES list, not in _ABSOLUTE_SENSITIVE),
so a chained reassign p='/etc/hosts'; p='/etc/passwd'; open(p) kept
/etc/hosts as the representative and slipped through. Duplicate the
open-call prefix list in the pre-pass scope so _looks_sensitive catches
/etc/passwd and the analogous /proc/<pid> reads too.
Refines the round-5 ``_looks_sensitive`` heuristic that biases
``_record_string_binding`` toward the dangerous value in a chained
reassignment. The substring hint set conflated ``/etc/shadow`` with
``/etc/hosts`` -- both contain ``/etc/`` -- so a payload like
``p = '/etc/hosts'; p = '/etc/shadow'; open(p)`` had ``cur`` already
flagged sensitive, the guard refused to update, and the resolved
value stayed at ``/etc/hosts`` (allow-listed). The chained shadow
binding then slipped through.
Now ``_looks_sensitive`` delegates to ``_find_sensitive_paths``, the
authoritative bash / file gate matcher, so the distinction is exact:
``/etc/hosts`` is allow-listed and ``/etc/shadow`` is sensitive.
``_record_string_binding`` also adopts a clean three-way rule mirroring
Python's last-wins semantics for sensitive values:
* New sensitive value: always wins (covers the chained shadow case).
* New benign value, current sensitive: keep current (static gate
cannot prove the new value executes; err on blocking).
* Both benign: latest seen wins.
Test suite still 528 passing.
Sonnet-panel review of round-4 surfaced seven concrete bypass classes
in the static gate. All seven are now closed (528 tests passing, 55
new R4 / R5 regression tests):
1. ``os.path.join`` alias bypasses. ``import os as o; o.path.join(...)``,
``from os.path import join`` (and ``as j``), ``from os import path``
(and ``as op``), ``import posixpath as pp``, ``from posixpath import
join`` -- previously the FQ match was literal-only (`os.path.join`,
`posixpath.join`, `ntpath.join`). A pre-pass walk collects every
alias of ``os`` / ``os.path`` / ``posixpath`` / ``ntpath`` and every
from-import of ``join`` / ``expanduser``; the resolver checks
``<alias>.join`` and bare aliased names too.
2. ``shutil`` alias bypasses. ``import shutil as sh; sh.copy(...)``,
``from shutil import copyfile``, ``from shutil import move as mv``,
etc. -- the file-copy gate matched only the literal ``shutil.X`` FQ.
The pre-pass now tracks shutil module aliases and from-import
aliases for ``copyfile`` / ``copy`` / ``copy2`` / ``copytree`` /
``move``; the gate canonicalises any matched alias to ``shutil.X``
so the error message identifies the operation.
3. First-assignment-wins binding bypass. ``p = '/tmp/safe'; p =
'/etc/shadow'; open(p)`` previously slipped because the pre-pass
guard ``_target.id not in string_bindings`` ignored every
reassignment, and the AST walk picked the safe value while Python
uses last-wins at runtime. New ``string_bindings_all`` tracks every
literal ever bound to a name; ``_record_string_binding`` biases the
representative value toward sensitive-shaped paths via a substring
hint set covering the credential / process-state root tokens. The
reverse order (``shadow`` then ``safe``) is also caught.
4. Brace-expansion off-by-one. ``cat ~/.aws/{x0,...,x62,credentials}``
exploited that ``_expand_brace_projections`` started with
``out = {original}`` (1 item) so a cap of 64 only left 63
alternative slots. The inner loop also broke per-alternative on
the cap, so the sensitive name at position 64+ was never reached.
Raised the cap to 1024 and the inner loop now expands all
alternatives of a brace in one pass before the outer cap can stop
the queue.
5. ``thread-self`` in shell-expansion regex. ``cat /proc/thread-self/
$(echo environ)`` was missed because ``_SENSITIVE_ROOT_WITH_EXPANSION_RE``
only listed ``self|\d+`` in the ``/proc/...`` alternation, while
``_ABSOLUTE_SENSITIVE`` correctly included ``thread-self``. One
alternation entry restores symmetry.
6. Eval / exec pre-pass not re-run. ``exec("p='/etc/shadow'\nopen(p)")``
slipped because the inner AST visit ran without the string-binding
pre-pass. Extracted the pre-pass into ``_run_string_binding_prepass``
and call it on each inner literal payload before the visitor
recurses, so payload-local variable assignments are visible.
7. Pathlib name binding pre-pass. ``p = Path('/etc/shadow');
p.read_text()`` slipped because the pre-pass only resolved string
literals -- pathlib constructor calls returned None and the bound
name remained unresolved. Pre-pass now falls back to
``_extract_pathlib_target`` using per-tree alias sets so
``import pathlib as pl; p = pl.Path(...)`` and ``from pathlib import
Path as P; p = P(...)`` both resolve. ``NamedExpr`` (walrus) is also
surfaced by the pre-pass so walrus-inside-eval expressions are
visible.
Pre-pass call order. The initial pre-pass invocation moves to AFTER
``_extract_pathlib_target`` is defined so the closure cell binds
correctly (Python looks up free variables in the enclosing scope at
CALL time, not at function-definition time).
Full sandbox suite: 528 passed (455 prior + 73 R4 / R5 regression tests).
Closes additional bypass classes surfaced while exercising the gate:
1. Module / function aliasing (`m = os; m.system(...)`,
`p = os.popen; p(...)`): `visit_Assign` now propagates the source
alias when one tracked-module name is bound to another, and tracks
bound method references into `shell_exec_aliases`. Previously only
`m = __import__('os')` was handled.
2. Importlib from-alias (`from importlib import import_module as IM;
IM('os').system(...)`): a new visitor-scope `import_module_aliases`
set plus a `_resolve_dynamic_module` wrapper recognises the
bound name in both inline-call and bound-name forms.
3. Shutil directory exfil (`shutil.copytree('~/.ssh', dst)`):
`_matches_sensitive_dir()` adds a directory-only matcher used by
the file-copy gate only. The boundary `(?=/?$|/?[\s'\";&|)<>])`
matches the path AS the directory but NOT a single file inside it,
so per-file allow-listed reads (`~/.ssh/known_hosts`, `~/.ssh/id_rsa.pub`)
still pass. Covers `.ssh`, `.aws`, `.config/gcloud`, `.gnupg`,
`.docker`, `.kube`, `.password-store`, plus `/etc`, `/etc/ssh`,
`/var/spool/cron`, `/proc/<pid>`.
4. Explicit-reader / aliased file readers (`io.FileIO('/etc/shadow')`,
`codecs.open('/etc/shadow')`, `from io import FileIO; FileIO(...)`):
the open-call detector now recognises these qualified forms and
the visitor tracks `from io|codecs import FileIO|open` aliases.
5. Bytes-literal paths (`open(b'/etc/shadow')`) and walrus
expressions (`open((p := '/etc/shadow'))`): `_extract_string_literal`
and `_extract_string_from_node` resolve `bytes` Constants via strict
UTF-8 decode and `NamedExpr` via RHS extraction (recording the
binding so later uses of the walrus target resolve too).
6. Tuple / list unpacking destructuring (`(a, b) = ('/etc', 'shadow');
open(a + '/' + b)` and `p, = ['/etc/shadow']; open(p)`): the
string-binding pre-pass now folds matched-length Tuple/List
destructurings element-wise.
7. Pandas / numpy file readers (`pd.read_csv('/etc/shadow')`,
`np.fromfile('/etc/shadow')`, etc.): suffix-match the common
reader method names so any alias of the source module flows
through the same sensitive-path gate as `open()`.
91 new regression tests cover each class, both blocked and legitimate
allow-list cases. Full sandbox suite: 473 passed.
Closes two static-bypass classes flagged during round-3 review:
1. __import__('os').system(...) / importlib.import_module('os').popen(...)
bypassed the bare os.system / subprocess.* gate because the receiver
was an ast.Call rather than an ast.Name in os_aliases. Adds
_resolve_dynamic_module_name() so:
* inline __import__('os').system(...)
* inline importlib.import_module('os').system(...)
* import importlib; mod = importlib.import_module('os'); mod.popen(...)
* m = __import__('subprocess'); m.run([...], shell=True)
all flow through the same shell-escape detection as
import os; os.system(...). Legit dynamic imports of safe modules
(json, pathlib, ...) remain allowed.
2. /proc/<pid>/cwd and /proc/<pid>/root are symlinks to the process
working directory and the filesystem root. The form
open(/proc/self/cwd/../../etc/shadow) bypassed
_normalize_path_separators because .. was collapsed against the
literal path, not the symlinked target. The form
open(/proc/self/root/etc/shadow) bypassed any chroot-style
defence. Adds matching entries to _ABSOLUTE_SENSITIVE so the bash
gate and the AST open() gate both block any access via these
symlink prefixes. Legitimate /proc/self/status etc. introspection
still flows.
Tests:
TestFollowup_DynamicImportShellEscape (2 cases, 10 parametrised)
TestFollowup_ProcSelfSymlinkTraversal (3 cases, 16 parametrised)
pytest studio/backend/tests/test_sandbox_hardening.py -q
-> 382 passed in 0.67s (was 356).
Round-4 follow-up on the hardening PR after a third 20-reviewer pass.
Closes the high-impact items from that review while preserving the
"do not regress legitimate tool calling" floor; lower-vote items that
would have measurable regression on legit code paths (broad shell
glob ?/*, $VAR in dynamic paths, ANSI-C $'...') are intentionally
deferred.
Parent-directory traversal: _normalize_path_separators now follows
.. segments through posixpath.normpath and reattaches the tilde or
${HOME} prefix, so cat /etc/apt/../shadow and
Path('/proc/self/fd/../environ').read_text() both reach the
canonical regex.
Built-in open() accepts PathLike: open(Path('/etc/shadow')) and
open(file=Path('/etc/shadow')) now flow through the pathlib resolver
the same way receiver reads do.
Pathlib home and transforms: Path.home() resolves to ~ so
(Path.home() / '.aws/credentials') hits the home regex;
.expanduser() / .resolve() / .absolute() are pass-throughs.
Pathlib semantics: _join_path_parts() now matches pathlib's
absolute-segment reset so Path('/tmp') / '/etc/shadow' resolves to
/etc/shadow as it does at runtime.
from builtins import exec as e: tracked in both visitors via
eval_exec_aliases so the aliased call still routes through the
literal-payload recursion.
Process state extensions: /proc/self/cmdline,
/proc/thread-self/*, and /proc/<pid>/task/<tid>/* are added to
_ABSOLUTE_SENSITIVE.
Numeric f-strings: f'/proc/{1}/environ' folds to a literal because
numeric ast.Constant values inside ast.FormattedValue are now
stringified.
os.path.join / os.path.expanduser: resolved statically by
_extract_string_from_node so the stdlib-helper construction paths
do not hide sensitive targets.
Variable assignment tracking: a pre-pass collects ``name = literal``
and ``name = eval`` / ``name = exec`` bindings; the visitors and the
pathlib resolver consult those bindings. The trusted-host gate
intentionally uses a separate strict literal extractor so legit
patterns like ``url = some_input; requests.get(url)`` still pass.
shutil.copyfile / copy / copy2 / copytree / move: the source argument
is gated the same way open() is, blocking file-copy exfil.
Concrete pathlib classes: PosixPath / WindowsPath / PurePath / etc.
are registered in path_aliases by default.
requests.request positional+keyword: for URL-second APIs, args[0] is
the HTTP method (not the URL); when there is only one positional, the
URL extraction falls through to the url= keyword instead of grabbing
the method.
Tests grow from 281 to 357 hardening cases; combined sweep 487 / 487.
Every fix has positive and negative coverage; legit tool calls
(open(Path('data.csv')), os.path.join('logs', 'today.log'),
url = some_input; requests.get(url), shutil.copyfile('a.txt', 'b.txt'))
continue to pass.
Round-3 follow-up on the same hardening PR after a second 20-reviewer
pass. Every change is detection-widening; legitimate tool calls remain
allowed.
Pathlib readers (Path.open, Path.read_text, Path.read_bytes) now share
one extraction path. The new _extract_pathlib_target helper resolves
Path(a), Path(a, b, ...), Path(...).joinpath(b), and Path(...) / b
through statically-resolvable string parts. NetworkAndIoVisitor tracks
Path aliases (from pathlib import Path as P) and pathlib module aliases
(import pathlib as pl) so the aliased forms hit the same gate. For
receiver-side reads the path is taken exclusively from the receiver --
Path('/etc/shadow').open('r') no longer mis-reads the mode flag as a
path.
The credential-path regex set now matches the POSIX ~user/ expansion
(cat ~ubuntu/.aws/credentials), and the SSH private-key end anchor
includes > so that redirect-attached forms (cat ~/.ssh/id_rsa>... )
are not split-tokenised through the gate. A new
_SENSITIVE_ROOT_WITH_EXPANSION_RE detects sensitive root prefixes
followed by $(...) or backtick substitution, and _find_sensitive_paths
now enumerates bash brace expansion {a,b} plus small glob char classes,
and runs every projection through path-separator normalisation that
collapses // and /./.
Network host validation reaches keyword arguments (url=, host=,
hostname=, address=), host-first APIs whose first positional arg is the
host (socket.create_connection, socket.getaddrinfo,
http.client.HTTPConnection, http.client.HTTPSConnection), and the
url-second APIs (requests.request, httpx.request).
builtins.exec, builtins.eval, and __builtins__.eval flow through the
same literal-payload recursion as the bare forms, including aliased
import builtins as b. open(file=...) and io.open(file=...) keyword
forms are gated alongside the positional form, and the open() path
candidates run through both backslash normalisation and the
//-collapse projection so equivalent spellings (/etc//shadow,
/etc/./shadow) cannot bypass.
Tests grow from 205 to 281 hardening cases (TestR2Finding1 through
TestR2Finding16) and from 131 + 205 = 336 to 131 + 281 = 412 in the
local sweep. Negative cases for every fix continue to ensure
legitimate tool use (Path('data.csv').open(), open(file='logs/today.log'),
requests.get(url='https://wikipedia.org/'), find src/, etc.) stays
allowed.
Preserves per-turn OpenAI image_url content parts in the standard GGUF /v1/chat/completions path so multi-image chat history keeps each image attached to its original turn. Legacy top-level image_base64 is injected as a synthetic image_url part only when no message-level image exists. Tool use is disabled whenever any GGUF image is present. Fixes#5470.
On Windows runners shlex.split(posix=False) leaves splice quotes in place, so cat /etc/sha''dow tokenises to ['cat', "/etc/sha''dow"] and the dequoted scan projection still misses the credential. The threat model is POSIX-shell quote splicing in either bash invoked on Windows or POSIX shells on Linux/macOS; the dequote always wants POSIX semantics. Pre-normalise backslashes so Windows drive paths survive POSIX shlex's escape handling.
Round-2 fixes for ten issues surfaced by a 20-reviewer code review of
the initial hardening patches. Every change is detection-widening or a
false-positive narrowing; legitimate tool calls keep working.
Direct Python open now flows through _find_sensitive_paths so
open('/home/u/.aws/credentials').read() is gated the same as
os.system('cat ~/.aws/credentials'). The previous wiring covered only
the bash and shell-exec sides.
Both SignalEscapeVisitor and NetworkAndIoVisitor fail-closed once the
eval / exec literal recursion cap is reached. Wrapping a payload in
four or more nested literal exec layers no longer silently bypasses
inspection.
_find_sensitive_paths scans three projections of the command (raw,
backslash-normalised, shlex-dequoted) and recurses into nested
bash -c and cmd /c shells. Quote-spliced and Windows-backslash forms
of credential paths are all caught.
_HOME_PREFIX_RE adds Windows-style home prefixes (USERPROFILE,
HOMEDRIVE HOMEPATH, env:USERPROFILE, drive-letter Users) so cross-OS
hardening actually applies on Windows. Both sensitive-path regexes
now have a path-token start anchor so project-local lookalike paths
under workspace, fixtures, and tmp are not blocked.
Network host validation for sock.connect and the requests / urllib
FQ-prefix branch now use _extract_string_from_node instead of raw
ast.Constant checks, so concatenated and f-string literal hosts
resolve the same way the open gate already did.
pathlib.Path('/etc/shadow').open() is now inspected; the path is
extracted from the receiver constructor when node.args is empty.
The static-string resolver depth cap moves from 6 to 64, removing the
single-character literal-concat bypass while leaving the recursion
well inside CPython's default frame limit.
The SSH private-key regex gains a filename-end boundary so reading a
public ".pub" key stays allowed (legit developer action) while the
matching private key is still denied.
New regression tests cover one class per finding (TestFinding1 through
TestFinding10) plus updated nested-depth coverage; the previous
test_nested_depth_does_not_crash assertion was inverted by the
fail-closed change and has been replaced. Local sweep: 336 passed
(131 upstream + 205 hardening).
* fix: patch loss functions for Qwen3_5ForConditionalGeneration to prevent OOM errors
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Narrow except scope and simplify LOSS_MAPPING sweep
Replace bare except Exception with the only two compatibility errors we
actually care about so genuine bugs in the sweep surface. Drop the
redundant _key != "ForCausalLM" guard since the __name__ predicate
already excludes the patched entry (UnslothForCausalLMLoss != ForCausalLMLoss).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: emit one comma-chained --spec-type for CPU/Mac MTP path
llama-server takes a single --spec-type whose value may be
comma-separated to chain implementations (e.g. ngram-mod,draft-mtp).
The CPU/Mac MTP branch in LlamaCppBackend.load_model was passing
--spec-type twice in the same invocation, which is not the documented
chaining mechanism and silently drops one of the two specs depending
on llama.cpp's argv handling.
Collapse the pair to --spec-type ngram-mod,{mtp_token} and update the
stale _extra_args_set_spec_type docstring that claimed llama-server
accumulates repeated --spec-type. Update the matching pass-through
fixture in test_llama_server_args.py.
* studio: align MTP ngram-mod knobs with llama.cpp upstream defaults
Two correctness fixes against the llama.cpp server README:
1. The CPU/Mac comma-chained branch was emitting
--spec-ngram-mod-n-max 6 with --spec-ngram-mod-n-min 48, which is
nonsensical (min > max). Per the upstream default the value is 64.
2. The standalone ngram-mod branch was emitting --spec-ngram-size-n,
--draft-min, --draft-max. llama.cpp removed those arg aliases for
ngram-mod (they live only on the ngram-simple / map families now);
the correct knobs are --spec-ngram-mod-n-match / n-min / n-max.
Also refresh the inline comment block to point at the server README
rather than the older docs/speculative.md draft- aliases.
Reads HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE in FastLanguageModel.from_pretrained and FastModel.from_pretrained, forcing local_files_only=True so all delegation paths (load_in_4bit, load_in_8bit, full_finetuning, qat_scheme) and direct FastModel callers (FastVisionModel, FastTextModel) honour offline mode. Also gates HF_HUB_ENABLE_HF_TRANSFER in unsloth/dataprep/synthetic.py and adds an early return in get_statistics. Pairs with unslothai/unsloth-zoo#675. Fixes#5316.
Two related issues on the chat toasts:
1. Close X did nothing. The lib/toast.ts wrapper defaulted every toast
to `dismissible: false` (originally to keep swipe capture from
stealing text selection). In sonner v2, `dismissible: false` makes
the close-button onClick a no-op, so the X looked clickable but
never dismissed the toast. The Toaster already sets
`swipeDirections={[]}` in components/ui/sonner.tsx, so the
per-toast swipe workaround is unnecessary and harmful. Replace the
wrapper with a thin re-export of sonner.
2. Close X hover collapsed to a near-black circle in light mode.
Sonner's default close-button styling uses fixed gray-scale tokens
(--gray2 hover, --gray12 text) that ignore the theme attribute.
Once the Toaster's inline style overrides --normal-bg with
var(--popover), the base background follows the app theme but the
hover state does not, so the hover bg lands on a color that has no
contrast with the X glyph. Pin both base and hover to theme tokens
(--popover, --muted, --popover-foreground, --border) so contrast
stays visible in both light and dark modes.
Repro: open chat, load any cached model, hover the X on the
"<name> loaded" toast in light mode -- before this change the circle
turned dark and the click did nothing; after, the circle stays light
and the click dismisses the toast.
The AST gate previously had no special handling of eval() / exec(),
so a literal payload would slip past every detector:
exec("import os; os.system('sudo whoami')") # ALLOWED before
exec("open('/etc/shadow').read()") # ALLOWED before
eval("__import__('blocked_mod').dangerous()") # still allowed (chained-call gap)
payload = '...'; exec(payload) # ALLOWED before
Both visitors (SignalEscapeVisitor and NetworkAndIoVisitor) now share
the same gate at the top of visit_Call: when the call is bare-name
eval / exec, try to resolve the first argument via the shared
_extract_string_from_node helper (Patch A); if it resolves, parse it
and recursively visit so every existing detector runs on the inner
code — signal tampering, shell escape, sensitive-file open, network
policy, upload denylist, etc.
When the payload is not statically resolvable, SignalEscapeVisitor
appends a `shell_escape_dynamic` finding — eval/exec of runtime data
is the textbook code-injection vector and there is no legitimate LLM
tool-call reason to dynamically eval an external string. Static
literals (eval('1 + 2'), exec('x = 1\\ny = 2')) are unchanged because
the recursive visit only flags what the rest of the AST gate would
already flag at top level.
Each visitor caps recursion at depth 3 (own counter on the instance)
so adversarial nested eval('eval(...)') cannot blow the stack.
Closes gaps #7, #8 (partial), #11 (partial) from the 13-gap audit.
Out-of-scope chained-call cases (__import__('os').system(...),
getattr(os, 'sys'+'tem')()) stay documented gaps — the OS sandbox is
the intended backstop, see PR 5468.
Regression: 131/131 studio/backend/tests/test_sandbox_tools.py pass.
Legitimate eval/exec on literal expressions (eval('1+2'),
exec('print("hi")'), exec('exec("print(1)")')) verified ALLOWED.
Adds _find_sensitive_paths() and wires it into _bash_exec (alongside the
existing _find_blocked_commands check) and into _check_args_for_blocked
(so the Python AST gate catches os.system('cat ~/.ssh/id_rsa') the same
way bash $ cat ~/.ssh/id_rsa is caught).
The pattern set is intentionally narrow — only clear-cut credential and
process-state targets:
Home-anchored (must be prefixed by ~, $HOME, ${HOME}, /home/<u>,
/root, /Users/<u>):
.ssh/id_rsa, .ssh/id_ed25519, .ssh/id_ecdsa, .ssh/id_dsa, .ssh/identity
.aws/credentials, .docker/config.json, .kube/config
.config/gcloud/{application_default_credentials,access_tokens,credentials}
.pypirc, .npmrc, .cargo/credentials
.netrc, .password-store, .gnupg/private-keys-v1.d
Absolute system targets (match anywhere):
/etc/shadow, /etc/sudoers, /etc/ssh/ssh_host_*
/proc/{self,<pid>}/{environ,mem,maps,auxv}
/proc/kcore, /proc/kallsyms
/var/spool/cron/
The home-anchored category uses a regex that requires a HOME-equivalent
prefix, so project-local rc files like ./project/.npmrc remain readable
while ~/.npmrc is denied. Legitimate LLM-developer-tool paths
(~/.gitconfig, ~/.bashrc, ~/.ssh/config, ~/.ssh/known_hosts, /etc/hosts,
~/.cache/, ~/.bash_history, project rc files) are intentionally NOT in
the list and still flow through unchanged.
Closes gaps #1, #2, #3, #12, #13 from the documented 13-gap audit.
Regression sweep:
* 131/131 studio/backend/tests/test_sandbox_tools.py pass
* 24 legitimate-use cases verified ALLOWED
* 17 attack patterns verified BLOCKED
Static-string resolution in _extract_string_from_node was limited to bare
ast.Constant. Concatenated string literals and f-strings with constant
parts evaluated to ast.BinOp / ast.JoinedStr and slipped past the
open() sensitive-file check, so:
open('/etc/' + 'shadow') # ALLOWED before
open(f'/etc/{"shadow"}') # ALLOWED before
open('/etc/passwd') # BLOCKED before (literal)
The helper now resolves ast.BinOp(Add) of two resolvable strings and
ast.JoinedStr whose parts are themselves resolvable. The open()
sensitive-file check uses the helper instead of an inline ast.Constant
isinstance check, so the same widening covers concatenated/f-string
paths without changing what was already blocked.
Resolution is depth-capped at 6 to keep adversarial deep nesting from
blowing the stack. All other call sites of the helper
(_check_args_for_blocked, dynamic-arg shell-escape detection, HF upload
path-shape inspection) automatically inherit the broader resolution.
Closes open() gaps #4 and #6 from the documented 13-gap audit. Does not
attempt to model variable flow (gap #5 stays open by design — the OS
sandbox is the right layer for runtime flow).
Regression: 131/131 studio/backend/tests/test_sandbox_tools.py pass.
* studio: read Playwright default model from defaults.py without importing it
The Playwright Chat UI job installs Studio with --no-torch and does not
have structlog. Importing core.inference.defaults pulls in
core/inference/__init__.py (eager orchestrator -> structlog) and
defaults.py's own `import utils.hardware.hardware as hw` (also
structlog), so the test died before the first page action.
Read DEFAULT_MODELS_GGUF as a literal via ast.literal_eval. Zero side
effects, no new test deps, the EXPECTED_DEFAULT_MODEL override still
wins.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: engage draft-mtp on vision MTP GGUFs
The draft-mtp auto-promotion in LlamaCppBackend.load_model was gated on
not effective_is_vision, and the spec-emit branch repeated the same
guard. Every Unsloth -MTP GGUF repo ships an mmproj projector, so
effective_is_vision was always True for those repos and the MTP speedup
silently never engaged out of the box.
llama.cpp #22673 explicitly states MTP is compatible with vision input.
The bundled b9204 server happily loads both: a manual run with
--mmproj ... --spec-type draft-mtp --spec-draft-n-max 6 logs
"loaded multimodal model" followed by
"adding speculative implementation 'draft-mtp'".
Drop the vision gate from both sites and rewrite the matching short
circuit in _already_in_target_state so reload checks reach the auto
promotion path on vision MTP loads. Add three regression tests covering
vision MTP match (auto and default), and non MTP vision repo unaffected.
Verified on a B200 with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
base decode 179.7 t/s vs MTP decode 253.8 t/s, draft acceptance 0.57,
1.41x speedup on a 255 token completion. mmproj still loads and image
input remains available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: prefer Qwen3.5 -MTP GGUF variants in default model lists
With the vision gate dropped in the previous commit, draft-mtp now
auto-engages on -MTP GGUF repos out of the box. Swap the four Qwen3.5
recommended entries in DEFAULT_MODELS_GGUF and DEFAULT_MODELS_STANDARD
to their -MTP-GGUF counterparts so new users get the speedup by default:
unsloth/Qwen3.5-4B-GGUF -> unsloth/Qwen3.5-4B-MTP-GGUF
unsloth/Qwen3.5-9B-GGUF -> unsloth/Qwen3.5-9B-MTP-GGUF
unsloth/Qwen3.5-35B-A3B-GGUF -> unsloth/Qwen3.5-35B-A3B-MTP-GGUF
unsloth/Qwen3.5-0.8B-GGUF -> unsloth/Qwen3.5-0.8B-MTP-GGUF
All four HF repos exist (HEAD 200) and ship the same UD-Q4_K_XL quant
layout as the non-MTP variants. Non-Qwen3.5 entries are untouched.
* bump version to 2026.5.4
Picks up the studio MTP vision-gate fix and the Qwen3.5 -MTP default
swap in this PR.
* studio: prefer Qwen3.6-35B-A3B-MTP-GGUF in default model lists
Same rationale as the previous Qwen3.5 swap. The Qwen3.6 MTP variant
exists at unsloth/Qwen3.6-35B-A3B-MTP-GGUF (HF HEAD 200) and now
auto-engages draft-mtp out of the box with the gate fix.
* studio: drop --spec-draft-n-max from 6 to 3 for draft-mtp
n=6 is too greedy: on Qwen3.6 the draft has to guess 6 tokens ahead
and acceptance crashes to ~0.45, leaving only ~14% throughput gain.
PR ggml-org/llama.cpp#22673's author benched n=3 at ~0.72 acceptance
and 2 to 3x speedup on the same Qwen3.6 family, and the README sample
command uses n=2 or n=3. Match that.
CPU/Mac branch already uses n=3, so this aligns both paths.
* studio: set --spec-draft-n-max back to 6 for draft-mtp on GPU
Reverts the n=3 tuning. n=6 is the original default; user-side comparisons
hold the larger draft window steady so the toggle (next commit) is the
primary on/off lever.
* studio: add Speculative Decoding toggle under Max Tokens
Adds a top-level kill switch (panel-switch under Max Tokens, mirroring
Auto-Healing Tool Calls) that forces the /load request's
speculative_type to "off" when disabled. The backend "off" branch in
LlamaCppBackend.load_model skips both the draft-mtp auto-promotion and
the spec-emit branch, so neither --spec-type draft-mtp nor
--spec-default reaches llama-server.
Wiring:
- chat-runtime-store: new speculativeDecodingEnabled bool, default
true, persisted to localStorage under unsloth_speculative_decoding,
plus a setSpeculativeDecodingEnabled setter.
- chat-settings-sheet: SpeculativeDecodingToggle rendered immediately
beneath the Max Tokens slider for non-external models.
- use-chat-model-runtime: when speculativeDecodingEnabled is false,
override speculative_type to "off" in the loadModel call so the
switch wins over any pre-existing speculativeType state (including
the existing per-model toggle in Model Settings).
Verified end to end on unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
toggle ON emits --spec-type draft-mtp --spec-draft-n-max 6; toggle
OFF emits zero --spec-* flags on the same MTP GGUF.
* studio: relocate Speculative Decoding toggle into Model Settings
Move the toggle out from under Max Tokens and back into the Model
Settings section, directly beneath KV Cache Dtype, where the existing
Apply/Reset workflow already drives a reload on dirty. This way flipping
the switch in the UI actually picks up: the section becomes dirty,
Apply re-runs /load with the new speculative_type.
Drop the !currentModelIsMultimodal gate so vision MTP GGUFs can also
disable speculative decoding from the UI.
Switch the toggle's off-value from null to "off" so the backend's "off"
short-circuit fires for MTP models too (null normalises to None which
re-triggers the draft-mtp auto-promotion).
Tooltip now reads "Faster generation with 0% accuracy hit".
Remove the now-redundant speculativeDecodingEnabled bool + setter from
the runtime store and the load-time override in use-chat-model-runtime;
the toggle binds directly to speculativeType.
* studio: restore OOM/TIGHT badge on recommended GGUF rows
The recommended-list row passed vramStatus=null for any GGUF repo
because the existing useRecommendedModelVram hook reads safetensors
totals from HF model info, which GGUF-only repos do not expose. As a
result, an OOM Q-quant repo would render with only a "GGUF" badge and
no visual signal that nothing in it fits.
Add useGgufRecommendedFit: per repo, fetch the variant list via the
existing /api/models/gguf-variants endpoint, take the smallest
variant's size_bytes, and classify with the same 0.7*GPU + 0.7*RAM
thresholds as GgufVariantExpander. Session-scoped cache + in-flight
dedup so a repo is requested at most once.
Wire the result into the three GGUF row sites in pickers.tsx so OOM
and TIGHT badges show on the collapsed cards.
* Revert "studio: restore OOM/TIGHT badge on recommended GGUF rows"
This reverts commit 07793b1240df72b13e51d6dc15f63c4ee8c6cba9.
The new useGgufRecommendedFit hook was treating the symptom. PR #5561
identified the real root cause: useGpuInfo was calling /api/system
with plain fetch instead of authFetch, so the session-auth check
failed silently and gpu.available stayed false everywhere. With no
GPU info, every fit check (variant expander, recommended carousel)
fell back to "no signal" and dropped the OOM/TIGHT badges.
Reverting the over-engineered hook and applying the authFetch fix
in the next commit, which restores the existing badges with one line.
* chore: replace qwen suggested with MTP variant
* fix: restore GPU info auth for GGUF fit badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
unsloth 2026.5.3 was just published to PyPI. Update install.sh and
install.ps1 so fresh installs pull the new release (5 occurrences each).
Co-authored-by: Daniel Han <info@unsloth.ai>
* studio/chat: release stuck IME flag when compositionend never fires
Chrome on Windows talking to a WSL-hosted Studio (issue #5546) fires
compositionstart + compositionupdate but no compositionend after the
IME commits. The earlier hardening in #5327 cleared the stale flag on
the next non-composing input event, which never arrives in this
sequence, so composingRef stays true forever and the Send button stays
disabled even though the committed CJK text is already in the textarea.
Add a watchdog in both useImeComposerInputHandlers (main + edit
composer) and SharedComposer (compare mode) that runs the same reset
the missing compositionend would have done. The timer is rearmed on
every compositionupdate and on every non-composing input so it only
fires when the IME pipeline has actually gone quiet — normal candidate
selection keeps it alive, the WSL stuck case lets it expire.
Extends the existing IME Playwright smoke with a stuck-compositionend
repro and adds a static guard so the watchdog can't be removed without
the regression tests catching it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-pin composing flag on IME keydown to close#5546 watchdog gap
The stuck-compositionend watchdog (PR #5551) releases composingRef after
2500 ms of IME silence so Send unwedges in the WSL+Chrome case. The same
release also fires during a long candidate-window pause in healthy IMEs,
which lets a subsequent IME-confirm Enter slip preedit text through
handleSubmit (main composer) or click-Send through send() (compare composer).
Add a keydown gate to both composers: when the browser still reports
nativeEvent.isComposing or keyCode 229, re-pin composingRef and cancel
any pending watchdog so the next form-submit / send() guard refuses.
The Send button stays visually enabled (avoids re-introducing the
stuck-UI bug) but the submit path is blocked until a real compositionend
or non-composing input arrives. Mirrors the existing isComposing guard
shape in shared-composer.onKeyDown.
Tests:
- tests/studio/test_composer_rtl_bidi_attribute.py: two new static
guards asserting the keydown gate wiring in both composer files.
- tests/studio/playwright_chat_ime_i18n.py: new section 6c repro that
fires the IME-confirm keydown after the watchdog has cleared, then
triggers form.requestSubmit() and asserts the preedit text is not
cleared (would indicate a leaked submit).
Verified across Chromium / Firefox / WebKit via a side-by-side pre-PR
vs post-PR simulation (54 scenarios, zero pageerror or console.error).
The #5546 stuck-end repro still passes (Send re-enables 2.5-3 s after
the silent commit) and the new keydown-repin probe confirms the submit
gate refuses on all three engines.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: re-arm IME watchdog after keydown re-pin (Codex P1)
The keydown re-pin added in 2c3c9793 closed the watchdog-race for
healthy IMEs, but on the same WSL+Chrome no-compositionend path this
PR targets it would re-lock Send permanently: setting composingRef=true
and only *clearing* the watchdog leaves the flag pinned forever if no
follow-up compositionend or non-composing input ever arrives.
Swap clearStuckTimer/clearStuckImeTimer for refreshStuckTimer/
refreshStuckImeTimer in both composer keydown gates so the watchdog
fires once more after every IME keypress. Same visual contract — Send
stays enabled — the submit gate just keeps a 2.5s window before
re-releasing instead of staying locked.
Extends the playwright IME smoke with section 6d: clears composing via
the watchdog, fires an IME keydown, then waits past the re-armed
watchdog window and asserts the form submit actually flushes the
textarea. Two new static guards in test_composer_rtl_bidi_attribute
lock the refresh call into both keydown handlers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Two near-identical Discord button images existed under images/, with
the only effective difference being the rendered button width. Keep
the narrower variant (formerly the lowercase "discord button.png")
and remove the wider "Discord button.png", consolidating to a single
"Discord button.png" file.
* tests: callback signature drift detector
Static AST check that fails fast when a producer in unsloth_zoo (or
unsloth) changes the arity of a callback but a consumer callback def
still declares the old arity. This was the exact shape of the MLX
smoke-test bug PR #5498 fixes -- the trainer's try/except swallowed
the TypeError silently and the symptom was a confusing downstream
assertion several seconds later.
What the detector does:
* Producer side: walks every .py and finds classes that own a
self._<name>_callbacks list, populated via .append() from an
add_<name>_callback method, and invoked via
`for cb in self._<name>_callbacks: cb(arg1, ..., argN)`. The
arity at the call site is the canonical expected arity.
* Consumer side: walks every <obj>.add_<name>_callback(fn) call,
resolves fn to a def or lambda in the same file, and asserts
arity matches. Consumers that use *args or **kwargs are
tolerantly accepted as any arity.
* Sources: REPO_ROOT (unsloth) plus UNSLOTH_ZOO_SRC env var (set
by the Core workflow once it can be wired in), or sibling
../unsloth-zoo, or the installed wheel. Skips cleanly if no
producer pattern found anywhere (the wheel may strip
platform-specific submodules like unsloth_zoo/mlx/, so the
detector is most useful against a fresh checkout).
Validated end-to-end:
* Reverted run_real_mlx_smoke.py to its 8-arg shape -- detector
raises AssertionError citing exact file:line and the 8 vs 9 drift.
* Restored the 9-arg shape -- detector PASSes.
* Total runtime ~7 s in pytest.
Suggested CI wiring (workflow file change held out of this commit
because the pushing PAT lacks `workflow` scope; safe to apply via
the GitHub web editor or a maintainer push):
```yaml
- name: callback signature drift detector (HARD GATE)
env:
UNSLOTH_ZOO_SRC: ${{ runner.temp }}/unsloth-zoo
run: |
python -m pytest -v --tb=short tests/test_callback_signature_drift.py
```
Drop the step into .github/workflows/consolidated-tests-ci.yml right
after the existing public-api drift detector step. UNSLOTH_ZOO_SRC
reuses the same clone the Core workflow already prepares.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: wire callback-signature drift detector into Core matrix
Drops a 6-line pytest step right after the public-api drift detector,
with UNSLOTH_ZOO_SRC pointed at the freshly cloned $RUNNER_TEMP/unsloth-zoo
so the detector sees unsloth_zoo/mlx/ (the wheel strips it).
Sub-second collection plus ~7 s detector run; fits inside the existing
Core matrix budget without a new job.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* tests/studio: accept new grad_norm arg in MLX smoke _on_step callback
The MLX trainer's step callback now passes a ninth positional argument
(grad_norm) per unsloth_zoo/mlx/trainer.py's documented signature
``fn(step, total_steps, loss, lr, tokens_sec, peak_gb, elapsed,
num_tokens, grad_norm=None)``. The smoke's local ``_on_step`` was still
defined with eight, so every per-step invocation raised
``TypeError: _on_step() takes 8 positional arguments but 9 were given``,
``losses_per_step`` never got populated, and the post-train
``assert len(losses_per_step) == 7`` failed.
Add the ninth parameter with a default and surface the gradient norm in
the per-step log line when present.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: pin max_grad_value=0 in MLX smoke so max_grad_norm=1.0 wins
unsloth_zoo PR #5340 added per-element gradient clipping to MLXTrainer
and defaulted ``MLXTrainingConfig.max_grad_value = 5.0``. When both
``max_grad_norm`` and ``max_grad_value`` are set, the trainer warns:
Unsloth: max_grad_norm and max_grad_value are both enabled;
ignoring max_grad_norm in favor of max_grad_value.
and silently drops the test's ``max_grad_norm=1.0``. +-5.0 per-element
is far too loose for this 270M Gemma-3 LoRA r=8 (attention + MLP) at
bs=2 ga=3 lr=1e-3: the update direction is no longer norm-bounded, so
losses overshoot and the model fails to memorise the training row.
Reproduced on a CUDA mirror (scripts/cuda_mlx_mirror_sim.py):
norm_1 (max_grad_norm=1.0, no clip): losses 7.64 -> 0.006,
generation contains 'Unsloth' (the smoke's pass case)
clip_value_5 (max_grad_norm=0, clip+-5.0): losses 7.29 -> 8.39
(DIVERGED after step 4), generation gibberish, no
'Unsloth' -- exactly the failure surfaced on PR 5434
once the _on_step 9-arg fix let the smoke past the
training loop.
Pin ``max_grad_value=0.0`` so the smoke uses the same ``max_grad_norm=
1.0`` clipping it was designed against. Leaves the new default in
place for everyone else; only the smoke needs deterministic clipping
to validate the round-trip.
* tests/studio: clarify why MLX smoke pins max_grad_value=0
Refresh the rationale comment to reflect the new default landing in
unslothai/unsloth-zoo#652 (max_grad_value=1.0, not 5.0). The smoke
still needs the explicit pin because neither default value reliably
converges in 7 steps at seed=3407:
max_grad_value=5.0 -- diverges after step 4 (loss 7.3 -> 8.4)
max_grad_value=1.0 -- stalls (loss ~3.2 plateau across seeds)
max_grad_value=0.5/0.25/0.1 -- noisier still
max_grad_norm=1.0 -- cleanly drops loss to <0.01, emits "Unsloth!"
Mention both the historical 5.0 default and the new 1.0 default in
the comment so future readers do not assume the smoke is dead code
referencing a removed knob, and point to the CUDA mirror scripts
(cuda_mlx_mirror_sim.py + cuda_mlx_clip1_vs_norm1.py) for the
empirical evidence.
No behaviour change; comment-only refresh.
* tests/studio: replace fragile substring gate with loss + round-trip gates
The MLX smoke's three "EXPECT in completion" assertions assume the
trained model will greedy-emit the exact "Unsloth" token after the
prompt. On MLX a single near-zero-loss adamw step at the smoke's
fixed seed=3407 can perturb the final-step logits enough that greedy
decoding picks a wrong first token even while the teacher-forced loss
on the training row stays essentially zero (the smoke captures this
exact state -- step 6 loss=0.049, step 7 grad=36.7, step 7 loss=0.17;
completion goes from "Unsloth!" to "5 lbs!"). Reproduced extensively
on CUDA via scripts/cuda_mlx_step7_*.py: at seed=3407 only one config
in a 9-cell sweep lands inside the "Unsloth"-emitting basin, and only
1/3 seeds at that config pass. This is a property of the assertion,
not of save/reload correctness.
Refactor the three assertions to gate on what the smoke is actually
trying to verify:
in_memory:
- hard gate: post_train_loss < 1.0 (training memorised the row).
- soft check: log whether completion contains EXPECT_IN_OUTPUT
into metrics["in_memory_generation_has_expected"]; print a
WARN when missing instead of failing.
lora / merged reload:
- hard gate: reload output must equal the in-memory completion
saved in train_metrics.json. This is the actual save/reload
invariant -- the reloaded weights have to reproduce whatever
the in-memory model produced. Falls back to the original
gibberish gate if train_metrics.json is unavailable.
gguf reload:
- hard gate: llama.cpp produced usable, non-empty output after
the prompt (>=4 chars). llama.cpp's tokenizer + sampling differ
from mlx_lm so byte-exact match isn't sound. Log
gguf_has_expected for visibility.
Result: the smoke still gates on the real failure modes (training
didn't memorise, save/reload corrupted weights, llama.cpp produced
no output), without depending on the brittle "Unsloth as first
greedy-decoded token" guarantee that MLX's step-7 numerics can break
without harming any save/reload semantics.
Cross-version constraint: no transformers / trl API touched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/studio: gate MLX reload on training-row loss, not greedy text
The strict reload assertion (out == in_mem_out) failed on macOS:
in-memory completion was '5 lbs!' and the reloaded completion was
'_________________________'. Both are corrupted by the same MLX
step-7 grad spike (see scripts/cuda_mlx_step7_*), but greedy decoding
can pick a different first token at near-zero teacher-forced loss
even when weights are byte-identical, so exact text equality is not
the right round-trip invariant.
Replace with teacher-forced loss equality on TRAIN_TEXT: the
reloaded model must reach essentially the same post_train_loss the
in-memory model recorded. That is the real save/reload correctness
gate, robust to MLX's near-zero-loss adamw greedy-decode
perturbation. Falls back to a non-empty-body check when
train_metrics.json is missing.
CUDA mirror at this seed converges cleanly to ~0.006 loss; on MLX
post_train_loss < 1.0 still holds via the existing memorisation
gate. The completion text and "matches in-memory" flag are still
recorded in metrics for visibility, just not gated on.
* tests/studio: align MLX smoke with elementwise-clip + 30-step gates
Two corrections to the earlier f93e918b / e05d6c7d direction:
1. max_grad_value=0.0, max_grad_norm=1.0 picked the memory-heavy
norm clip. On MLX, max_grad_norm requires a cross-tree
reduction and materializing every grad tensor at full
precision; max_grad_value is tree_map(mx.clip) per leaf with
no reduction. MLXTrainingConfig defaults to max_grad_value=1.0
for exactly this reason. Flip the smoke to
max_grad_norm=0.0, max_grad_value=1.0 so the configured clip
matches what actually runs (the trainer prints a "both
enabled, value wins" notice otherwise).
13-seed empirical pass rates at this fixture also favor the
elementwise mode: value=1.0 62%, norm=1.0 46%, value=5.0 33%,
value=0.5 77%. Cheaper default = higher pass rate, no
tradeoff. (See PR #5498 / staging-2#119 rounds A-AT.)
2. max_steps=7 was below the convergence horizon at every clip
tested. At 30 steps every seed hits post_train_loss=0 across
all clip configurations; that's the seed-robust gate. Bump
max_steps 7 -> 30, tighten the memorisation gate from
post_loss < 1.0 to post_loss < 0.1.
3. Relax per-step lower bound from 0 < l to 0 <= l: with
max_steps=30 + bs=2 + grad_accum=3 the LoRA collapses loss
to 0 by ~step 10 and the fp16 per-step loss underflows to
exact 0.0 from then on. That's the success signal, not a bug.
Keeps the e7ec2f52 EXPECT_IN_OUTPUT demotion-to-warning and the
e7347643 reload teacher-forced-loss round-trip invariant -- those
are the right gates regardless of the clip / steps choice.
* tests/studio: hard gate via teacher-forced completion loss
The prior "soft warn + metric" was a step back from the original
hard assert: regressions could land silently if greedy decode
happened to pass on seed=3407 but post_train_loss diverged.
A true hard gate is needed.
Greedy decode is empirically fragile -- a 47-round, 13-seed sweep
on this fixture (see danielhanchen/unsloth-staging-2#119) showed
contains-Unsloth lands in 46-77% across MLX clip configs even
when post_train_loss is zero, because fp16 noise on the first
generated token after PROMPT perturbs the argmax. Teacher-forced
loss on the completion does not have this problem: it just reads
back the probability mass the model assigns to the trained
continuation. In every config where post_train_loss < 0.1, the
completion loss is essentially zero.
Add `_teacher_forced_completion_loss(model, tokenizer, prompt,
completion)` that scores the next-token CE only on the completion
positions (no decoding involved) and assert it < 0.5. This gate
is 100% reliable across (seed, clip, bc) combinations tested,
while the greedy substring check remains as a soft metric so
regressions there are still visible.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: hide Current password input on first boot
PR #5490 added a third Current password input to the change-password form
so the admin-forced must_change_password reset path could supply a current
password (the bootstrap is empty in that path). The side effect is that the
dominant first-boot UX, which has window.__UNSLOTH_BOOTSTRAP__ present and
silently fed into currentPassword, now shows three visible inputs instead
of the two it had before.
Render the Current password input only when window.__UNSLOTH_BOOTSTRAP__
is absent. The loadBootstrap effect already seeds the password state from
the bootstrap and currentPassword keeps the bootstrap fallback, so
handleSubmit sees the same value as before. On admin-forced resets where
the bootstrap is undefined, the Current password input still appears so
the user can type their actual current password.
Verified end-to-end against a local install via UNSLOTH_STUDIO_HOME +
install.sh --local with Playwright driving the page: bootstrap present
renders two inputs (New, Confirm) and completes change-password into
/chat; bootstrap suppressed via a non-configurable property descriptor
init script renders the three inputs (Current, New, Confirm) and keeps
the #5490 fix intact.
* studio/frontend: add deterministic input-count tests for auth-form
Pure-source pytest covering the change-password JSX contract. No
browser, no Studio boot, no JS toolchain -- runs on any CI runner.
Complements the Playwright probe in tests/studio/playwright_chat_ui.py
which exercises the same contract end to end.
Pins seven invariants with explicit failure reasons:
1. hasBootstrapPassword is derived from window.__UNSLOTH_BOOTSTRAP__
so a future swap to a localStorage flag or prop cannot silently
drift from the backend's _inject_bootstrap contract in
studio/backend/main.py.
2. Exactly one !hasBootstrapPassword conditional exists; multiple
would split rendering into branches these tests cannot reason
about.
3. The Current password input sits inside that conditional, so it
never renders on first boot (the regression PR #5490 introduced
and that this fix reverses).
4. The New password input sits outside it, so it always renders in
change-password mode (admin-forced reset still works).
5. Confirm password: same as New.
6. The change-password JSX subtree declares exactly current /
new / confirm; a fourth password input would almost certainly
break the 2-input first-boot contract.
7. The login JSX subtree declares exactly one password input.
Verified the tests fail loudly on the pre-fix auth-form.tsx at
c4575ca0 (5/7 fail with descriptive reasons) and pass on the fixed
version (7/7).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fast_generate: unify legacy/new logits kwarg + fix Mistral merge site
Two related issues caught by review on PR #5538:
1. unsloth_fast_generate (models/llama.py)
The previous patch promoted num_logits_to_keep -> logits_to_keep
unconditionally whenever the caller supplied num_logits_to_keep,
and only popped num_logits_to_keep (not logits_to_keep). On
transformers older than 4.50 (legacy spelling is the only one the
model forward accepts), the promotion broke things; symmetrically,
a caller supplying logits_to_keep on those older transformers also
went unchecked.
Switch to the unified normalize-then-inspect pattern from the
review:
_provided_num = kwargs.pop("num_logits_to_keep", None)
_provided_logits = kwargs.pop("logits_to_keep", None)
_provided = _provided_logits if _provided_logits is not None else _provided_num
_fwd_params = inspect.signature(self.forward).parameters
if "logits_to_keep" in _fwd_params:
kwargs["logits_to_keep"] = _provided if _provided is not None else 1
elif "num_logits_to_keep" in _fwd_params:
kwargs["num_logits_to_keep"] = _provided if _provided is not None else 1
Inspect the runtime forward signature first, then choose the
spelling it actually accepts, then route either user-supplied value
under that spelling. Backward-compatible in both directions.
2. MistralForCausalLM_fast_forward (models/mistral.py)
The max(num_logits_to_keep, logits_to_keep) merge was inside the
`if UNSLOTH_RETURN_HIDDEN_STATES:` block, so it only fired on the
GRPO hidden-states path. On the normal generation path the elif at
line 316 only checked num_logits_to_keep, so a caller (including
unsloth_fast_generate itself) passing logits_to_keep=1 ended up
computing full prompt logits instead of slicing to the last token.
For long prompts that reintroduces the large prefill logits
allocation the default keep=1 was avoiding.
Move the max() merge above the env-var branching so the normal
generation path slices correctly too. Llama already did this
merge at the top (unsloth/models/llama.py:1501); Mistral now
matches.
No behaviour change on the default GRPO / SFT paths. Targets only the
edge cases the review flagged.
* fast_generate: preserve caller logits kwarg when signature inspect fails
If `inspect.signature(self.forward)` raises TypeError/ValueError (opaque
C-extension or compiled wrappers), the previous fix set `_fwd_params = {}`
which silently dropped the caller-supplied `logits_to_keep` /
`num_logits_to_keep`. Fall back to the spelling the caller used (default
`logits_to_keep=1` when neither was supplied) so generation still honors
the requested logits slice.
* fast_forward: do not max() int against tensor logits_to_keep
HF accepts logits_to_keep as a 1-D LongTensor of positions for
selective decode. The merge in mistral.py (added by this PR) and
the pre-existing one in llama.py both run max(int, Tensor), which
casts the comparison to a bool and raises on multi-element tensors.
Branch on type and skip the merge when either argument is a tensor;
downstream int-slice path is unchanged, so tensor callers fall
through with num_logits_to_keep == 0, matching pre-merge behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate/forward: shorten kwarg-merge comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/frontend: soften toast shadow and tighten vertical padding
Sonner's defaults felt heavy in the chat header surface: a 16px
all-around padding made the box taller than the two-line content
warranted, and the 4/12/0.10 drop shadow read as a hard slab
against the light background. Trim padding to 10px vertical
(horizontal unchanged at 16px) and dial the shadow back to
0 2px 6px / 0.08 so the toast still lifts off the surface without
casting a heavy halo.
* studio/frontend: annotate why toast override needs !important
Sonner injects its base styles at runtime from inside its JS bundle,
so a plain cascade tie can lose depending on injection order. One
short comment above the override saves the next reader the dig.
* studio/frontend: boost toast shadow opacity in dark mode
Sonner's lighter 0.08 shadow disappears on the dark popover surface:
quantitative measurement of the shadow band (10px below the toast)
across Chromium / Firefox / WebKit showed only a ~3% luminance drop
vs background, well below perceptual threshold. Bump the dark-mode
opacity to 0.3, matching the existing .shadow-border light/dark ratio
(0.1 -> 0.3) and bringing the toast in line with .menu-soft-surface's
dark-mode shadow (0.28). Light mode keeps the original 0.08.
