Round 19 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Record a global/nonlocal sink alias in its TARGET scope (module for global, enclosing for
nonlocal): global s; s = os.system; s('...') now resolves s to the shell sink instead of
being skipped as a rebound name (and a global name no longer shadows an outer alias).
- Fail closed on any expansion in COMMAND POSITION, not just command substitution: a
variable-expanded command word (p=python3; $p -c ...) / ${VAR} is unprovable. Argument
position ($HOME, echo $(date)) stays allowed.
- Honor wrapper/command-position logic in the mutating-utility and shell-script scans via a
shared command-word index, so env sed -i / timeout 5 bash s.sh no longer hide behind a
wrapper prefix.
- Use the subprocess-exec callee resolver for the shell=True read scan, so
from subprocess import run as r; r('head /etc/passwd', shell=True) and r = subprocess.run
aliases are tokenized.
- Resolve sys.modules aliases (m = sys.modules; m.pop('_io')) for the loader-table mutation
checks (bound and unbound forms).
- Treat a __dict__ subscript keyed by a gadget dunder (type(open).__dict__['__closure__'].
__get__(open)) as gadget access, closing the descriptor-lookup route around the attribute
gadget scan.
- Unwrap inline literal containers in the higher-order sink check, so
list(map([eval][0], [...])) / map({'e': exec}['e'], ...) are flagged.
- Runtime: the mutator guard now accepts the path via its public keyword (os.makedirs(name=),
os.mkdir(path=)) instead of raising TypeError, while still confining the write.
Adds TestRound19Bypasses and keyword-path runtime tests; full sandbox suite green.
Round 18 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Fail closed on a command-position command substitution ($(printf touch) / `printf touch`
as the command word): the expansion becomes the command name and cannot be proven safe. An
argument-position substitution (echo $(date), x=$(cmd)) stays allowed.
- Model direct imports of the os C backend (posix/nt -> os aliases) and pty: posix.system(...)
resolves as an os shell sink and pty.spawn(...)/pty.fork() is flagged as an unguarded child.
- Reject unbound sys.modules mutation (dict.pop(sys.modules, '_io'),
type(sys.modules).__delitem__(sys.modules, ...)) alongside the bound sys.modules.pop form.
- Treat mutating flags of normally read-only utilities as child writers: sed -i, sort -o FILE,
find ... -delete, dd of=FILE, tee FILE, truncate. Non-mutating uses stay allowed.
- Detect os / subprocess re-exported through a stdlib module (pathlib.os.system,
tempfile.os.system, subprocess.os.system): the .os attribute IS the os module.
- Resolve instance-attribute sink aliases (c.e = exec; c.e(payload) / obj.s = os.system;
obj.s('rm -rf /')) tree-wide, alongside the existing class-attribute aliases.
- Import the guard's remaining pure-Python dep (shutil) with the workdir still stripped from
sys.path; the restore now runs at the very end of the prelude, so no workdir/shutil.py can
shadow it.
- Network policy: resolve import aliases (import requests as r; r.get(...)) and inspect the
url= / address= keyword arguments so aliased or keyword-host calls to a metadata / untrusted
host are no longer skipped.
Adds TestRound18Bypasses and extends the workdir-shadowing runtime test to shutil; full
sandbox suite green.
Round 17 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Track a code object built through a LOCAL compile alias (cfn = compile; co = cfn(src);
types.FunctionType(co, {})()) by resolving the callee through the scope exec-builtin map,
so the FunctionType execution gadget still gets the recursive payload analysis.
- Recognize type(lambda: None) as the function constructor: type(lambda: None) IS
types.FunctionType, so type(lambda: None)(code, {})() executed a compile() code object
without the eval/exec gate.
- Include lambda and comprehension scopes in the alias index: (lambda e=exec: e(payload))()
and [e(payload) for e in [exec]] now resolve e back to the exec sink. Lambdas/comprehensions
become their own alias scopes, and a one-element comprehension generator binds its target.
- Record annotated single-assignment aliases (e: object = exec; e(payload)) alongside plain
assignments, so the AnnAssign RHS is analyzed.
- Expand ${IFS} / $IFS to whitespace before shell command matching, so a separator-obfuscated
writer/reader (rm${IFS}-rf${IFS}/, cat${IFS}/etc/shadow) is tokenized as bash runs it.
- Import the child-guard's stdlib deps (os/io/pathlib/re) with the workdir stripped from
sys.path, then restore it, so a malicious workdir/os.py or pathlib.py cannot shadow a guard
import and run unguarded at import time.
- Fail closed on shell redirection to any real-file target (the unguarded child follows a
pre-existing symlink); only fd duplications (>&2) and the standard device sinks
(/dev/null, ...) are allowed. This also fixes a pre-existing false positive where a benign
redirect to /dev/null was blocked.
Adds TestRound17Bypasses and a workdir-shadowing runtime test; full sandbox suite green.
Round 16 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Apply the runtime sensitive-read backstop to Path.open reads (not just write
modes), so a dynamically assembled pathlib receiver (Path(globals()['P']).read_text())
cannot exfiltrate a host secret. On Python <= 3.11 pathlib holds the original io.open,
so confining at the public Path.open level is the version-robust fix. read_text /
read_bytes route through the same self.open() and are covered.
- Scan the -c payload of a subprocess shell argv for sensitive reads. subprocess.run(
['sh', '-c', 'head -1 /etc/passwd']) has no blocked command, but the unguarded child
prints the secret, so the payload is now tokenized and read-scanned like a string sink.
- Normalize bash ANSI-C ($'...') and locale ($"...") quoting before command matching.
shlex leaves $'touch' as the literal $touch, so a writer / interpreter hidden behind
ANSI-C quoting ($'touch' x, $'\x74ouch' x) previously evaded the command blocklist;
the escapes bash resolves (\n, \xHH, octal, \uHHHH) are decoded first.
- Recognize from-imported subprocess exec names as read sinks: from subprocess import
run as r; r(['cat', '../../etc/shadow']) now hits the traversal / sensitive-read check.
- Treat a shell glob that can expand outside the workdir (absolute / ~ rooted, e.g.
head /etc/shad*) as an escaping read expansion and fail closed, for reader arguments
and input redirects. A relative in-workdir glob (grep foo *.txt) stays allowed.
Adds TestRound16Bypasses and pathlib runtime backstop tests; full sandbox suite green.
Round 15 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Resolve single-assignment aliases for shutil.copy* and subprocess exec read
sinks (c = shutil.copy; c('../../etc/passwd', ...); r = subprocess.run; r([...]))
so the traversal / sensitive-path check fires on the aliased callee.
- Block gc.get_referents / get_referrers / get_objects (and from-import aliases):
they walk the object graph to a guarded wrapper's closure cell to recover the
original unguarded open / os.* callable.
- Cap list / tuple concatenation during constant folding so a doubling chain
(a + a + a + ...) cannot materialize an oversized sequence in the parent process
before the child rlimits apply.
- Add a runtime sensitive-read backstop in the child prelude: deny a read whose
realpath resolves to a known host secret (SSH / cloud / kube / netrc / HF-token /
/etc/passwd family / /proc) outside the workdir. This covers opaque read paths the
static scanner cannot fold (open(globals()['x'])) and pre-existing in-workdir
symlinks to secrets, while leaving benign outside reads and library imports intact.
- Fail closed on relative multi-component shell redirect targets (echo x > sub/out.txt)
whose subdirectory component could be a symlink traversing outside the workdir; a
bare single-component target stays allowed.
Adds TestRound15Bypasses and runtime backstop tests; full sandbox suite green.
Static classifier:
- Flag mro().pop(i) / __mro__.pop(i) base-class extraction alongside the subscript and __getitem__ forms.
- Normalize operator.attrgetter('name')(obj) as attribute access whether or not the result is immediately invoked, so attrgetter('__closure__')(open)[0] gadget recovery is caught.
- Resolve a container-hidden open alias (o = [open][0]; o('../../etc/passwd').read()) as a read callee.
- Fail closed on an opaque read path assembled from obfuscation primitives (open(''.join(map(chr, ...))).read()), matching the exec-payload obfuscation policy.
- Treat cd behind the command / builtin shell wrappers as a cwd escape before allowing a relative redirect.
- Block importlib file loaders as execution sinks (SourceFileLoader(...).load_module(), spec.loader.exec_module(...)).
- Add the in-cluster Kubernetes service-account credential path to the sensitive-read list.
Adds TestRound12Bypasses covering each vector plus benign controls.
Static classifier:
- Flag __getattribute__/__getattr__ with a non-foldable (runtime-assembled) attribute name as an obfuscated gadget access.
- Recognize POSIX noclobber redirect targets (>| path) in the shell redirect scanner.
- Detect a container-wrapped compile() code object passed to types.FunctionType.
- Flag mro().__getitem__(i) / __mro__.__getitem__(i) base-class extraction alongside the subscript form.
- Scan shell command strings (os.system / subprocess shell=True / getoutput) for embedded sensitive-file reads.
- Treat a default-parameter value that is a dangerous callable (def f(e=exec)) as a sink alias.
- Normalize a trailing .__call__ for shell/import/deserializer sinks (os.system.__call__, __import__.__call__, pickle.loads.__call__).
- Resolve a container-unwrapped sink assigned first (s = [os.system][0]; s(...)).
- Look through no-op pathlib methods (resolve/absolute/expanduser) when resolving a read receiver.
- Recognize from-imported shutil copy sinks (from shutil import copy as c).
- Extend the higher-order first-class-value check to shell and deserializer sinks (map(os.system, ...), partial(subprocess.getoutput, ...), map(pickle.loads, ...)).
- Normalize operator.attrgetter('name')(obj) as attribute-access obfuscation.
Adds TestRound11Bypasses covering each vector plus benign controls.
Static classifier:
- Flag mro()[i] base-class extraction alongside subscripted __mro__ (FileIO C base recovery).
- Fail closed on non-literal shell redirect targets and cd to an outside directory.
- Block sys.modules mutating methods (pop/popitem/clear/setdefault/update) that drop a guarded module for reimport.
- Detect indirect eval/exec: <builtin>.__call__(payload) and eval/exec/compile passed by reference to a higher-order call (map/reduce/partial), including starred literals.
- Block inspect.getclosurevars() closure recovery of a guarded wrapper.
- Track runpy run_path/run_module from-import aliases.
- Expand starred literal path arguments in the sensitive-read scanner.
- Resolve container-hidden deserializers (([pickle.loads][0])(payload)).
Runtime backstop:
- Guard the low-level posix/nt chdir (cwd escape) and fchmod/fchown fd metadata mutators, matching the os.* deniers.
Adds TestRound10Bypasses and low-level posix runtime tests.
Static classifier:
- deny sys.modules Store/Del: del sys.modules['posix']; import posix drops the guard-patched module for a fresh unwrapped C module
- flag code.InteractiveInterpreter().runcode / InteractiveConsole().runsource as code-object execution sinks (opaque compile results run un-analyzed)
- resolve indirect read-only open callees: from os/io/builtins import open as X (os.open read-only is deliberately allowed outside the workdir, so traversal must be caught statically)
- fold os.path.normpath / abspath on literals so a traversal that only emerges after normalization is scanned
- resolve inline-container-hidden exec/eval: ({'e': exec}['e'])(...) / [exec][0](...)
- normalize the bound one-arg __getattribute__ form obj.__getattribute__('name') (builtins.open.__getattribute__('__closure__'))
- descend into a literal list/tuple argv so subprocess.run(['cat', '/etc/passwd']) is caught
- enforce the analyzer node budget: charge each tree's node count against _MAX_ANALYZER_NODES and fail closed above it (parent-process DoS guard)
Runtime backstop:
- capture and re-pin os.lstat / os.readlink / os.getcwd / os.stat before realpath, since it consults the live symlink helpers -- monkeypatching os.lstat to fail could stop realpath following an in-workdir symlink that points outside
Static classifier:
- flag cell_contents (the only closure-cell reader) as a gadget so recovering a guarded wrapper's original callable via __closure__ fails closed even when the __closure__ name is built at runtime
- treat a class body as its own alias scope (class C: e = eval; e(...) now recognized) while keeping methods lexically skipping the class scope, so a same-named class attr does not shadow the module-level sink a method reaches
- block output redirection (> / >> / &> / N>) to an absolute / ~ / .. target: a child shell runs unguarded; relative in-workdir redirects stay allowed
- resolve pathlib constructor import aliases (from pathlib import Path as P) before traversal reads
- flag an integer-indexed __mro__ (io.FileIO.__mro__[1]) that extracts the unguarded FileIO C base class; plain iteration / slicing stays allowed
- resolve aliased read callees before traversal checks: o = open and import shutil as sh; sh.copy(...)
- recurse into env -S / --split-string operands so env -S 'python3 -c ...' still detects the interpreter
Static classifier:
- fix scope-local walker so a nested def/class reassigning an alias name no longer inflates the outer single-assignment count and drops a real module-level sink alias
- track non-bare compile aliases (builtins.compile, from builtins import compile as comp) for the types.FunctionType(c) code-object gadget
- block child-process file writers at shell command position (touch/tee/cp/mv/mkdir/install/truncate/mkfifo/mknod/shred/unlink): a spawned child runs without the in-process write guard
- expand a literal **{...} unpack in the read scanner so open(**{'file': '../../etc/passwd'}) is resolved
- resolve a pathlib expression bound to a single-assignment name before read methods (p = Path('..')/'etc'/'passwd'; p.read_text())
- keep a wrapper's separated option argument in command position so stdbuf -o L python -c ... still detects the interpreter (env -i rm still caught; no FP on grep patterns)
- treat object.__getattribute__ / type.__getattribute__ as attribute obfuscation, covering gadget dunders and sensitive-module attrs (also closes __closure__ recovery of a guarded wrapper's original callable)
- block runpy.run_path / runpy.run_module execution sinks
- treat shutil.copy*/move SOURCE as a read callee so a .. traversal source is caught
Runtime backstop:
- normalize a bytes realpath (fsdecode) before the workdir prefix compare so a legitimate in-workdir bytes write is not denied by a TypeError; outside bytes writes still denied
Static classifier:
- resolve pathlib expressions passed to open()/read callees so open(Path('/etc') / 'passwd') blocks like open('/etc/passwd')
- flag getattr()/setattr() of an introspection gadget dunder (__globals__, __subclasses__, ...) regardless of receiver
- add the .get() twin of the globals()/locals()/vars() namespace-dict subscript guard
- constant-fold sys.modules[...] and sys.modules.get(...) keys so a concatenated key is caught
- track 'from builtins import __import__ as imp' as a dynamic import alias
- treat deserializer module aliases (pickle, dill, ...) as sensitive targets for getattr/vars/__dict__
- flag code objects executed through types.FunctionType(compile(src, ...), ...), including the c = compile(src); FunctionType(c) two-step
- block language interpreters (python/perl/ruby/node/...) at shell command position: a spawned child runs without the in-process write guard
Runtime backstop:
- pin os.fspath/os.path.realpath to captured originals inside _within so a sandboxed reassignment of os.fspath cannot make realpath resolve an outside write target to an in-workdir path
Scope-aware assignment aliases (extends the per-scope index):
- Resolve single-assignment aliases of dangerous callables in the call's own scope:
e = builtins.eval, im = importlib.import_module (and imp = __import__), and
l = pickle.loads (incl. aliased modules). Previously only bare-name and from-import
aliases were recognized.
- Count function parameters as local bindings so a parameter lexically shadows an
outer sink alias of the same name (fixes a false positive where def f(s): s(...)
with a module-level s = os.system flagged the parameter call).
Sensitive-read scanner:
- Resolve pathlib join receivers -- (Path('/etc') / 'passwd').read_text() and
Path('/etc').joinpath('passwd') -- not just a bare Path(...) constructor.
- Normalize path spellings (collapse redundant separators / '.' and resolve '..')
before the exact / dir checks, so /etc//passwd, /etc/./passwd and
/tmp/../etc/passwd are matched.
- Fold function-local single-assignment path constants (def f(): p = '/etc/passwd';
open(p)), not only module-level constants.
- Flag sys.modules.get('os') as the method-call twin of sys.modules['os'].
Runtime realpath backstop:
- Path.open coerces a str-subclass mode through the base str (matching the other open
wrappers) so a lying __contains__ cannot skip the write check.
- Path.rename/replace/link materialize the target once so a stateful __fspath__ cannot
return an in-workdir path for the check and an outside one for the real call (the
pre-3.11 accessor path where this wrapper is the only confinement).
Adds regression tests across the classifier, aliasing and runtime-backstop suites.
Scope-aware alias resolution (replaces the flat, module-wide alias maps):
- A new per-scope index resolves shell-sink, exec-builtin and compiled-code aliases
with Python lexical scoping. This fixes two problems the flat maps had: a safe
`c = compile('1+1')` in one function no longer shadows a dynamic `exec(c)` in
another (a real bypass), and a `s = os.system` in one function no longer makes a
benign `s = print` call in another look like a shell sink (a false positive), while
still catching a genuine function-local sink and honoring local shadowing of a
module-level alias.
Runtime realpath backstop:
- io.FileIO now passes the MATERIALIZED fspath to the real constructor (a stateful
__fspath__ could otherwise return an outside path to the C constructor).
- Deny an integer fd path for the mutating single-path wrappers (os.chmod(fd) etc.):
a read-only fd opened on an outside file could otherwise mutate host metadata.
Constant-folder allocation DoS:
- Refuse dynamic printf widths/precisions ('%*s', '%.*f') that draw their size from a
runtime argument.
- Bound str.replace / str.join output before it allocates (a long replacement over
many occurrences, or joining many long parts, can build a multi-gigabyte string).
Static classifier:
- Flag builtins / a sensitive module reached through the namespace dict:
globals()['__builtins__'], locals()[...] and globals()['os'].
Adds regression tests across the aliasing, runtime-backstop, const-fold and classifier
suites for every item above.
Runtime realpath backstop:
- Guard the low-level posix / nt module mutators (os re-exports from them, so
posix.open / posix.rename / ... stayed reachable with the originals).
- Guard io.FileIO / _io.FileIO constructors for write modes (a C constructor that
opens a file without routing through open()).
- Add os.mkfifo / os.utime / os.setxattr / os.removexattr (and lchflags) to the
guarded single-path mutators.
- Materialize fspath ONCE per call so a stateful __fspath__ cannot return a workdir
path for the check and an outside path for the syscall (TOCTOU).
- Coerce open() mode through the base str and os.open flags through the base int, so
a str-subclass __contains__ or an int-subclass __and__ cannot lie to the guard.
Constant-folder allocation DoS:
- Refuse str.format templates with a nested width field ({:{}}) driven by an
oversized numeric argument before format() allocates.
Static classifier:
- Reconstruct the full pathlib receiver path (all constructor args, joined) and
accept module-qualified pathlib.Path so Path('/etc', 'passwd').read_text() and
pathlib.Path(...) reads are inspected, not just single-arg bare Path(...).
- Treat builtins.__import__ / __builtins__.__import__ as a dynamic import.
- Count alias single-assignment per function scope instead of tree-wide, so two
functions binding the same local name no longer cancel out and miss a real sink.
Adds regression tests across the runtime-backstop, const-fold, aliasing and
classifier suites for every item above.
Fixes a further batch of P1 bypasses and analysis-time DoS vectors found in review.
Static classifier:
- Decode exec/compile bytes payloads the way CPython does (PEP 263 coding cookie
via tokenize.detect_encoding), then analyze the real source. A bytes payload whose
UTF-8 view is pure comments but whose utf-7 decode runs hidden code no longer slips
through; a payload decoding to a blocked op blocks, a benign one stays allowed.
- Resolve exec-builtin aliases assigned in nested scopes (def f(): e = exec; e(...)),
matching the shell-sink aliasing (stored-once guard keeps it low false-positive).
- Treat deserializer modules (pickle/marshal/dill/...) as dangerous dynamic-import
targets so __import__('pickle').loads(blob) is caught.
- Flag vars(os) / vars(__builtins__) as a module-__dict__ obfuscation, like os.__dict__.
- Inspect the pathlib receiver path for read methods: Path('../../.ssh/id_rsa').read_text()
/ read_bytes() / open() now check the constructor path, not only call args.
- Fold literal os.path.join / posixpath.join so open(os.path.join('/etc','passwd')).read()
is seen by the sensitive-read scanner instead of treated as opaque.
Constant-folder allocation DoS (folding runs in-process, before subprocess rlimits):
- Reject oversized f-string / str.format / %-format widths and precisions before
format() allocates the padded string.
- Reject oversized str padding-method widths (ljust/rjust/center/zfill).
- Cap list/tuple repetition (seq * n) as str/bytes repetition already was.
Runtime realpath backstop:
- Do not publish __wrapped__ on the guard wrappers (functools.wraps would expose the
original unguarded callable, e.g. open.__wrapped__(outside, 'w')).
- Guard the low-level _io.open entry point (io.open / builtins.open originate there).
- Confine os.chdir to the workdir and deny os.fchdir so a cwd escape cannot turn a
later relative read/write into a host-path access.
- Deny fd-based metadata mutators (os.fchmod / os.fchown) that could reuse a read-only
descriptor opened on an outside file.
Adds regression tests across the const-fold, aliasing, exec-recursion and runtime-
backstop suites for every item above.
Harden the code-exec sandbox against bypasses raised in review, keeping the
static gate a pure classifier (it never executes the tool call):
- Shell scan: `timeout` duration/float args (5m, 0.5, 2h) no longer drop the
following command out of command position, and `find -exec CMD ... ;` rescans
the whole slice so a wrapped `env`/`timeout`/`sh -c` target is still caught.
- exec/eval/compile of a bytes payload that is not valid UTF-8 Python now blocks:
those sinks honor PEP 263 coding cookies (e.g. utf-7) that the static UTF-8 view
cannot see; plain ASCII bytes payloads stay allowed.
- Resolve aliased/indirect reaches to exec, dynamic import, sensitive modules and
deserialization sinks: builtins.eval / __builtins__.exec, from builtins import
exec as e, importlib aliases, sys.modules[...] (and the getattr form), os.__dict__,
posix/nt, and pickle/marshal module-and-symbol aliases plus the *.load variants.
- Shell-sink aliasing walks the whole tree, so a function-local `s = os.system`
alias is resolved (the stored-once guard keeps it low false-positive).
- Drop __mro__ / __code__ from the introspection-gadget dunders: on their own they
do not reach an execution primitive and are read by ordinary ML/debug code.
- Refuse folding oversized `bytes(n)` / `bytearray(n)` so static analysis cannot OOM.
Runtime realpath backstop:
- Fail closed on a mutating dir_fd / src_dir_fd / dst_dir_fd (an fd-relative path
cannot be confined by a string realpath) for os and shutil mutators.
- Confine Path.rename/replace/symlink_to/hardlink_to when the destination is passed
as the `target=` keyword, not only positionally.
- Splice the guard after a leading docstring and `from __future__` imports instead
of prepending it, so future-import programs no longer raise SyntaxError while the
sandbox is still established before the first real statement.
Adds regression tests for each gap across the shell, const-fold, exec-recursion,
aliasing and runtime-backstop suites.
On Python <= 3.11, pathlib._NormalAccessor captures io.open (and os.* mutators)
into class attributes at pathlib import time. A C builtin captured there does not
bind on instance access, but a Python wrapper does: self shifts into the next
positional, so Path.open / Path.write_text raised 'open() argument mode must be
str, not PosixPath' once the guard had replaced io.open with a Python wrapper
before pathlib was imported. (3.12+ dropped the accessor, which is why it only
failed on the 3.10 CI leg.)
Import io and pathlib at the top of the guard, before any patching, so the
accessor captures the original builtins, and confine Path.open by wrapping the
public method directly (mode-aware) rather than relying on the io.open patch to
reach it. Direct io.open() writers are still guarded for the zipfile-based cases.
The static filesystem write-confinement (the LOCAL/ESCAPE/UNKNOWN path resolver
_resolve_path / _resolve_path_call plus the _FS_* mutating-op inventory) was the
largest and most complex part of the classifier, and for writes it duplicated the
runtime realpath backstop, which is strictly more robust: it resolves the true
realpath at the syscall boundary, so it also catches dynamic paths, pre-existing
symlinks, and library writers the static pass could not prove.
Make the runtime backstop the single filesystem-write boundary and delete the
static resolver:
- Harden the backstop to close the gaps the static layer used to cover: guard the
low-level os.open (any mutating flag confines the target; a mutating dir_fd fails
closed) and io.open (which also carries pathlib.Path.open('w')), and add
os.mknod / lchmod / lchown / chflags and shutil.chown / copymode / copystat to
the wrapped set. Native-C writers (cv2.imwrite) and the realpath TOCTOU window
remain documented residuals that only OS-level isolation can close.
- Remove _resolve_path, _resolve_path_call, _resolve_join, _classify_path_string,
_is_pathlib_expr and the _FS_* / _PATHLIB_CTORS / _PATH_DEPTH_CAP constants, and
the write half of the filesystem visitor plus the FS_READ_STRICT knob.
- Reads are not confined by the backstop, so keep a small static sensitive-read
scanner (_is_sensitive_abs_path) that still blocks host-secret reads via a
sensitive absolute / ~ literal in any call arg (covers open, os.open, and library
loaders such as pandas.read_csv('/etc/passwd')) and .. / ~ traversal on the
dedicated open/read callees.
Net: about 300 fewer lines in tools.py and one fewer concept to audit; static
analysis now scopes to exec, shell, network and sensitive-reads while writes are
confined at runtime. Rework the filesystem tests around the new contract and add
os.open / io.open / Path.open / dir_fd escape cases to the backstop suite.
* Studio: add Vulkan llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address gemini's feedback
* Studio: move the Vulkan VRAM probe into a standalone script
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve Vulkan probe error reporting
* Resolve llama-server symlink so Vulkan build is detected
* Drop unreachable Vulkan fallback in GPU free-memory dispatcher
* Skip the Intel GPU probe when NVIDIA or ROCm is present
* Reserve host RAM headroom for Vulkan integrated GPUs
* Add a `UNSLOTH_FORCE_VULKAN` environment variable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor GGML_VK_VISIBLE_DEVICES, reserve discrete Vulkan VRAM headroom, and clear Intel GPU on --cpu-fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route Intel and forced-Vulkan hosts to the upstream Vulkan prebuilt, add arm64 Vulkan, keep Vulkan out of RAG auto-detect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the fork release pin when routing a Vulkan host to the upstream repo
* Gate auto-Vulkan routing on no physical NVIDIA so hidden CUDA devices aren't used
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin Vulkan launches with --device Vulkan<i> instead of the raw GGML_VK_VISIBLE_DEVICES index space
* Let user --device override the Vulkan pin, and gate direct Vulkan asset picks on no physical NVIDIA
* Update RAG auto-backend test mocks for the _resolve_auto binary and Vulkan probes
* Keep the add_dll_directory handle alive through the Vulkan probe DLL loads
* Revert RAG auto Vulkan guard, guard multi-backend Vulkan detection, and preserve forced Vulkan across updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use getattr for RTLD_GLOBAL in the Vulkan probe CDLL mode
* Skip CUDA/ROCm APU and datacenter GPU tuning on Vulkan builds
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not
CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and
_apply_datacenter_env were reading the wrong device. On a mixed AMD APU
plus discrete GPU host that could raise a spurious system-RAM shortfall
and block a valid discrete-GPU load. Gate all three call sites on
not is_vulkan_backend; the Vulkan path already reserves iGPU host
headroom and the backend ignores GGML_CUDA_* anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Vulkan-guard comment in load_model
* Reduce comments in Vulkan support to be more succinct
* Resolve shell-wrapper llama-server entrypoint to the real lib dir
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install
root when it cannot symlink into build/bin. _find_llama_server_binary
returns that root entrypoint, but Path.resolve() does not follow a shell
wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend
missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device
pin on an otherwise valid Vulkan install. Follow the wrapper's exec target
to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Tighten the eval/exec/compile dynamic policy so an executing sink (eval/exec/
runpy) applied to a payload that cannot be statically recovered is refused
unconditionally, not only when an RCE-core module happens to be imported in the
snippet. An un-analyzable executing payload can synthesize any shell, network,
or filesystem escape at runtime, so the prior in-scope-import heuristic left
exec(input()) and eval(user_var) allowed whenever no such import was present.
compile() of the same payload stays allowed since it does not run.
A payload that is fully recovered as a constant but is invalid Python for the
sink's mode (for example eval("data = 1") or eval("not python !!")) is now
allowed: its exact source is known and it raises SyntaxError at runtime, so it
is not an execution vector. Only genuinely opaque, non-recoverable payloads
(for example eval of a runtime-computed f-string) reach the block.
Remove the now-unused _RCE_CORE_MODULES set and _scope_imported_roots helper,
and move the opaque-f-string case in the tests to the blocked set.
Add ln to the bash command denylist so a symlink escape cannot be created from
the terminal tool. In the sandboxed (non-bypass) _python_exec path, prepend a
one-line guard to the generated temp module that monkeypatches only MUTATING file
ops (builtins.open in write/append/x/+ modes, os remove/unlink/rmdir/removedirs/
rename/renames/replace/truncate/chmod/chown/mkdir/makedirs/symlink/link, shutil
rmtree/move/copy/copy2/copyfile/copytree, pathlib write_text/write_bytes/unlink/
rename/replace/mkdir/rmdir/chmod/symlink_to/hardlink_to/touch) to resolve the true
os.path.realpath of the target and raise PermissionError unless it lands inside
the injected session workdir. Reads are left unpatched. The guard runs in its own
namespace so helper names never leak into user globals, and it is skipped
entirely under disable_sandbox. This catches what the static gate cannot prove:
pre-existing symlink escapes and dynamic library-writer paths that funnel through
builtins.open. Benign in-workdir relative writes and library imports are
unaffected (importlib swallows out-of-workdir bytecode-cache write failures).
Add pragmatic aliasing so an aliased shell sink with a dangerous argument is
caught: a name stored exactly once and bound to a resolved os/subprocess sink
(s = os.system; s('rm -rf /')) and inline literal-container indexing
([os.system][0](...), (os.system,)[0](...), {'k': os.system}['k'](...)) both feed
the existing _find_blocked_commands argument check. Resolution is deliberately
low-false-positive: only unambiguous single assignments and inline literal
containers, never a flow-insensitive union, so s = os.system; s = print; s('hi')
is not aliased. The shell-sink set is lifted to module scope (_SHELL_SINK_FUNCS)
so the alias pre-pass and the visitor share one definition. Interprocedural and
flow-sensitive taint remain out of scope (deferred to a full fixpoint).
Add a filesystem_violations category backed by _resolve_path, a LOCAL / ESCAPE /
UNKNOWN classifier that constant-folds strings and understands os.path.join,
pathlib Path()/'/'/joinpath, and f-strings with real join plus absolute-reset
semantics. expanduser / expandvars / os.environ / getcwd / dynamic parts collapse
to UNKNOWN. A new _FilesystemPolicyVisitor inventories destructive and mutating
ops (open write/append/x/+, os remove/unlink/rmdir/rename/replace/truncate/chmod/
chown/mkdir/makedirs/mknod/symlink/link/chdir, shutil rmtree/move/copy*, pathlib
write_text/write_bytes/unlink/rename/replace/mkdir/rmdir/chmod/symlink_to/touch,
tempfile dir=, and a curated numpy/pandas/torch/joblib/PIL/matplotlib/cv2 writer
set) and applies prove-or-block: mutating LOCAL allows, UNKNOWN/ESCAPE blocks.
rename/move check src and dst; symlink/link check both target and link path;
chdir must be LOCAL; tempfile dir= must be LOCAL. Reads block only on a provable
escape (sensitive absolute path or ..'/~ traversal), with an FS_READ_STRICT knob
for prove-or-block reads. A callee-independent literal-sensitive-path scan blocks
loaders like pandas.read_csv('/etc/shadow'). Library writers block only on a
provable escape so in-memory buffers are not over-blocked; the Stage 5 runtime
backstop covers the dynamic residual.
Replace the blanket dynamic_exec block with a recursive payload analyzer gated by
UNSLOTH_STUDIO_SINK_ANALYZER (default on; =0 restores the legacy ban). For eval /
exec / compile (and single-assignment aliases like e = exec), constant-fold the
first argument; a recovered source string is bracket-depth pre-scanned, size and
recursion-depth bounded, then re-classified through the full analyzer. An inner
sink blocks and surfaces the inner reason; a clean inner payload allows; a bound
or budget breach fails closed. Non-foldable payloads follow a low-false-positive
dynamic policy: block when assembled from decode / fetch / runtime-assembly
primitives (including nested exec and large string repetition) or when an
RCE-core module is imported in scope, else allow.
Keep the gadget-dunder and dynamic-import blocks but constant-fold import names
so __import__('hugging'+'face_hub') resolves to a real module. Refine getattr /
setattr on a sensitive module so a benign constant attribute (getattr(os,
'getpid')) is allowed while a dynamic or dangerous constant attribute blocks. Add
pickle/marshal/dill.loads as unverifiable code-deserialization sinks. eval('2+2'),
compile('a+b','<s>','eval') and ast.literal_eval now pass; base64/hex/rot13/chr
and gadget-obfuscated escapes still block. Wire a filesystem_violations category
through is_safe, the info dict, and the reason assembly (populated in a later
stage). The three legacy tests that asserted the blanket ban are updated to the
new recurse-the-payload behavior.
Introduce _const_fold, a whitelist-only, bounded, side-effect-free partial
evaluator plus a single-assignment const-prop environment builder. It recomputes
pure transforms on literals only (concat, repeat, join, format, f-strings,
slice/reverse, chr/ord, base64/hex/rot13/zlib decode, pure builtins and string
methods) and never executes, imports, or reflects on user code. Depth, op, size,
and sequence caps guarantee it can only fail to recover a value, never crash or
hang. This is the foundation the later eval/exec unwrapping and filesystem path
resolver build on.
* Studio: render thinking blocks for safetensors inference with prefilled <think> templates
Reasoning templates like Qwen3.6 end the generation prompt with an open
<think> tag. skip_prompt streaming drops it, so the frontend never sees
the opening tag and shows reasoning as plain text. Detect the prefill
and re-emit it at the start of the stream on the transformers and MLX
paths. Also stop stripping think tags in _clean_generated_text when a
tokenizer marks them special.
* Studio: guard think re-emit for special close tags, yield prefill early
Address review feedback:
- Guard: skip re-emitting the open <think> when the tokenizer marks </think>
as a special token, since skip_special_tokens would strip the model's close
tag and leave an unclosed block that swallows the answer. Falls back to
plain text (pre-fix behaviour) for those tokenizers.
- Yield the prefilled <think> before the first token so the thinking block
renders during prompt prefill instead of after the first generated token.
- Drop the now-unnecessary _clean_generated_text think-tag exemption; the
guard handles the special-token case at the source.
No mainstream reasoning model (Qwen3.6, Qwen3, DeepSeek-R1, QwQ, GLM-4.6)
marks think tags special, so behaviour is unchanged for them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lyxot <longyixing331@gmail.com>
* Studio: allow CPU-only DiffusionGemma by granting the diffusion runner the CPU device
* Studio: mark CPU-only DiffusionGemma as non-GPU-resident for training VRAM preflight
* Studio: keep the CPU DiffusionGemma change minimal (revert VRAM-flag tweak; Metal hosts still hold unified memory)
* Studio: keep CPU DiffusionGemma fallback fully CPU-masked so a masked GPU host does not re-expose GPU 0
The python tool's static safety analysis (_check_signal_escape_patterns) was
purely name-based with no attribute visitor, so obfuscated routes to the shell
/ network / file policies it already enforces slipped past: eval / exec /
compile, __import__ / importlib with a computed or dangerous module name,
getattr / setattr aimed at os / subprocess / sys / builtins, and dunder gadget
chains (().__class__.__bases__[0].__subclasses__()).
Add a dynamic_exec category covering those, surfaced through _check_code_safety
alongside the existing categories. Dynamic import stays allowed for a benign
literal module name (huggingface_hub, json, numpy) so real workflows and the HF
upload gate keep working; ordinary getattr(obj, "field") and __class__ access
stay benign. Bypass Permissions (disable_sandbox) still skips the check.
Tests: TestDynamicExecObfuscation in test_sandbox_tools.py with matching benign
cases, giving _check_signal_escape_patterns its first direct coverage.
* feat(cli): detect MLX distributed launch context
* feat(mlx): wire distributed inference backend
* feat(cli): broadcast MLX distributed chat turns
* fix(cli): wait indefinitely for distributed chat turns
* fix(cli): report MLX distributed load errors cleanly
* fix(mlx): route distributed vlm through loader
* fix(cli): detect inline MLX host JSON
* fix(studio): harden distributed object sharing
* fix(studio): select JACCL distributed backend
* fix(cli): abort distributed error paths
* Distinguish real stream errors from model text via GenStreamError in distributed CLI
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail loud when MLX distributed init returns a singleton group
The worker only reaches this block when distributed was explicitly
requested. A singleton (size 1) group means the launch failed to form a
real group (MLX built without distributed support, or an invalid launch
env/hostfile); silently continuing leaves nonzero ranks looping forever
on share_distributed_object. Raise instead so the surrounding handler
returns a clear load error.
* Tighten MLX distributed inference comments
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: handle case-variant GGUF cache hits for unsloth start
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gguf cache: keep split shards co-located and isolate cache tests properly
When a cached main shard was reused from an older snapshot, the extra shards
were resolved independently and could come from a different snapshot dir (or a
fresh download into the current ref), leaving llama.cpp unable to load a
multi-shard GGUF whose pieces are split across directories. Only reuse a cached
main shard when every sibling shard sits in the same snapshot; otherwise fetch
the whole set together so they stay co-located.
Also patch huggingface_hub.constants.HF_HUB_CACHE (not just the HF_HUB_CACHE env
var) in the two cache tests that seeded a temp cache: the snapshot lookup reads
the module constant, so the env-only override let the real cache leak in and
skip an asserted download.
* Do not let a companion-only cache snapshot shadow real GGUF variants
When listing GGUF variants from the local HF cache, a newer snapshot may
contain only a companion file (for example a vision projector fetched on
demand) while the actual quant files live in an older snapshot. The prior
scan returned the first snapshot whose vision flag was set, yielding an
empty variant list and hiding the real quants. Keep scanning older
snapshots for actual variants and carry the vision flag across snapshots.
Also record the disk-space fallback variant's size in expected_sizes so
the later cache-reuse probe can size-verify the fallback main shard
instead of only checking for its existence.
* Propagate cached repo casing to companions and preflight split co-location
Two fixes to the case-variant GGUF cache reuse:
- Resolve the requested repo id to its cached canonical casing once in
load_model, up front, and pass it to the main GGUF and its companions
(mmproj / MTP drafter). Previously only _download_gguf resolved the
casing internally, so a case-variant request loaded the main file from
the canonical cache dir while the companions kept the requested casing
and missed the cached vision projector / drafter offline. Extracted the
resolution into a shared _resolve_repo_id_casing helper.
- Apply the split-shard co-location check in the disk-space preflight. When
a split GGUF's shards are cached across different snapshots the whole set
is refetched later, so counting them as cached made the preflight read 0
bytes to download, skip the smaller-variant fallback, and then fail the
full download on a low-disk machine.
* Reuse a co-located split GGUF snapshot and fix split fallback size probe
- When reusing a cached split GGUF, scan snapshots for one that holds the
whole set co-located instead of taking the newest snapshot's first shard.
A newer snapshot with only the first shard no longer shadows an older
complete snapshot, so an already-cached split model is reused rather than
refetched (which would fail offline).
- The disk-space fallback records its size in expected_sizes only for a
single-file fallback. _find_smallest_fitting_variant returns the whole
variant size, so using it as the first shard's expected size rejected a
valid cached first shard of a split fallback and forced a re-download.
* Scan for a complete split snapshot in the preflight; require a loaded catalog hit
- The disk-space preflight now uses the same co-located snapshot scan as the
download path (_cached_colocated_split_main) instead of the newest-snapshot
probe, so a newer snapshot holding only the first shard no longer masks an
older complete one and trips the smaller-variant fallback for a fully cached
split model.
- _resolve_model only attaches to a /v1/models entry that is actually loaded
(loaded != False). /v1/models also lists cached-but-unloaded catalog entries,
and matching one by case skipped /api/inference/load and left the agent
pointed at a model that is not resident.
* Restrict cross-snapshot GGUF cache reuse to offline
Reusing a same-name blob from an older or case-variant snapshot bypasses the
Hub revision/etag check, so a repo that updates a GGUF in place could serve
stale weights online. Gate the cross-snapshot and case-variant reuse (both the
disk-space preflight accounting and the download path) on HF_HUB_OFFLINE.
Online, hf_hub_download fetches the current revision and resumes a partial
download, so the reuse is unnecessary there; offline it remains the resilience
fallback. Marked the two reuse regression tests as the offline scenarios they
represent and added an online test asserting a fresh fetch.
* Harden offline cache reuse and hub-id detection
Three follow-ups on the case-variant GGUF cache path:
- Honor every truthy HF_HUB_OFFLINE spelling (1/true/yes/on), not just "1", when
gating the cross-snapshot and case-variant cache reuse. With HF_HUB_OFFLINE=true
the Hub calls are already offline, so the reuse must trigger or the cached GGUF
fails to load; route both the preflight accounting and the download path through
the same offline parse the rest of the backend uses.
- Resolve mmproj/MTP companions from the actual cached snapshot when offline.
resolve_cached_repo_id_case can keep a partial lower-case spelling when any dir
exists under the requested casing, so an hf_hub_download on that casing misses the
canonical companion; scan every case-variant snapshot and return the cached path.
- Restrict the case-insensitive model-id match to syntactically valid hub ids
(a single namespace/name over the HF charset). A server-side relative path such
as models/Llama/Foo.gguf is no longer treated as a hub id, so it cannot
casefold-match a differently cased path on a case-sensitive filesystem. This is
host independent, unlike the local-existence probe which cannot see a server path.
* Only casefold-match model ids against a loopback Studio
A two-segment string like Models/Foo is indistinguishable from a hub id, and the
local Path.exists() probe in _is_hub_model_id cannot see a path that exists only
on a remote Studio host. So against a remote server, casefolding could attach to
a distinct server-side path (Models/Foo vs models/foo) on a case-sensitive
filesystem. Gate the case-insensitive match on is_loopback_url(base): only a
local Studio, where the existence probe is authoritative, casefolds. For a remote
Studio the match is exact and a case-mismatched request falls through to
/api/inference/load, whose already-loaded dedup resolves it correctly.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Studio: heal DiffusionGemma tool calls into structured tool_calls
* Fall back to supports_tools for backends without the passthrough capability
* Route DiffusionGemma client tools through passthrough when enable_tools is on
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop orphaned strip_tool_call_markup import after syncing with main
* Tighten supports_tool_passthrough comment
* Re-run CI on current main
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Speed up Studio startup path
* Studio: recheck managed binary executability on preflight cache hit and ignore stale unauthenticated platform fetches
Preflight: a matching capability cache fingerprint no longer skips the
runnability check when the managed binary's executable bit was cleared
(size and mtime unchanged, since chmod bumps ctime not mtime). The cache
fast path now confirms the binary is still executable, otherwise it falls
back to the CLI help probe so preflight reports Stale and can repair,
instead of returning Ready and failing later at backend start. Adds a
regression test.
Frontend: now that first render is no longer gated on fetchDeviceType,
the initial unauthenticated health call can resolve after an
authenticated platform fetch. Guard the store so a late unauthenticated
or failed non-forced response cannot overwrite an already authoritative
device type, tunnel URL, or secure flag. Forced refreshes and the first
unauthenticated load are unaffected.
* Studio: use access(X_OK) for the preflight cache executability guard
A mode bitmask treats any execute bit as launchable, but the executable
bits can be set only for another owner or group, or be denied by an ACL,
so the current user could still hit PermissionDenied at launch and the
cached fast path would wrongly return Ready. access(X_OK) checks real
executability for the calling user, so an ownership or permission change
correctly falls back to the CLI help probe and the Stale repair path.
* Studio: ignore any stale non-forced platform fetch once authoritative
Extend the platform store guard so a non-forced health response never
overwrites an already authoritative result, not only unauthenticated
ones. With a saved token the post-render non-forced request can be
authenticated but older than a later forced refresh that already picked
up the tunnel URL and secure flag; if that earlier request resolves last
it would null those fields. Now any non-forced response is dropped once
the store holds a server-reported platform. Forced refreshes and the
first authoritative write are unaffected.
* Studio: run the managed CLI help probe before trusting the preflight cache
Restore running the managed CLI help probe before returning Ready from
the desktop capability cache, so a managed install whose venv interpreter
or a runtime dependency is broken (while path, size, mtime, and markers
are unchanged) is reported Stale for repair rather than proceeding to a
backend start that cannot spawn. The capability cache still skips the
heavier desktop-capabilities probe on a hit, so a warm cache runs one
probe instead of two. Removes the executable-access shortcut, which the
help probe now subsumes.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: account for DeepSeek-V4 compute buffer in context auto-fit
DeepSeek-V4-Flash's lightning indexer plus compressed sparse attention reserve a
large context-scaling compute buffer that _compute_buffer_ctx_bytes did not model
(the KQ-mask and dequant-scratch rates both miss it, even with an f16 cache).
Measured on UD-Q4_K_XL at ub 512 it is about 65.5 GiB at 1M context, which the
mask estimate puts near 1.5 GiB, so the auto-fit kept the full 1M train context
and llama-server OOM'd allocating the ~70 GB buffer, then spilled to CPU (~4
tok/s). Add a deepseek4-gated flat plus per-token term so the fit caps the context
(about 256k on a B200) and the model stays fully on GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add DeepSeek-V4-Flash-GGUF to Studio with none/high/max reasoning
Adds unsloth/DeepSeek-V4-Flash-GGUF as a default selectable model with the
recommended decoding defaults (temperature 1.0, top_p 1.0 from the official
generation_config.json) and its three tier reasoning control. The high/max
ladder is surfaced for deepseek-v4 model ids and flows through the existing
enable_thinking_effort reasoning style via chat_template_kwargs, so no
frontend changes are needed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio DeepSeek-V4: segment-scope high, enable thinking for lone effort, render tests
Match deepseek-v4 on whole repo-name segments so a future deepseek-v40 or
deepseek40 cannot false-match the synthetic 'high'. In _request_reasoning_kwargs,
emit enable_thinking when a named effort level is sent without it, so the
newly exposed High mode renders thinking-on over the API (the UI already sent
it explicitly). Add a none/high/max render-path test file (jinja behind
importorskip) with a lone-high regression.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: apply presence_penalty on the safetensors and MLX inference paths
The safetensors and MLX generate paths resolved the inference config and
then dropped presence_penalty before generation, so the same model applied
the configured value under GGUF and 0 under safetensors/MLX. Thread the
already-resolved presence_penalty through the orchestrator command, worker
gen_kwargs, and the safetensors/MLX generate calls, and apply it with a
small logits processor (subtract once per distinct completion token,
prompt excluded, presence not frequency, zero is a no-op, negatives raise).
Backwards compatible: presence_penalty defaults to 0.0 (byte-identical
output when unset) and the GGUF path is unchanged. Also forward min_p on
the legacy /generate/stream route and add the missing min_p field to
GenerateRequest.
* Studio: bound presence_penalty generated ids to valid vocab range on both paths
The presence-penalty logits processors index by generated token ids. The
torch path filtered only the upper bound (seen < vocab_size), so a negative
id would silently wrap to the wrong row; the MLX path had no bound at all,
and MLX out-of-bounds indexing is documented undefined behavior (crash or
memory corruption on Apple Silicon), unlike torch's harmless negative wrap.
Bound generated ids to [0, vocab) consistently on both paths:
- torch: seen[(seen >= 0) & (seen < vocab_size)] (zero-regression safety net;
real completion tokens are always in range).
- MLX: route out-of-range/negative ids to a discarded scratch slot via
mx.where and a (vocab + 1)-wide scatter-assign mask, then subtract. MLX has
no boolean-mask filtering (data-dependent output shape), so this keeps a
fixed shape, stays on-device, and preserves once-per-distinct-token
semantics without any torch/numpy dependency.
Add torch tests for out-of-range and negative ids (only in-range distinct
ids penalized, stray ids ignored, no wrong-index wrap) and a bound-documenting
MLX test that runs on the arm64 macOS CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: serialize the compare-mode dispatcher lifecycle to fix a start race
_generate_dispatched (compare mode) bypasses _gen_lock so two concurrent
compare requests can both reach _start_dispatcher. The check-then-spawn there
had no lock, so both could observe no live dispatcher and each spawn one. The
extra dispatcher is orphaned (self._dispatcher_thread tracks only the last) and
during a later unload it can consume the 'unloaded' reply off _resp_queue before
unload_model's _wait_response, hanging the unload on its timeout.
Add _dispatcher_lifecycle_lock and take it around the whole body of both
_start_dispatcher and _stop_dispatcher, so start/stop cannot interleave and the
second concurrent starter sees the dispatcher alive and returns. _start_dispatcher
now returns whether it actually spawned the thread, and _generate_dispatched
derives dispatcher_preexisting from that atomic result instead of a separate
unlocked is_alive() read.
No call site holds _mailbox_lock when calling start/stop, so joining the
dispatcher (which takes _mailbox_lock) under the new lock cannot deadlock; the
lock order is always _gen_lock then _dispatcher_lifecycle_lock and is never
inverted.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refuse dispatcher start queued behind an unload's stop
A compare request could pass the early _unload_pending check, then block in _start_dispatcher on _dispatcher_lifecycle_lock behind an unload's _stop_dispatcher. When the unload released the lock the start spawned a fresh dispatcher, which became the resp_queue reader and consumed the worker's unroutable 'unloaded' reply before unload_model's _wait_response saw it, hanging the unload for 300s.
Gate _start_dispatcher on _unload_pending under the lifecycle lock, and set _unload_pending under the same lock ahead of the stop, so any start queued behind the stop observes the unload and refuses. Ordering stays _gen_lock -> _dispatcher_lifecycle_lock. Adds a regression test forcing the queued-behind-stop interleaving.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fix six race conditions when a user switches or cancels a model while a
previous load or generation is still in flight, across the inference
orchestrator and the /load and /unload routes:
- Cancel an in-flight generation on a safetensors/MLX model switch and
serialize unload with load under the inference lifecycle gate.
- Cancel an in-flight load off the lifecycle gate so a Stop-loading
cancel does not wait out the multi-minute load; guard the dispatched
mailbox against a racing unload.
- Recheck the loading marker after spawn and again after the load
response before publishing, so a load cancelled mid-flight is reaped
instead of going live.
- Discard the loading marker before tearing the subprocess down in
cancel_load, closing a spawn-after-cancel window and an orphaned
compare-mode dispatcher during unload.
- Match the unload target before canceling an in-flight GGUF load and
add an off-gate fast path for the still-loading GGUF case.
- Run the Unsloth unload off the event loop so a paused SSE stream
holding _gen_lock cannot block the loop.
Adds studio/backend/tests/test_orchestrator_unload_cancel.py covering
the unload/cancel/switch race paths.
* Studio chat: tool-call nudging on by default (API stays opt-in)
Healing is already default-on everywhere and the nudge retry from the
client-tool passthrough is opt-in on the API. Studio chat had neither
signal: the frontend never sent nudge_tool_calls, and the safetensors
and MLX server-side loop lacked the GGUF loop's plan-without-action
re-prompt entirely.
Backend: the re-prompt helpers move from llama_cpp.py into
tool_call_parser.py (shared, cycle-free; the GGUF loop imports them
under its old names with zero behavior change) and
run_safetensors_tool_loop now re-prompts once at the streaming
no-tool-call exit, gated on Auto-Heal, active tools, nothing executed
yet, and short forward-looking text. Re-prompts do not consume tool
iterations.
Frontend: the chat adapter sends nudge_tool_calls from a new
nudgeToolCalls runtime setting (default true) with the same
persistence, hydration, and settings toggle plumbing as Auto-Heal.
Request-model defaults are untouched, so raw API callers stay opt-in.
* Address review: persist the nudge setting, consume the flag in the loops, skip the re-prompt after RAG autoinject
ChatSettingsPayload uses extra forbid, so a settings patch containing
nudgeToolCalls failed to persist any settings; the field is now typed
and round-trips. nudge_tool_calls now plumbs into both server-side tool
loops and gates the plan-without-action re-prompt with None meaning on,
so API callers keep today's behavior, explicit false disables it, and
Studio's default-on flag actually controls the path Studio chat runs.
The safetensors loop no longer re-prompts after RAG autoinject: the
injected retrieval bypasses the tool controller, so the nothing-executed
gate saw an empty history and re-asked after a successful retrieval.
* Safetensors loop: the plan-without-action retry requires an explicit nudge flag
The retry is new on this loop, so an omitted nudge_tool_calls must not
change existing API behavior; Studio opts in explicitly. The GGUF loop
keeps None as on because its re-prompt predates the flag.
* Suppress the plan-without-action re-prompt after a denied tool confirmation
A denial appends TOOL_REJECTED_MESSAGE but records nothing in the tool
controller history, so the nothing-executed gate re-prompted the model
to call the tool the user had just rejected, producing another
confirmation prompt. A denial now suppresses the re-prompt for the rest
of the request, mirroring the RAG autoinject handling.
* Tighten plan-without-action re-prompt comments
* Tighten plan-without-action re-prompt comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: match unified plan-without-action nudge cap to GGUF default of 3
The shared MAX_ACT_REPROMPTS was set to 1, but GGUF's established default
(llama_cpp.py) has re-prompted a stalling model up to 3 times since #5620.
Restore the GGUF-matched cap so safetensors and MLX inherit the same
behavior, and update the safetensors cap test to assert the cap dynamically.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
