Round 19 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Record a global/nonlocal sink alias in its TARGET scope (module for global, enclosing for
nonlocal): global s; s = os.system; s('...') now resolves s to the shell sink instead of
being skipped as a rebound name (and a global name no longer shadows an outer alias).
- Fail closed on any expansion in COMMAND POSITION, not just command substitution: a
variable-expanded command word (p=python3; $p -c ...) / ${VAR} is unprovable. Argument
position ($HOME, echo $(date)) stays allowed.
- Honor wrapper/command-position logic in the mutating-utility and shell-script scans via a
shared command-word index, so env sed -i / timeout 5 bash s.sh no longer hide behind a
wrapper prefix.
- Use the subprocess-exec callee resolver for the shell=True read scan, so
from subprocess import run as r; r('head /etc/passwd', shell=True) and r = subprocess.run
aliases are tokenized.
- Resolve sys.modules aliases (m = sys.modules; m.pop('_io')) for the loader-table mutation
checks (bound and unbound forms).
- Treat a __dict__ subscript keyed by a gadget dunder (type(open).__dict__['__closure__'].
__get__(open)) as gadget access, closing the descriptor-lookup route around the attribute
gadget scan.
- Unwrap inline literal containers in the higher-order sink check, so
list(map([eval][0], [...])) / map({'e': exec}['e'], ...) are flagged.
- Runtime: the mutator guard now accepts the path via its public keyword (os.makedirs(name=),
os.mkdir(path=)) instead of raising TypeError, while still confining the write.
Adds TestRound19Bypasses and keyword-path runtime tests; full sandbox suite green.
Round 18 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Fail closed on a command-position command substitution ($(printf touch) / `printf touch`
as the command word): the expansion becomes the command name and cannot be proven safe. An
argument-position substitution (echo $(date), x=$(cmd)) stays allowed.
- Model direct imports of the os C backend (posix/nt -> os aliases) and pty: posix.system(...)
resolves as an os shell sink and pty.spawn(...)/pty.fork() is flagged as an unguarded child.
- Reject unbound sys.modules mutation (dict.pop(sys.modules, '_io'),
type(sys.modules).__delitem__(sys.modules, ...)) alongside the bound sys.modules.pop form.
- Treat mutating flags of normally read-only utilities as child writers: sed -i, sort -o FILE,
find ... -delete, dd of=FILE, tee FILE, truncate. Non-mutating uses stay allowed.
- Detect os / subprocess re-exported through a stdlib module (pathlib.os.system,
tempfile.os.system, subprocess.os.system): the .os attribute IS the os module.
- Resolve instance-attribute sink aliases (c.e = exec; c.e(payload) / obj.s = os.system;
obj.s('rm -rf /')) tree-wide, alongside the existing class-attribute aliases.
- Import the guard's remaining pure-Python dep (shutil) with the workdir still stripped from
sys.path; the restore now runs at the very end of the prelude, so no workdir/shutil.py can
shadow it.
- Network policy: resolve import aliases (import requests as r; r.get(...)) and inspect the
url= / address= keyword arguments so aliased or keyword-host calls to a metadata / untrusted
host are no longer skipped.
Adds TestRound18Bypasses and extends the workdir-shadowing runtime test to shutil; full
sandbox suite green.
Round 17 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Track a code object built through a LOCAL compile alias (cfn = compile; co = cfn(src);
types.FunctionType(co, {})()) by resolving the callee through the scope exec-builtin map,
so the FunctionType execution gadget still gets the recursive payload analysis.
- Recognize type(lambda: None) as the function constructor: type(lambda: None) IS
types.FunctionType, so type(lambda: None)(code, {})() executed a compile() code object
without the eval/exec gate.
- Include lambda and comprehension scopes in the alias index: (lambda e=exec: e(payload))()
and [e(payload) for e in [exec]] now resolve e back to the exec sink. Lambdas/comprehensions
become their own alias scopes, and a one-element comprehension generator binds its target.
- Record annotated single-assignment aliases (e: object = exec; e(payload)) alongside plain
assignments, so the AnnAssign RHS is analyzed.
- Expand ${IFS} / $IFS to whitespace before shell command matching, so a separator-obfuscated
writer/reader (rm${IFS}-rf${IFS}/, cat${IFS}/etc/shadow) is tokenized as bash runs it.
- Import the child-guard's stdlib deps (os/io/pathlib/re) with the workdir stripped from
sys.path, then restore it, so a malicious workdir/os.py or pathlib.py cannot shadow a guard
import and run unguarded at import time.
- Fail closed on shell redirection to any real-file target (the unguarded child follows a
pre-existing symlink); only fd duplications (>&2) and the standard device sinks
(/dev/null, ...) are allowed. This also fixes a pre-existing false positive where a benign
redirect to /dev/null was blocked.
Adds TestRound17Bypasses and a workdir-shadowing runtime test; full sandbox suite green.
Round 16 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Apply the runtime sensitive-read backstop to Path.open reads (not just write
modes), so a dynamically assembled pathlib receiver (Path(globals()['P']).read_text())
cannot exfiltrate a host secret. On Python <= 3.11 pathlib holds the original io.open,
so confining at the public Path.open level is the version-robust fix. read_text /
read_bytes route through the same self.open() and are covered.
- Scan the -c payload of a subprocess shell argv for sensitive reads. subprocess.run(
['sh', '-c', 'head -1 /etc/passwd']) has no blocked command, but the unguarded child
prints the secret, so the payload is now tokenized and read-scanned like a string sink.
- Normalize bash ANSI-C ($'...') and locale ($"...") quoting before command matching.
shlex leaves $'touch' as the literal $touch, so a writer / interpreter hidden behind
ANSI-C quoting ($'touch' x, $'\x74ouch' x) previously evaded the command blocklist;
the escapes bash resolves (\n, \xHH, octal, \uHHHH) are decoded first.
- Recognize from-imported subprocess exec names as read sinks: from subprocess import
run as r; r(['cat', '../../etc/shadow']) now hits the traversal / sensitive-read check.
- Treat a shell glob that can expand outside the workdir (absolute / ~ rooted, e.g.
head /etc/shad*) as an escaping read expansion and fail closed, for reader arguments
and input redirects. A relative in-workdir glob (grep foo *.txt) stays allowed.
Adds TestRound16Bypasses and pathlib runtime backstop tests; full sandbox suite green.
Round 15 review follow-ups on the Studio code-exec sandbox classifier and runtime guard:
- Resolve single-assignment aliases for shutil.copy* and subprocess exec read
sinks (c = shutil.copy; c('../../etc/passwd', ...); r = subprocess.run; r([...]))
so the traversal / sensitive-path check fires on the aliased callee.
- Block gc.get_referents / get_referrers / get_objects (and from-import aliases):
they walk the object graph to a guarded wrapper's closure cell to recover the
original unguarded open / os.* callable.
- Cap list / tuple concatenation during constant folding so a doubling chain
(a + a + a + ...) cannot materialize an oversized sequence in the parent process
before the child rlimits apply.
- Add a runtime sensitive-read backstop in the child prelude: deny a read whose
realpath resolves to a known host secret (SSH / cloud / kube / netrc / HF-token /
/etc/passwd family / /proc) outside the workdir. This covers opaque read paths the
static scanner cannot fold (open(globals()['x'])) and pre-existing in-workdir
symlinks to secrets, while leaving benign outside reads and library imports intact.
- Fail closed on relative multi-component shell redirect targets (echo x > sub/out.txt)
whose subdirectory component could be a symlink traversing outside the workdir; a
bare single-component target stays allowed.
Adds TestRound15Bypasses and runtime backstop tests; full sandbox suite green.
Static classifier:
- Flag mro().pop(i) / __mro__.pop(i) base-class extraction alongside the subscript and __getitem__ forms.
- Normalize operator.attrgetter('name')(obj) as attribute access whether or not the result is immediately invoked, so attrgetter('__closure__')(open)[0] gadget recovery is caught.
- Resolve a container-hidden open alias (o = [open][0]; o('../../etc/passwd').read()) as a read callee.
- Fail closed on an opaque read path assembled from obfuscation primitives (open(''.join(map(chr, ...))).read()), matching the exec-payload obfuscation policy.
- Treat cd behind the command / builtin shell wrappers as a cwd escape before allowing a relative redirect.
- Block importlib file loaders as execution sinks (SourceFileLoader(...).load_module(), spec.loader.exec_module(...)).
- Add the in-cluster Kubernetes service-account credential path to the sensitive-read list.
Adds TestRound12Bypasses covering each vector plus benign controls.
Static classifier:
- Flag __getattribute__/__getattr__ with a non-foldable (runtime-assembled) attribute name as an obfuscated gadget access.
- Recognize POSIX noclobber redirect targets (>| path) in the shell redirect scanner.
- Detect a container-wrapped compile() code object passed to types.FunctionType.
- Flag mro().__getitem__(i) / __mro__.__getitem__(i) base-class extraction alongside the subscript form.
- Scan shell command strings (os.system / subprocess shell=True / getoutput) for embedded sensitive-file reads.
- Treat a default-parameter value that is a dangerous callable (def f(e=exec)) as a sink alias.
- Normalize a trailing .__call__ for shell/import/deserializer sinks (os.system.__call__, __import__.__call__, pickle.loads.__call__).
- Resolve a container-unwrapped sink assigned first (s = [os.system][0]; s(...)).
- Look through no-op pathlib methods (resolve/absolute/expanduser) when resolving a read receiver.
- Recognize from-imported shutil copy sinks (from shutil import copy as c).
- Extend the higher-order first-class-value check to shell and deserializer sinks (map(os.system, ...), partial(subprocess.getoutput, ...), map(pickle.loads, ...)).
- Normalize operator.attrgetter('name')(obj) as attribute-access obfuscation.
Adds TestRound11Bypasses covering each vector plus benign controls.
Static classifier:
- Flag mro()[i] base-class extraction alongside subscripted __mro__ (FileIO C base recovery).
- Fail closed on non-literal shell redirect targets and cd to an outside directory.
- Block sys.modules mutating methods (pop/popitem/clear/setdefault/update) that drop a guarded module for reimport.
- Detect indirect eval/exec: <builtin>.__call__(payload) and eval/exec/compile passed by reference to a higher-order call (map/reduce/partial), including starred literals.
- Block inspect.getclosurevars() closure recovery of a guarded wrapper.
- Track runpy run_path/run_module from-import aliases.
- Expand starred literal path arguments in the sensitive-read scanner.
- Resolve container-hidden deserializers (([pickle.loads][0])(payload)).
Runtime backstop:
- Guard the low-level posix/nt chdir (cwd escape) and fchmod/fchown fd metadata mutators, matching the os.* deniers.
Adds TestRound10Bypasses and low-level posix runtime tests.
Static classifier:
- deny sys.modules Store/Del: del sys.modules['posix']; import posix drops the guard-patched module for a fresh unwrapped C module
- flag code.InteractiveInterpreter().runcode / InteractiveConsole().runsource as code-object execution sinks (opaque compile results run un-analyzed)
- resolve indirect read-only open callees: from os/io/builtins import open as X (os.open read-only is deliberately allowed outside the workdir, so traversal must be caught statically)
- fold os.path.normpath / abspath on literals so a traversal that only emerges after normalization is scanned
- resolve inline-container-hidden exec/eval: ({'e': exec}['e'])(...) / [exec][0](...)
- normalize the bound one-arg __getattribute__ form obj.__getattribute__('name') (builtins.open.__getattribute__('__closure__'))
- descend into a literal list/tuple argv so subprocess.run(['cat', '/etc/passwd']) is caught
- enforce the analyzer node budget: charge each tree's node count against _MAX_ANALYZER_NODES and fail closed above it (parent-process DoS guard)
Runtime backstop:
- capture and re-pin os.lstat / os.readlink / os.getcwd / os.stat before realpath, since it consults the live symlink helpers -- monkeypatching os.lstat to fail could stop realpath following an in-workdir symlink that points outside
Static classifier:
- flag cell_contents (the only closure-cell reader) as a gadget so recovering a guarded wrapper's original callable via __closure__ fails closed even when the __closure__ name is built at runtime
- treat a class body as its own alias scope (class C: e = eval; e(...) now recognized) while keeping methods lexically skipping the class scope, so a same-named class attr does not shadow the module-level sink a method reaches
- block output redirection (> / >> / &> / N>) to an absolute / ~ / .. target: a child shell runs unguarded; relative in-workdir redirects stay allowed
- resolve pathlib constructor import aliases (from pathlib import Path as P) before traversal reads
- flag an integer-indexed __mro__ (io.FileIO.__mro__[1]) that extracts the unguarded FileIO C base class; plain iteration / slicing stays allowed
- resolve aliased read callees before traversal checks: o = open and import shutil as sh; sh.copy(...)
- recurse into env -S / --split-string operands so env -S 'python3 -c ...' still detects the interpreter
Static classifier:
- fix scope-local walker so a nested def/class reassigning an alias name no longer inflates the outer single-assignment count and drops a real module-level sink alias
- track non-bare compile aliases (builtins.compile, from builtins import compile as comp) for the types.FunctionType(c) code-object gadget
- block child-process file writers at shell command position (touch/tee/cp/mv/mkdir/install/truncate/mkfifo/mknod/shred/unlink): a spawned child runs without the in-process write guard
- expand a literal **{...} unpack in the read scanner so open(**{'file': '../../etc/passwd'}) is resolved
- resolve a pathlib expression bound to a single-assignment name before read methods (p = Path('..')/'etc'/'passwd'; p.read_text())
- keep a wrapper's separated option argument in command position so stdbuf -o L python -c ... still detects the interpreter (env -i rm still caught; no FP on grep patterns)
- treat object.__getattribute__ / type.__getattribute__ as attribute obfuscation, covering gadget dunders and sensitive-module attrs (also closes __closure__ recovery of a guarded wrapper's original callable)
- block runpy.run_path / runpy.run_module execution sinks
- treat shutil.copy*/move SOURCE as a read callee so a .. traversal source is caught
Runtime backstop:
- normalize a bytes realpath (fsdecode) before the workdir prefix compare so a legitimate in-workdir bytes write is not denied by a TypeError; outside bytes writes still denied
Static classifier:
- resolve pathlib expressions passed to open()/read callees so open(Path('/etc') / 'passwd') blocks like open('/etc/passwd')
- flag getattr()/setattr() of an introspection gadget dunder (__globals__, __subclasses__, ...) regardless of receiver
- add the .get() twin of the globals()/locals()/vars() namespace-dict subscript guard
- constant-fold sys.modules[...] and sys.modules.get(...) keys so a concatenated key is caught
- track 'from builtins import __import__ as imp' as a dynamic import alias
- treat deserializer module aliases (pickle, dill, ...) as sensitive targets for getattr/vars/__dict__
- flag code objects executed through types.FunctionType(compile(src, ...), ...), including the c = compile(src); FunctionType(c) two-step
- block language interpreters (python/perl/ruby/node/...) at shell command position: a spawned child runs without the in-process write guard
Runtime backstop:
- pin os.fspath/os.path.realpath to captured originals inside _within so a sandboxed reassignment of os.fspath cannot make realpath resolve an outside write target to an in-workdir path
Scope-aware assignment aliases (extends the per-scope index):
- Resolve single-assignment aliases of dangerous callables in the call's own scope:
e = builtins.eval, im = importlib.import_module (and imp = __import__), and
l = pickle.loads (incl. aliased modules). Previously only bare-name and from-import
aliases were recognized.
- Count function parameters as local bindings so a parameter lexically shadows an
outer sink alias of the same name (fixes a false positive where def f(s): s(...)
with a module-level s = os.system flagged the parameter call).
Sensitive-read scanner:
- Resolve pathlib join receivers -- (Path('/etc') / 'passwd').read_text() and
Path('/etc').joinpath('passwd') -- not just a bare Path(...) constructor.
- Normalize path spellings (collapse redundant separators / '.' and resolve '..')
before the exact / dir checks, so /etc//passwd, /etc/./passwd and
/tmp/../etc/passwd are matched.
- Fold function-local single-assignment path constants (def f(): p = '/etc/passwd';
open(p)), not only module-level constants.
- Flag sys.modules.get('os') as the method-call twin of sys.modules['os'].
Runtime realpath backstop:
- Path.open coerces a str-subclass mode through the base str (matching the other open
wrappers) so a lying __contains__ cannot skip the write check.
- Path.rename/replace/link materialize the target once so a stateful __fspath__ cannot
return an in-workdir path for the check and an outside one for the real call (the
pre-3.11 accessor path where this wrapper is the only confinement).
Adds regression tests across the classifier, aliasing and runtime-backstop suites.
Scope-aware alias resolution (replaces the flat, module-wide alias maps):
- A new per-scope index resolves shell-sink, exec-builtin and compiled-code aliases
with Python lexical scoping. This fixes two problems the flat maps had: a safe
`c = compile('1+1')` in one function no longer shadows a dynamic `exec(c)` in
another (a real bypass), and a `s = os.system` in one function no longer makes a
benign `s = print` call in another look like a shell sink (a false positive), while
still catching a genuine function-local sink and honoring local shadowing of a
module-level alias.
Runtime realpath backstop:
- io.FileIO now passes the MATERIALIZED fspath to the real constructor (a stateful
__fspath__ could otherwise return an outside path to the C constructor).
- Deny an integer fd path for the mutating single-path wrappers (os.chmod(fd) etc.):
a read-only fd opened on an outside file could otherwise mutate host metadata.
Constant-folder allocation DoS:
- Refuse dynamic printf widths/precisions ('%*s', '%.*f') that draw their size from a
runtime argument.
- Bound str.replace / str.join output before it allocates (a long replacement over
many occurrences, or joining many long parts, can build a multi-gigabyte string).
Static classifier:
- Flag builtins / a sensitive module reached through the namespace dict:
globals()['__builtins__'], locals()[...] and globals()['os'].
Adds regression tests across the aliasing, runtime-backstop, const-fold and classifier
suites for every item above.
Runtime realpath backstop:
- Guard the low-level posix / nt module mutators (os re-exports from them, so
posix.open / posix.rename / ... stayed reachable with the originals).
- Guard io.FileIO / _io.FileIO constructors for write modes (a C constructor that
opens a file without routing through open()).
- Add os.mkfifo / os.utime / os.setxattr / os.removexattr (and lchflags) to the
guarded single-path mutators.
- Materialize fspath ONCE per call so a stateful __fspath__ cannot return a workdir
path for the check and an outside path for the syscall (TOCTOU).
- Coerce open() mode through the base str and os.open flags through the base int, so
a str-subclass __contains__ or an int-subclass __and__ cannot lie to the guard.
Constant-folder allocation DoS:
- Refuse str.format templates with a nested width field ({:{}}) driven by an
oversized numeric argument before format() allocates.
Static classifier:
- Reconstruct the full pathlib receiver path (all constructor args, joined) and
accept module-qualified pathlib.Path so Path('/etc', 'passwd').read_text() and
pathlib.Path(...) reads are inspected, not just single-arg bare Path(...).
- Treat builtins.__import__ / __builtins__.__import__ as a dynamic import.
- Count alias single-assignment per function scope instead of tree-wide, so two
functions binding the same local name no longer cancel out and miss a real sink.
Adds regression tests across the runtime-backstop, const-fold, aliasing and
classifier suites for every item above.
Fixes a further batch of P1 bypasses and analysis-time DoS vectors found in review.
Static classifier:
- Decode exec/compile bytes payloads the way CPython does (PEP 263 coding cookie
via tokenize.detect_encoding), then analyze the real source. A bytes payload whose
UTF-8 view is pure comments but whose utf-7 decode runs hidden code no longer slips
through; a payload decoding to a blocked op blocks, a benign one stays allowed.
- Resolve exec-builtin aliases assigned in nested scopes (def f(): e = exec; e(...)),
matching the shell-sink aliasing (stored-once guard keeps it low false-positive).
- Treat deserializer modules (pickle/marshal/dill/...) as dangerous dynamic-import
targets so __import__('pickle').loads(blob) is caught.
- Flag vars(os) / vars(__builtins__) as a module-__dict__ obfuscation, like os.__dict__.
- Inspect the pathlib receiver path for read methods: Path('../../.ssh/id_rsa').read_text()
/ read_bytes() / open() now check the constructor path, not only call args.
- Fold literal os.path.join / posixpath.join so open(os.path.join('/etc','passwd')).read()
is seen by the sensitive-read scanner instead of treated as opaque.
Constant-folder allocation DoS (folding runs in-process, before subprocess rlimits):
- Reject oversized f-string / str.format / %-format widths and precisions before
format() allocates the padded string.
- Reject oversized str padding-method widths (ljust/rjust/center/zfill).
- Cap list/tuple repetition (seq * n) as str/bytes repetition already was.
Runtime realpath backstop:
- Do not publish __wrapped__ on the guard wrappers (functools.wraps would expose the
original unguarded callable, e.g. open.__wrapped__(outside, 'w')).
- Guard the low-level _io.open entry point (io.open / builtins.open originate there).
- Confine os.chdir to the workdir and deny os.fchdir so a cwd escape cannot turn a
later relative read/write into a host-path access.
- Deny fd-based metadata mutators (os.fchmod / os.fchown) that could reuse a read-only
descriptor opened on an outside file.
Adds regression tests across the const-fold, aliasing, exec-recursion and runtime-
backstop suites for every item above.
Harden the code-exec sandbox against bypasses raised in review, keeping the
static gate a pure classifier (it never executes the tool call):
- Shell scan: `timeout` duration/float args (5m, 0.5, 2h) no longer drop the
following command out of command position, and `find -exec CMD ... ;` rescans
the whole slice so a wrapped `env`/`timeout`/`sh -c` target is still caught.
- exec/eval/compile of a bytes payload that is not valid UTF-8 Python now blocks:
those sinks honor PEP 263 coding cookies (e.g. utf-7) that the static UTF-8 view
cannot see; plain ASCII bytes payloads stay allowed.
- Resolve aliased/indirect reaches to exec, dynamic import, sensitive modules and
deserialization sinks: builtins.eval / __builtins__.exec, from builtins import
exec as e, importlib aliases, sys.modules[...] (and the getattr form), os.__dict__,
posix/nt, and pickle/marshal module-and-symbol aliases plus the *.load variants.
- Shell-sink aliasing walks the whole tree, so a function-local `s = os.system`
alias is resolved (the stored-once guard keeps it low false-positive).
- Drop __mro__ / __code__ from the introspection-gadget dunders: on their own they
do not reach an execution primitive and are read by ordinary ML/debug code.
- Refuse folding oversized `bytes(n)` / `bytearray(n)` so static analysis cannot OOM.
Runtime realpath backstop:
- Fail closed on a mutating dir_fd / src_dir_fd / dst_dir_fd (an fd-relative path
cannot be confined by a string realpath) for os and shutil mutators.
- Confine Path.rename/replace/symlink_to/hardlink_to when the destination is passed
as the `target=` keyword, not only positionally.
- Splice the guard after a leading docstring and `from __future__` imports instead
of prepending it, so future-import programs no longer raise SyntaxError while the
sandbox is still established before the first real statement.
Adds regression tests for each gap across the shell, const-fold, exec-recursion,
aliasing and runtime-backstop suites.
On Python <= 3.11, pathlib._NormalAccessor captures io.open (and os.* mutators)
into class attributes at pathlib import time. A C builtin captured there does not
bind on instance access, but a Python wrapper does: self shifts into the next
positional, so Path.open / Path.write_text raised 'open() argument mode must be
str, not PosixPath' once the guard had replaced io.open with a Python wrapper
before pathlib was imported. (3.12+ dropped the accessor, which is why it only
failed on the 3.10 CI leg.)
Import io and pathlib at the top of the guard, before any patching, so the
accessor captures the original builtins, and confine Path.open by wrapping the
public method directly (mode-aware) rather than relying on the io.open patch to
reach it. Direct io.open() writers are still guarded for the zipfile-based cases.
The static filesystem write-confinement (the LOCAL/ESCAPE/UNKNOWN path resolver
_resolve_path / _resolve_path_call plus the _FS_* mutating-op inventory) was the
largest and most complex part of the classifier, and for writes it duplicated the
runtime realpath backstop, which is strictly more robust: it resolves the true
realpath at the syscall boundary, so it also catches dynamic paths, pre-existing
symlinks, and library writers the static pass could not prove.
Make the runtime backstop the single filesystem-write boundary and delete the
static resolver:
- Harden the backstop to close the gaps the static layer used to cover: guard the
low-level os.open (any mutating flag confines the target; a mutating dir_fd fails
closed) and io.open (which also carries pathlib.Path.open('w')), and add
os.mknod / lchmod / lchown / chflags and shutil.chown / copymode / copystat to
the wrapped set. Native-C writers (cv2.imwrite) and the realpath TOCTOU window
remain documented residuals that only OS-level isolation can close.
- Remove _resolve_path, _resolve_path_call, _resolve_join, _classify_path_string,
_is_pathlib_expr and the _FS_* / _PATHLIB_CTORS / _PATH_DEPTH_CAP constants, and
the write half of the filesystem visitor plus the FS_READ_STRICT knob.
- Reads are not confined by the backstop, so keep a small static sensitive-read
scanner (_is_sensitive_abs_path) that still blocks host-secret reads via a
sensitive absolute / ~ literal in any call arg (covers open, os.open, and library
loaders such as pandas.read_csv('/etc/passwd')) and .. / ~ traversal on the
dedicated open/read callees.
Net: about 300 fewer lines in tools.py and one fewer concept to audit; static
analysis now scopes to exec, shell, network and sensitive-reads while writes are
confined at runtime. Rework the filesystem tests around the new contract and add
os.open / io.open / Path.open / dir_fd escape cases to the backstop suite.
* scripts: refresh scan_packages allowlist baseline
Regenerate scripts/scan_packages_baseline.json against the current
resolved dependency set so the blocking pip scan-packages gate matches
what the scanner now finds. Refreshes evidence hashes for benign
findings whose code shifted lines (unsloth-zoo mlx loader, gguf/mlx
test /tmp fixtures) and adds two mainstream-library entries that were
newly surfaced (torch inductor codecache base64+subprocess compile
cache, torch testing common_utils socket import). Stale entries whose
matching code changed and no longer triggers are dropped.
All entries remain CRITICAL/HIGH findings manually judged benign;
matched on (package, file, check, evidence_hash).
* ci(security-audit): re-run scan when the allowlist baseline changes
The security-audit pull_request trigger listed the scanners but not
their allowlist baselines, so a baseline-only edit never re-ran the
scan that consumes it. A refreshed baseline could therefore merge
without CI confirming its evidence hashes match what the scanner finds.
Add scan_packages_baseline.json and scan_npm_packages_baseline.json to
the paths filter so baseline changes are validated on their own PR.
* Keep native RoPE scaling when extending context; carry rope_theta for linear
When max_seq_length exceeds a model's native window, the loader overwrote the
model's rope_scaling with linear scaling. For models that already ship a scaled
RoPE (llama3/yarn/longrope) that is far worse for long context, and on
transformers v5 the linear dict omitted rope_theta (v5 keeps it under
rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens.
Keep the native scaling and just widen the window; only synthesize linear for
plain-RoPE models, and carry rope_theta so v5 keeps the real base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only preserve native llama3 when extending context; keep linear fallback otherwise
The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear,
llama3 and longrope and its longrope branch reads a top-level
original_max_position_embeddings, so preserving yarn or a nested-only longrope config
would raise during construction on transformers <= 4.47.1. Keep only llama3 native;
yarn/longrope/other types fall back to the linear override, still carrying rope_theta.
* Correct long-context extension comment to match llama3-only preservation
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix fast_gemv crash on compressed-tensors FP8 models
Loading a compressed-tensors FP8 checkpoint (for example
unsloth/Llama-3.2-1B-Instruct-FP8-Block) with fast_inference=False and
running a forward crashed with 'Parameter object has no attribute absmax'
inside fast_gemv.
A compressed-tensors CompressedLinear exposes an already dequantized bf16
weight at forward time while keeping a weight_scale Parameter. The quant
state resolution in get_lora_parameters/get_lora_parameters_bias fell back
to that weight_scale, so a bf16 weight was routed into the bitsandbytes
fast_gemv/fast_dequantize path, which expects a bitsandbytes QuantState
with an absmax attribute.
Only fall back to weight_scale_inv/weight_scale when the weight is still
fp8. A decompressed bf16 weight then resolves to no quant state and flows
through the normal bf16 path, which already handles bias and the LoRA
backward. Real fp8 and bitsandbytes 4bit weights are unchanged.
* Skip the fast_gemv dispatch test before importing unsloth when bitsandbytes is absent
* Fix Windows installer torch index override
* Clear inherited uv index env vars for pinned installs in studio/setup.ps1 (#6898)
* Harden setup.ps1 index-var clearing to truly remove vars (#6898)
* Apply UV_DEFAULT_INDEX torch index fix to Linux/Mac install.sh (#6898)
* Neutralize all uv index env vars for pinned torch installs (#6898)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Vulkan llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address gemini's feedback
* Studio: move the Vulkan VRAM probe into a standalone script
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve Vulkan probe error reporting
* Resolve llama-server symlink so Vulkan build is detected
* Drop unreachable Vulkan fallback in GPU free-memory dispatcher
* Skip the Intel GPU probe when NVIDIA or ROCm is present
* Reserve host RAM headroom for Vulkan integrated GPUs
* Add a `UNSLOTH_FORCE_VULKAN` environment variable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor GGML_VK_VISIBLE_DEVICES, reserve discrete Vulkan VRAM headroom, and clear Intel GPU on --cpu-fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route Intel and forced-Vulkan hosts to the upstream Vulkan prebuilt, add arm64 Vulkan, keep Vulkan out of RAG auto-detect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the fork release pin when routing a Vulkan host to the upstream repo
* Gate auto-Vulkan routing on no physical NVIDIA so hidden CUDA devices aren't used
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin Vulkan launches with --device Vulkan<i> instead of the raw GGML_VK_VISIBLE_DEVICES index space
* Let user --device override the Vulkan pin, and gate direct Vulkan asset picks on no physical NVIDIA
* Update RAG auto-backend test mocks for the _resolve_auto binary and Vulkan probes
* Keep the add_dll_directory handle alive through the Vulkan probe DLL loads
* Revert RAG auto Vulkan guard, guard multi-backend Vulkan detection, and preserve forced Vulkan across updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use getattr for RTLD_GLOBAL in the Vulkan probe CDLL mode
* Skip CUDA/ROCm APU and datacenter GPU tuning on Vulkan builds
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not
CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and
_apply_datacenter_env were reading the wrong device. On a mixed AMD APU
plus discrete GPU host that could raise a spurious system-RAM shortfall
and block a valid discrete-GPU load. Gate all three call sites on
not is_vulkan_backend; the Vulkan path already reserves iGPU host
headroom and the backend ignores GGML_CUDA_* anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Vulkan-guard comment in load_model
* Reduce comments in Vulkan support to be more succinct
* Resolve shell-wrapper llama-server entrypoint to the real lib dir
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install
root when it cannot symlink into build/bin. _find_llama_server_binary
returns that root entrypoint, but Path.resolve() does not follow a shell
wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend
missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device
pin on an otherwise valid Vulkan install. Follow the wrapper's exec target
to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio /v1/messages: accept thinking and unknown content blocks
The Anthropic-compatible /v1/messages endpoint modeled a message's content as
Union[str, list[{text|image|tool_use|tool_result}]], so any other block type
made Pydantic reject the whole request with
`messages.N.content.str: Input should be a valid string`. Resuming a Claude
session commonly replays assistant turns that carry `thinking` (extended
thinking) blocks, and sometimes a null content for a tool-only turn, both of
which tripped this and returned a 400.
Accept them:
- Add a permissive AnthropicUnknownBlock fallback (any block whose type is not
one of the four known ones), so thinking/redacted_thinking/provider-specific/
future blocks validate. A validator keeps known types on their typed models,
so a malformed known block (e.g. a tool_use without id) still fails cleanly.
- Coerce a null message (and tool_result) content to "" so the converter's
`for block in content` stays safe.
The converter already drops block types it does not translate, so a thinking
block is not forwarded to the model.
* Studio /v1/messages: keep user content validation strict
Make the thinking/null leniency role-aware so it never silently drops real
user input. Assistant turns (replayed history) still accept unknown/thinking
blocks and coerce a null tool-only turn to empty. User turns keep the strict
boundary: a null user content is rejected, and a content block the converter
cannot translate is rejected instead of being dropped into an empty prompt.
Also remove an empty file committed by accident.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: coalesce resumed user turns and tighten content checks
- The /v1/messages count and generation paths now coalesce the adjacent user
turns that dropping an empty or null assistant turn can leave behind, so a
strict GGUF chat template no longer 400s on non-alternating roles.
- A user content block with a non-string type (list / dict) is rejected as a
clean 400 instead of raising TypeError and escaping as a 500.
- The assistant null-to-empty coercion only applies to an explicit null; an
assistant turn that omits content entirely still fails required-field
validation instead of being silently coerced to an empty string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* unsloth start: add --resume to persist and reopen agent sessions
`unsloth start <agent>` launches a coding agent whose home is a throwaway
temp dir wiped on exit, so codex/openclaw/hermes/pi (which relocate their
whole home there) cannot resume a conversation after you quit. opencode and
claude keep their session data in a fixed user dir, so they already resume.
Add an opt-in --resume/--no-resume flag: it routes the launch to the stable
Unsloth agents dir (the same one --no-launch already uses) so the session
survives the exit, never touching the user's own ~/.<agent>. A bare --resume
also reopens the last conversation via the agent's native flag (codex
`resume --last`, opencode/claude/pi `--continue`). The default is unchanged:
a plain launch still uses a temp dir and persists nothing.
Add a dispatch-only `resume` job to the Local Agent Guides CI that drives the
real launch path and asserts the split: codex/pi are wiped without --resume
and persist with it, while opencode/claude persist either way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unsloth start: rename --resume to --persist
The session flag collided with agents' own resume flags. `unsloth start
claude --resume <id>` used to forward `--resume <id>` straight to Claude
(which keeps its history in ~/.claude regardless), so a boolean --resume on
unsloth start would have swallowed the session id and turned it into a stray
prompt. Name the persistence flag --persist instead, so every agent's native
resume flag (claude --resume <id>, codex resume, opencode --continue, ...)
still passes through untouched. Behavior is otherwise identical: --persist
keeps a launched agent's session under the Unsloth agents dir, and a bare
--persist reopens the last conversation.
Add a regression test that `--resume <id>` passes through verbatim, and in the
CI resume experiment skip the redundant second pass for opencode/claude (they
persist either way, and a second CPU turn only risks a timeout).
* unsloth start: correct --persist help and drop the buggy auto-resume
Reword the --persist help to be accurate: claude and opencode keep sessions in
the user's own stores and resume regardless, so --persist only stabilizes the
otherwise-ephemeral relocated home of codex/openclaw/hermes/pi. Drop the
bare-launch auto-append of native resume tokens: it errored on a first launch
with no prior session, and was inconsistent between launch and no-launch.
--persist now only keeps the session dir; resume via the agent's own command
(e.g. `unsloth start codex --persist resume`), which now finds it.
In the CI resume experiment, fail the pass when the launched turn exits
non-zero, so a write-then-error is not misread as PERSISTED.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Silence torch._check_is_size FutureWarning and shim it if torch removes it
bitsandbytes 4-bit dequant calls torch._check_is_size, which torch
deprecated with a FutureWarning ("Use _check(i >= 0) instead") that prints
on every bnb-4bit load. Silence that warning in suppress_cuda_printf, and
add fix_torch_check_is_size so a future torch that removes _check_is_size
gets it shimmed to _check(i >= 0) (honoring the max bound) and bitsandbytes
keeps working.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten fix_torch_check_is_size docstring
Lead with what the shim does and drop the redundant line; two lines
instead of three, same intent.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Mirror dynamo/inductor config sets into defaults so torch 2.12 worker threads honor them
torch 2.12 stores config user overrides in ContextVars, so direct
assignments like torch._dynamo.config.recompile_limit = 1024 no longer
reach the autograd engine worker threads. Gradient checkpointing
recomputes fullgraph-compiled gpt-oss kernels inside backward on those
threads, which then read the default recompile limit of 8 and raise
FailOnRecompileLimitHit at step 0 of GRPO/SFT. Mirror direct config
assignments into the process-global entry defaults on torch >= 2.12,
restoring the torch <= 2.11 cross-thread semantics while leaving the
context-scoped config.patch API untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep config.patch thread-local when mirroring dynamo/inductor sets
config.patch(...) also assigns through ConfigModule.__setattr__, so the
default-mirror was leaking its scoped, thread-local writes into the
process-global entry default. Track patch enter/exit with a per-thread
depth counter (wrapping ConfigModule.patch) and skip mirroring while
inside a patch, so only genuine direct assignments restore the torch
2.11 cross-thread semantics and config.patch stays context-local.
* Also keep config.load_config thread-local when mirroring config sets
load_config restores a saved dynamo/inductor config by calling setattr
per key, which the default-mirror would otherwise leak process-wide just
like config.patch did. Wrap load_config with the same per-thread depth
counter (renamed to _scoped_depth) so both scoped writers skip the mirror
and stay context-local, while genuine direct assignments still restore the
torch 2.11 cross-thread default.
* Drop the pre-existing override replay from the config thread fix
The replay was redundant: this runs from _gpu_init before unsloth sets any
dynamo/inductor config, so the __setattr__ wrapper already mirrors every
later assignment (recompile_limit included). It could also read a value
that belonged to a config.patch context still active at import time and
write that thread-local override into the global default. Removing it keeps
the cross-thread fix and drops the now-unused _inductor.config import.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
When a coding agent is missing, `unsloth start <agent>` offers to run the
vendor's own installer (curl | bash, irm | iex, or npm) after an interactive
confirm. Those installers execute with the user's privileges and there is no
signature or hash check on the fetched content, so a blind "yes" is a
supply-chain risk if the delivery path is compromised.
Keep the auto-install convenience but make consent informed: before the prompt,
name the exact remote source the installer fetches (or the command it runs for a
package installer) and state that nothing verifies a signature or hash. Behavior
is otherwise unchanged: non-interactive stdin still never executes anything, and
the confirm still defaults to no.
* Fix FastSentenceTransformer Qwen embedding preprocessing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Document Transformer.load embedding modality fix for #6881
* Harden #6881 fix and add forwards/backwards-compatible regression tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fall back to Transformer constructor on legacy sentence-transformers without Hub-capable load
* Mirror legacy sentence-transformers fallback in embedding-parity tripwire test
* Tighten #6881 comments and docstrings
* Skip embedding-parity test on CPU-only runners since FastSentenceTransformer requires CUDA
* Honor the transformer module's saved subfolder when loading
modules.json records a path for the Transformer module (root for
decoder embedders like Qwen3-Embedding, 0_Transformer for the classic
layout). Pooling/Normalize already load from their saved path; thread the
same path into Transformer.load as subfolder so config and tokenizer
resolve like stock ST. stays a no-op, so single-module models are
unchanged.
* Make embedding-parity test bf16-aware
fp16 overflows to NaN on bf16-native embedders such as EmbeddingGemma
(Gemma3), producing a false parity failure. Prefer bf16 when the GPU
supports it so the tripwire can guard the full documented embedding
matrix (Qwen3-Embedding, EmbeddingGemma, BGE-M3, all-MiniLM, GTE-ModernBERT),
not just fp16-safe models.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Retry the Studio UI shutdown re-login on transient goto timeout
The Chat UI Playwright smoke intermittently failed at the pre-shutdown
re-login: page.goto('/login') can hit a 60s TimeoutError on a slow runner
even while the server is healthy, and the surrounding except only tolerated
ERR_ABORTED / interrupted-navigation, so a plain timeout hard-failed the job.
Wrap the re-login goto/wait/fill/submit in the same 3-attempt retry the
change-password step already uses (recover_or_replace_page between tries,
per-attempt fail screenshots, wait_for_health pre-gate). The composer wait
stays outside the loop so a retry never re-navigates after login has set
tokens (which would redirect to /chat via the guest guard); it remains the
authoritative confirmation, so a genuinely broken login still fails.
* Catch transient login-request failures and preserve error listeners on recovery
Wait on the /api/auth/login POST inside the retry (via click_and_wait_for_response)
so a transient 4xx/5xx is retried in-loop instead of surfacing only at the
out-of-loop composer wait, matching the change-password step. When
recover_or_replace_page swaps in a fresh page, re-attach the pageerror/console
listeners so error tracking survives the replacement.
Tighten the eval/exec/compile dynamic policy so an executing sink (eval/exec/
runpy) applied to a payload that cannot be statically recovered is refused
unconditionally, not only when an RCE-core module happens to be imported in the
snippet. An un-analyzable executing payload can synthesize any shell, network,
or filesystem escape at runtime, so the prior in-scope-import heuristic left
exec(input()) and eval(user_var) allowed whenever no such import was present.
compile() of the same payload stays allowed since it does not run.
A payload that is fully recovered as a constant but is invalid Python for the
sink's mode (for example eval("data = 1") or eval("not python !!")) is now
allowed: its exact source is known and it raises SyntaxError at runtime, so it
is not an execution vector. Only genuinely opaque, non-recoverable payloads
(for example eval of a runtime-computed f-string) reach the block.
Remove the now-unused _RCE_CORE_MODULES set and _scope_imported_roots helper,
and move the opaque-f-string case in the tests to the blocked set.
* Stabilize floating monitor drag
* Restore floating monitor exit animation
* Harden Windows Studio smoke checks
* Keep API menu badge removed
* Apply no-build-tools env overrides in-script
The runner does not apply step-level env keys containing parentheses,
so ProgramFiles(x86) kept its real value and Find-VsBuildTools still
detected VS through vswhere. Set the overrides inside each pwsh step
instead; child processes inherit them. The resolver step moves to pwsh
because bash cannot export a variable named ProgramFiles(x86).
* Reset chat UI session without a second browser context
macOS runs Chromium with --single-process, where closing the last
context tears down the whole browser, so the shutdown re-login died
with TargetClosedError on new_page. Clear cookies and swap pages
inside the same context instead, opening the replacement page before
closing the old one.
* Keep the no-build-tools Path filtered across session refreshes
install.ps1's Refresh-SessionPath and setup.ps1's Refresh-Environment
rebuild the session Path from the Machine and User registry scopes, so
the process-level filter could be undone mid-install and re-expose
CMake. Filter those scopes in the Prepare step with normalized dir
matching and restore them in cleanup.
* Drop stale localStorage auth tokens before re-login
Auth tokens live in localStorage, not cookies, and the login guest
guard redirects on their mere presence. Remove them during the session
reset so the /login navigation is deterministic instead of relying on
the tolerated redirect bounce.
Add ln to the bash command denylist so a symlink escape cannot be created from
the terminal tool. In the sandboxed (non-bypass) _python_exec path, prepend a
one-line guard to the generated temp module that monkeypatches only MUTATING file
ops (builtins.open in write/append/x/+ modes, os remove/unlink/rmdir/removedirs/
rename/renames/replace/truncate/chmod/chown/mkdir/makedirs/symlink/link, shutil
rmtree/move/copy/copy2/copyfile/copytree, pathlib write_text/write_bytes/unlink/
rename/replace/mkdir/rmdir/chmod/symlink_to/hardlink_to/touch) to resolve the true
os.path.realpath of the target and raise PermissionError unless it lands inside
the injected session workdir. Reads are left unpatched. The guard runs in its own
namespace so helper names never leak into user globals, and it is skipped
entirely under disable_sandbox. This catches what the static gate cannot prove:
pre-existing symlink escapes and dynamic library-writer paths that funnel through
builtins.open. Benign in-workdir relative writes and library imports are
unaffected (importlib swallows out-of-workdir bytecode-cache write failures).
Add pragmatic aliasing so an aliased shell sink with a dangerous argument is
caught: a name stored exactly once and bound to a resolved os/subprocess sink
(s = os.system; s('rm -rf /')) and inline literal-container indexing
([os.system][0](...), (os.system,)[0](...), {'k': os.system}['k'](...)) both feed
the existing _find_blocked_commands argument check. Resolution is deliberately
low-false-positive: only unambiguous single assignments and inline literal
containers, never a flow-insensitive union, so s = os.system; s = print; s('hi')
is not aliased. The shell-sink set is lifted to module scope (_SHELL_SINK_FUNCS)
so the alias pre-pass and the visitor share one definition. Interprocedural and
flow-sensitive taint remain out of scope (deferred to a full fixpoint).
Add a filesystem_violations category backed by _resolve_path, a LOCAL / ESCAPE /
UNKNOWN classifier that constant-folds strings and understands os.path.join,
pathlib Path()/'/'/joinpath, and f-strings with real join plus absolute-reset
semantics. expanduser / expandvars / os.environ / getcwd / dynamic parts collapse
to UNKNOWN. A new _FilesystemPolicyVisitor inventories destructive and mutating
ops (open write/append/x/+, os remove/unlink/rmdir/rename/replace/truncate/chmod/
chown/mkdir/makedirs/mknod/symlink/link/chdir, shutil rmtree/move/copy*, pathlib
write_text/write_bytes/unlink/rename/replace/mkdir/rmdir/chmod/symlink_to/touch,
tempfile dir=, and a curated numpy/pandas/torch/joblib/PIL/matplotlib/cv2 writer
set) and applies prove-or-block: mutating LOCAL allows, UNKNOWN/ESCAPE blocks.
rename/move check src and dst; symlink/link check both target and link path;
chdir must be LOCAL; tempfile dir= must be LOCAL. Reads block only on a provable
escape (sensitive absolute path or ..'/~ traversal), with an FS_READ_STRICT knob
for prove-or-block reads. A callee-independent literal-sensitive-path scan blocks
loaders like pandas.read_csv('/etc/shadow'). Library writers block only on a
provable escape so in-memory buffers are not over-blocked; the Stage 5 runtime
backstop covers the dynamic residual.
Replace the blanket dynamic_exec block with a recursive payload analyzer gated by
UNSLOTH_STUDIO_SINK_ANALYZER (default on; =0 restores the legacy ban). For eval /
exec / compile (and single-assignment aliases like e = exec), constant-fold the
first argument; a recovered source string is bracket-depth pre-scanned, size and
recursion-depth bounded, then re-classified through the full analyzer. An inner
sink blocks and surfaces the inner reason; a clean inner payload allows; a bound
or budget breach fails closed. Non-foldable payloads follow a low-false-positive
dynamic policy: block when assembled from decode / fetch / runtime-assembly
primitives (including nested exec and large string repetition) or when an
RCE-core module is imported in scope, else allow.
Keep the gadget-dunder and dynamic-import blocks but constant-fold import names
so __import__('hugging'+'face_hub') resolves to a real module. Refine getattr /
setattr on a sensitive module so a benign constant attribute (getattr(os,
'getpid')) is allowed while a dynamic or dangerous constant attribute blocks. Add
pickle/marshal/dill.loads as unverifiable code-deserialization sinks. eval('2+2'),
compile('a+b','<s>','eval') and ast.literal_eval now pass; base64/hex/rot13/chr
and gadget-obfuscated escapes still block. Wire a filesystem_violations category
through is_safe, the info dict, and the reason assembly (populated in a later
stage). The three legacy tests that asserted the blanket ban are updated to the
new recurse-the-payload behavior.
Introduce _const_fold, a whitelist-only, bounded, side-effect-free partial
evaluator plus a single-assignment const-prop environment builder. It recomputes
pure transforms on literals only (concat, repeat, join, format, f-strings,
slice/reverse, chr/ord, base64/hex/rot13/zlib decode, pure builtins and string
methods) and never executes, imports, or reflects on user code. Depth, op, size,
and sequence caps guarantee it can only fail to recover a value, never crash or
hang. This is the foundation the later eval/exec unwrapping and filesystem path
resolver build on.
* unstructured block removal
* Enhance unstructured block handling
* Restrict block cleanup to upload UIDs
* cleanup for seed block uploads
* upload cleanup queue for unstructured blocks in recipe studio
* Fix unstructured upload cleanup edge cases
* Fix unstructured upload import ownership
* Fix-unstructured-import-path-ownership
* Guard failed-delete restore against stale block in unstructured drop zone
* Drain queued upload cleanups when autosave is skipped
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
