* Replace standalone Studio wording with Unsloth
Replace the single word Studio with Unsloth wherever it is used as
shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n
locales, workflow display names, comments and docstrings.
Kept unchanged: the full name Unsloth Studio, third party product
names (LM Studio, Visual Studio, Mac Studio), feature names
(Recipe Studio, Fine-tuning Studio and its translations), and all
identifiers such as env vars, commands, paths and filenames.
* Address review feedback on the Studio wording rename
Use "an" before Unsloth where the rename left the article as "a".
Restore the split brand where Unsloth and Studio render as two halves
of the full product name: the onboarding sidebar subtitle and the
IPv6 localhost warning. Scope two messages to the full name Unsloth
Studio where plain Unsloth was misleading: the AMD README bullet and
the CLI studio setup error.
* scan_packages: key baseline on matched-code hash
The baseline matched on (package, package-relative file, check), which
excluded the matched code, so a future finding of the same check in the
same file was suppressed regardless of what the code did. A malicious
future version of an already-baselined package could place a payload in
the same file under the same check and pass the enforcing gate.
Key the baseline on a hash of the matched code too. The hash is over the
deduped, sorted set of matched spans with L<NN>: line markers stripped, so
version bumps, line shifts and match reordering stay stable while new or
changed flagged code reopens the finding. Version is left out of the key so
routine dependency bumps do not reopen every entry. The hash is capped and
recomputable from the stored evidence.
Regenerate scan_packages_baseline.json against the current dependency set;
the hf-stack, studio and extras scan shards pass enforcing (no active
CRITICAL or HIGH).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: refresh baseline for newer unsloth-zoo release
A newer unsloth-zoo published after the first regenerate added
tests/test_mlx_save_export_regressions.py, a benign test fixture
(temporary_location="/tmp/ignored") that trips the /tmp dropper check.
Regenerate the hf-stack shard against the current set so the entry is
allowlisted; studio and extras are unchanged.
* scan_packages: harden baseline loading against malformed JSON
Guard against a non-dict top-level baseline and non-dict entries so a
corrupt or hand-edited allowlist warns and fails closed instead of
crashing with AttributeError, and treat an explicit evidence: null as
empty.
* scan_packages: hash the full match set, keep indentation, strip only the marker
Address the evidence-hash review feedback:
- Capture every matching line, not the first three, so a payload appended
after existing matches in a baselined file and check reopens the finding
instead of riding the sample.
- Preserve leading indentation so a flagged line moved out of a guarded block
reads as changed.
- Strip only each span's prefix up to the first L<NN>: marker, so an L<NN>:
inside the matched code is kept and a change to it reopens the finding.
Evidence and its hash are stored in full and stay recomputable from the stored
field. Regenerate the baseline; hf-stack, studio and extras pass enforcing with
no active CRITICAL or HIGH.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind baseline evidence to full matched code
Address review feedback on the evidence-hash baseline key:
- Split evidence only on real span delimiters (" | " before an L<NN>:
marker, or a newline), so a bitwise-or or union type in matched code
is no longer split apart into separate spans.
- Record matched lines in full (drop the 160-char per-line cap) and
record every distinct multiline match, so code appended past the cap
or a second cross-line match reopens the finding instead of riding the
first one.
- Give the large-JS-bundle and .pth base64-blob findings a content
digest instead of empty or prefix-only evidence, and record all .pth
import lines, so a changed bundle, blob or import no longer inherits a
baselined empty or truncated key.
- Warn when a loaded baseline has entries without evidence_hash so a
legacy baseline is regenerated rather than silently degraded.
Regenerate scripts/scan_packages_baseline.json against the current dep
set and add regression tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: harden multiline and duplicate evidence handling
Follow-up hardening so the evidence hash tracks the full matched code:
- For DOTALL patterns that match across lines, record every line the match
spans (not just the start line), so a change on a continuation line (the
URL inside a baselined C2 loop, a swapped credential path) reopens the
finding. A pathological greedy span is bounded to its head line plus a
digest of the rest.
- Keep duplicate spans in the canonical evidence so a second identical
matched line in a new code path changes the key instead of deduping away.
- Anchor the evidence prefix to strip only a genuine leading label or
line-number marker, leaving a marker-like "L<NN>:" inside raw .pth code
intact.
- Make the legacy-baseline warning explicit that entries without an
evidence_hash reopen rather than suppress under a coarse key.
Regenerate scripts/scan_packages_baseline.json (same finding set; entries
for same-file repeated checks are now tracked separately) and add tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind every combo and large finding to its full content
Close the remaining asymmetric-evidence gaps so a changed payload cannot
ride a reviewed baseline entry:
- Digest a capped multiline span from the code without line markers, so a
pure line shift stays stable while a continuation-line change reopens.
- Give the "Unusually large executable .pth" finding a content digest
instead of keying on byte size and import-line count alone.
- Record both contributing signals for the JS credential+network stealer,
the shell credential+network and persistence-hook combos, and the hidden
network+exec docstring payload, so changing the network/exec side reopens.
- Allow punctuation in an evidence label prefix so a "network+exec:" label
is stripped and line shifts do not change the key.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* scan_packages: bind remaining Python combos; key npm baseline on evidence
Python scanner: the openssl+key, anti-analysis, DNS-exfil and base64+exec+blob
combos recorded only one contributing signal, so a changed payload on the other
side could ride a reviewed baseline entry. Each now binds every co-occurring
signal (and the blob is digested, since it can sit on a separate line from the
decode call).
npm scanner: scan_npm_packages.py keyed its allowlist on (package, path,
pattern) only, the same coarse-key bypass the Python scanner just closed. Add an
evidence hash to the key (schema v3, fail-closed on older baselines) and store
full evidence. The committed baseline stays empty by design.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_npm_packages: bind full blob evidence and harden baseline loader
Follow-up on the npm evidence-hash key:
- _evidence now records every match and, when a snippet is truncated for
display, appends a digest of the full match. The obfuscated-blob key was
hashing only the truncated first-match snippet, so a changed payload tail or
an appended blob in the same package/file/pattern could ride a reviewed entry.
- _load_baseline guards that the root is an object, entries is a list, and each
entry is a dict before reading it, so a malformed baseline warns and fails
closed instead of raising AttributeError.
Add tests for a changed blob tail reopening the key and for malformed entries.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: symmetric baseline-loader guards; bind npm outbound host context
- Python _load_baseline now rejects a non-list "entries" with a warning instead
of raising TypeError, matching the npm loader.
- npm cred-surface-host (outbound) records the host with its URL path / fetch
call / host config, so a changed outbound path, headers or body reopens the
key rather than riding the bare host literal.
Add tests for both.
* scan_npm_packages: migrate v2 baselines and bind host-config outbound context
- _load_baseline now migrates schema v2 entries by recomputing the evidence
hash from stored evidence (with a legacy warning), matching the Python
loader, instead of discarding them; only pre-v2 basename schemas are rejected.
- The cred-surface-host (outbound) host-config branch now captures the whole
line (path, headers, body), so a changed outbound payload on the same
hostname line reopens the key instead of riding the bare host snippet.
Add tests for v2 migration and the host-config context binding.
* scan packages: bind PEM key bodies and npm windowed evidence to baseline keys
scan_packages: embedded-key findings now pin the full PEM block (BEGIN..END)
via a content digest, so a key body swapped under the same marker reopens the
finding instead of riding the unchanged BEGIN line. Single-line and DER keys
were already bound by their full matched line; marker-only references with no
END block (validation header lists) are unaffected, so the committed baseline
is unchanged.
scan_npm_packages: _evidence now digests the full containing line whenever the
shown snippet is only a window into it (short match on a long line, or a
truncated payload), so a changed payload tail outside the display window
reopens the key. The npm baseline is empty, so this changes no suppressions.
Adds regression tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan packages: bind multi-line evidence and every blob to baseline keys
_extract_evidence now extends each single-line match over its bracket
continuations, so a multi-line call binds its argument lines and a changed
URL or body on a continuation line reopens the key. After the per-line pass it
also records cross-line matches the scan cannot otherwise see (a DOTALL regex,
or a multi-line construct appended under a check that already had a one-line
match), so an appended multiline payload reopens instead of riding the key.
_blob_digest hashes every large base64 blob (not just the first) for the
base64+exec finding and the .pth large-blob finding, so an appended or swapped
second encoded payload reopens; single-blob files keep the same digest.
scan_npm_packages _evidence digests the full logical line (the matched line
plus its bracket-continuation lines), so a multi-line fetch's option and header
lines bind and a changed payload on a following line reopens the outbound key.
Regenerated the Python baseline: same package/file/check set, 24 entries pick
up the wider multi-line evidence. Adds regression tests for each case.
* scan packages: stop giant greedy spans from binding a whole-file digest
When a greedy DOTALL pattern (reverse shell socket...subprocess, C2 loop) has
its anchor tokens far apart, the match span covers the whole file. Digesting
that span bound thousands of unrelated lines, so the evidence hash drifted on
any edit between the anchors (a dependency bump reshuffling the file), which
made a baselined finding reopen on an upstream release. The multiline pass now
skips an oversized span when the per-line pass already bound the signal lines,
so the evidence is the stable matched lines; a genuinely appended multi-line
construct stays under the cap and is still recorded.
Regenerated the Python baseline against Python 3.12 (the version the scan CI
shards run) so the resolved dependency set matches CI. Same package/file/check
set. Adds a regression test.
* scan packages: tighten evidence binding (order, string brackets, span size)
Address review follow-ups on the evidence extraction:
- _canon_evidence keeps discovery (line) order instead of sorting. Line-shift
stability already comes from stripping the L<NN>: markers, so order stays
significant and reordering matched lines (a multi-line call's arguments)
reopens the finding.
- _logical_line_end (Python) and _logical_line_text (npm) blank string literals
before counting brackets, so a ) inside a string argument does not close the
logical line early and drop later argument lines.
- The oversized-span skip now only drops a giant whole-file bridge (over 60
lines); a genuinely appended multi-line construct is recorded so its payload
reopens, rather than riding an existing one-line match.
- npm _logical_line_text binds the enclosing bracket group, so a host-config
object whose { is on a prior line binds its path/headers/body lines.
Regenerated the Python baseline (Python 3.12, matching the scan CI shards):
same package/file/check set. Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan npm packages: normalize and bound the logical-line digest
- _evidence whitespace-normalizes the logical line before digesting (matching
_evidence_hash), so a formatter-only reindent of the bound continuation lines
does not change the sha256 suffix and reopen an unchanged finding.
- _logical_line_text follows a bracket group to its close up to a hard 200-line
cap (digest input only), so a config object longer than the backward window
still binds its whole tail instead of silently truncating.
Adds regression tests. npm baseline is empty, so no regeneration is needed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: cap single-line evidence and widen npm opener window
Cap each rendered evidence line at 200 chars in scan_packages.py: a long
or minified one-line file is shown as a bounded prefix plus a sha256 of the
full line, so a packed payload cannot dump unbounded content into the CI
logs or baseline while a change past the cutoff still changes the digest
and reopens the finding. Mirrors how the npm scanner bounds its snippets.
Widen the npm backward opener window (_MAX_CONT_LINES 12 to 200, symmetric
with the forward cap) so a host deep inside a large options object binds
the whole object, not just its own line; a changed path, header, or body on
any property reopens.
Regenerate the Python baseline with Python 3.12: only the protobuf
nspkg.pth and unsloth-zoo compiler.py evidence change, both from the new
line cap; the package/file/check key set is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: bind all host contexts, deep call continuations, far-back npm openers
Three fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: measure the forward bracket-group cap from the matched
line (idx + _MAX_GROUP_LINES) instead of the opener, so an opener found near
the widened backward limit no longer consumes the forward budget and drops
the path, headers, or body that follow the host.
scan_npm_packages.py: _outbound_host_evidence now records every outbound
context form for a host (URL, fetch-context, host-config), claiming each
non-overlapping match in form order, so a separate host-config request added
beside an already-baselined URL changes the evidence and reopens the key.
The common single-context case keeps its existing snippet.
scan_packages.py: follow a matched Python call over its continuations up to a
separate _MAX_CALL_LINES (40), decoupled from the 12-line display threshold,
so a multi-line requests.post( binds its whole argument list in the digest
and a changed body deep in the call reopens; bounded so a miscounted bracket
cannot swallow unrelated code. No baseline change: the current dependency set
has no matched call that closes between 13 and 40 lines, confirmed by a
Python 3.12 regenerate that produced a byte-identical baseline.
* scan: clamp npm depth, pin large bundles, follow backslash and bound .pth dump
Four fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: clamp the backward opener scan at depth 0 so a leading
unmatched closer (a preceding block whose opener is outside the backward
window) no longer drives depth negative and masks the real enclosing opener
that follows; a host-config object after such a block now binds and a changed
path reopens.
scan_packages.py: a large JS bundle now pins its whole content even when
another JS heuristic already fired. The bundle digest was only added when no
other finding existed; it is now appended to every finding's evidence on a
large bundle, so an unchanged obfuscation signature no longer lets changed
payload elsewhere ride the matched-line key.
scan_packages.py: _logical_line_end follows explicit backslash line
continuations, so a call split with a backslash before its parenthesis binds
the continuation line (URL/body) instead of returning at the zero-depth API
line.
scan_packages.py: the catch-all .pth import evidence is bounded through
_cap_line (prefix plus a digest of every line) so a large .pth of benign
imports cannot dump the whole member into the logs or baseline while an
appended or swapped import still reopens.
Baseline regenerated with Python 3.12: key set unchanged; one entry
(unsloth-zoo compiler.py) gains the backslash-continued banner lines now
bound by the continuation fix.
* scan: handle multi-line strings, lifecycle bodies, and de-quadratic evidence
Addresses a review round plus a performance audit of the evidence extractor.
Correctness (fail-closed):
- Bind the UNION of the single-line-blanked and multi-line-blanked bracket spans
in both scanners. The multi-line view blanks a triple-quoted Python string or a
backtick template literal that spans lines, so a `)` inside such a string no
longer closes the enclosing call early and drop later arguments. The single-line
view still counts a payload embedded INSIDE a string, so a dropper that hides a
call in a string keeps its argument lines bound. Taking the larger span never
shrinks the binding below either view, avoiding a fail-open regression.
- cred-env-in-lifecycle now pins the whole lifecycle script body via a digest, so
a changed non-token line (e.g. adding a curl exfil beside the token reference)
reopens, not just a change on the token line.
Performance / DoS (the scanner runs on attacker-controlled package files up to the
64 MiB / 16 MiB member caps, with no per-file time budget):
- _extract_evidence precomputes newline offsets once and maps match offsets with
bisect, removing the O(matches) whole-file content.count per match that made the
finditer fallback quadratic (a crafted minified file went from ~13 s/MiB and
hours at the cap to linear).
- npm _index_text splits and string-blanks the file once per evidence call instead
of per match (was O(matches x file) time and allocation).
- Bound evidence output: _MAX_EVIDENCE_SPANS (Python) and _MAX_EVIDENCE_MATCHES
(npm) fold the remainder into a digest so a file with thousands of matches cannot
build a multi-megabyte evidence/baseline blob while an added/removed match past
the cap still changes the key.
- _outbound_host_evidence caps matches per form and bounds the overlap claim so a
host repeated many times cannot make it quadratic.
No baseline change: a Python 3.12 regenerate is byte-identical (the union equals the
legacy single-line span for every current dependency file; the cap thresholds sit
above the largest real entry), so these are forward-looking hardening with no drift.
* scan: count all overflow matches, bind their context, blank JS regex literals
Follow-ups on the evidence output caps from the previous commit.
- _outbound_host_evidence no longer truncates each pattern's match iterator with
islice; it iterates every match and runs the overlap dedup only while the
display list is below the cap (so claimed stays bounded and the check is O(cap)
per match, not quadratic), folding every match past the cap into the overflow
digest. A host context beyond the 64th is counted again, so it reopens.
- The overflow digest (both scanners, via a shared _overflow_digest) binds each
overflow match's logical-line context, not just the regex match text, so a
changed payload on an over-cap line reopens even with the matched token
unchanged.
- The multi-line JS blanked view now blanks regex-literal bodies (tracking the
previous significant char for regex-vs-division and char classes for a literal
`/` inside `[...]`), so a `)` inside `/)/` no longer closes an outbound call
early. The bound span is the union of the single-line and multi-line views, so
an imperfect regex decision only ever grows the span, never shrinks it.
- The Python overflow digest canonicalizes spans (strips L<NN>: markers via
_canon_evidence) before hashing, restoring line-shift stability for the
over-cap region.
No baseline change: the overflow branches only trigger above the per-finding caps
(above the largest real entry), and the npm baseline is empty, so a Python 3.12
regenerate is byte-identical.
* scan: refresh baseline for ipython interactiveshell.py span drift
A newer ipython release changed the filesystem-enumeration span in
IPython/core/interactiveshell.py, so its content digest no longer matched the
baselined evidence and the studio scan shard flagged it as a non-baselined
CRITICAL. Regenerated with Python 3.12: only the ipython entry's evidence_hash
changes; the package/file/check key set is unchanged, and a studio enforcing
spot-check exits 0.
* Bound scanner evidence memory: stream overflow spans and cap lifecycle baseline size
scan_packages.py: _extract_evidence no longer materializes a rendered span
per match before slicing at the display cap. Once out holds _MAX_EVIDENCE_SPANS
spans, further spans fold straight into a running digest, so a minified or
padded file with hundreds of thousands of matching lines keeps memory bounded
to the display cap instead of the match count. The fold reproduces
_canon_evidence(" | ".join(overflow)) byte for byte, so the overflow digest and
every baseline key are unchanged.
scan_npm_packages.py: lifecycle-fetch-exec and cred-path-in-lifecycle stored the
entire install script body as evidence, so --write-baseline on a package with a
multi-MiB lifecycle script bloated the baseline JSON. Both now store a bounded
matched snippet plus a body-sha256 digest, matching cred-env-in-lifecycle. The
digest still binds the whole body, so a change to any line reopens the finding.
Adds tests for the streamed overflow bound and the bounded-but-reopens lifecycle
evidence. Baseline unchanged (byte-identical Python evidence; npm baseline empty).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make npm bracket-group scan order-aware so a same-line close-then-open binds
_scan_group counted brackets with a per-line net (opens minus closes), which
collapses intra-line order: a line that closes a prior block and then opens the
host-config object on the same line, e.g. `}); const opts = {`, nets to <= 0, so
the trailing `{` was dropped and the group started at the hostname line. A
changed path/headers on the following lines then hashed to the same evidence and
could ride an existing baseline key.
Replace the net count with an order-aware (L, R) reduction per line (L closers
needing an opener to the left, R openers needing a closer to the right) and apply
it in order in both the backward and forward scans, clamping stray closers at 0.
The trailing opener now stays visible so the whole object binds and a changed
payload reopens. Per-line cost is unchanged (one C-level bracket findall), so the
existing outbound-host evidence is byte-identical on all prior shapes; only the
previously-dropped same-line case changes. Adds a regression test for it.
* Harden scanner evidence: bound memory and bind Python call tails fail-closed
Five fixes across both scanners, none of which change the committed baseline (a
full regen of all three pip shards produced a byte-identical 185-key set).
scan_npm_packages.py: _evidence and _outbound_host_evidence collected every regex
match into a list before applying the 64-match display cap, so a text file under
the size cap that repeats a cheap signal (such as NPM_TOKEN) millions of times
could allocate a huge list of re.Match objects and stall or OOM before the
overflow digest ran. They now stream from finditer and fold overflow as matches
arrive via a shared _fold_overflow_match helper, byte-identical to the prior
digest.
scan_packages.py:
- _extract_evidence kept inserting every unique over-cap span into the seen set
even after it stopped appending to the display list, so a generated file with
millions of one-line matches still grew that set unbounded. It now tracks spans
only while filling the display list (per-line spans are unique by line number,
so dropping them past the cap cannot miss a dedup).
- _scan_line_end counted brackets with a per-line net, so a continued statement
that closes on the same line it opens a flagged call (a leading "]" before
"requests.post(") had the call's open paren cancelled and bound only the opener
line. It now applies brackets in order via _bracket_lr (leading closers clamp at
0), matching the npm bracket fix.
- a single-quoted string continued by a trailing backslash was not tracked across
lines, so a close paren inside the continued string on the next line closed the
call early; _blank_code_strings now carries the continuation.
- a call with more argument lines than the soft cap was hashed only through the
cap, so a changed data=/headers tail past it stayed suppressed; a closing call
is now followed to its real close under a 200-line hard limit (a never-closing
opener still stops at the 40-line soft cap so it cannot swallow the file).
Adds regression tests for each. npm baseline is empty; the Python baseline is
unchanged (verified byte-identical by regenerating all three shards).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bind giant DOTALL span anchors and add context to constant IOC evidence
Two fail-closed gaps where a changed payload could keep the same evidence hash
and stay suppressed by the baseline.
scan_packages.py: a giant greedy DOTALL span (a cross-line IOC match bridging
more than 60 lines, e.g. RE_TEMP_EXEC matching a /tmp line and a much-later
subprocess line) was dropped entirely once the per-line pass had any match, so an
appended cross-line payload -- a new /tmp line plus a later subprocess line that
share no single line, so the per-line pass never binds them -- produced the same
evidence and rode the key. The span is no longer dropped: it is bound by its head
and tail anchor lines plus a digest over just those (no line numbers, so a pure
line shift is stable). An added or moved anchor reopens the finding, while churn
in the bridged interior stays stable, so this does not reintroduce whole-file
drift. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry
such a span and are refreshed; a full three-shard regen confirmed only those two
keys change.
scan_npm_packages.py: known-ioc-string and cred-surface-host (always-bad) recorded
only the bare needle/host as evidence, so a reviewed tarball that kept the IOC
string while altering the adjacent fetch/exfil body produced an identical key.
They now bind matched-line context: known-ioc-string via the matched line and its
bracket-group continuation, cred-surface-host (always-bad) via the outbound call
context (path/headers/body, falling back to the bare host when not in an outbound
call). A changed payload on the same call now reopens.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
only the two giant-span entries.
* Hash giant-span interiors, bind exec/eval trigger, JS content, intra-literal whitespace
Four fail-closed gaps where a changed payload could keep the same evidence hash.
scan_packages.py:
- A giant bridged DOTALL span was bound only by its head and tail anchors, so a
cross-line payload inserted into the bridged interior between unchanged outer
anchors kept the same key. The whole span content is now digested (via _render),
so any interior change reopens; a pure line shift stays stable because the digest
is over the markerless code. Two baseline entries (multiprocess test, unsloth-zoo
scanner file) carry such a span; with full-interior binding, multiprocess
resolved at two versions across shards now yields two distinct entries where the
anchor digest had collapsed them into one.
- The exec/eval-with-hidden-payload findings omitted the visible exec/eval line
that makes the hidden string executable, so flipping a harmless eval("1+1") to
exec(__doc__) kept the same key while arming the payload. The trigger line from
the real-code view is now bound into the evidence.
- check_js_file extracted evidence with the Python-string-aware extractor, which
does not blank JS backtick template literals, so a template containing a close
paren closed a call's bracket span early and omitted later option/body lines. The
full file content digest is now pinned to every JS finding (not just large
bundles), binding the whole call.
scan_npm_packages.py: the evidence canon collapsed all whitespace via split(),
erasing whitespace inside JS string literals along with harmless indentation, so a
changed request body 'a b' -> 'a b' kept the same key. A new _canon_preserve_strings
collapses whitespace only OUTSIDE string literals (reindent-stable) while preserving
it INSIDE single/double/backtick literals (intra-payload edits reopen). Used for the
evidence hash and the logical-line digests.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
the two giant-span entries and adds the second multiprocess version's entry.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths
Follow-up hardening on the now-blocking scanners so the enforcing gate cannot
report clean while a malicious artifact goes unscanned.
scan_packages.py
- Hidden payload: also flag a network call AND an os/subprocess exec that live
only in a blanked docstring/string of an exec/eval file (the fetch-then-run
shape of an exec(__doc__) dropper). Either alone in real code was already
covered; hidden together they are the payload.
- Pinned releases fail closed: _release_files no longer falls back to the latest
artifact when a pinned version is missing or empty, so a yanked/bad pin is an
error instead of a different file being scanned in its place.
- requires_dist is read from the pinned release's metadata, not the project-level
(latest) document, so a sdist-only pin follows its own dependency tree.
- Environment markers are evaluated (PEP 508) instead of dropping any marker that
merely contains the word extra, so default-true markers like extra != 'dev' are
kept; conservative fallback keeps a dep on any uncertainty.
- Transitive recovery is a depth-bounded worklist: a wheel dependency whose own
child is sdist-only is fetched (--no-deps) and scanned, then its children are
recovered in turn, rather than being silently skipped.
scan_npm_packages.py
- Baseline keys use the package-relative path instead of the basename, so the
same basename in a different directory is not over-suppressed.
Tests cover each case; full scripts pass AST and ruff checks.
* Address review: tighten marker scope, decoy-proof the dropper check, fail closed on missing pin metadata
- Markers: keep any dep whose marker can hold on another install target
(sys_platform == 'win32', python_version == '3.13'); only drop a marker that
depends solely on extra and is false with no extra. A scanner runs on one
target but must cover code installed on others. Pure-extra markers are
evaluated against default_environment() with extra unset.
- Hidden dropper: the network+exec docstring check now inspects the removed
(blanked) span directly, so a benign visible network or subprocess call cannot
mask a payload that still lives in a docstring. Carrier checks stay
blanked-only (an in-code carrier is already caught by the normal check), so
corpus findings are unchanged.
- requires_dist: a pinned version whose own metadata cannot be fetched recovers
nothing rather than substituting the latest release's dependency tree.
- Transitive recovery: the last-ditch direct-sdist branch also chases the
recovered package's declared deps, matching the other branches.
- npm baseline: schema bumped to v2 (package-relative keys); a pre-v2 baseline
with entries is ignored (fail closed) instead of mis-applying basename keys.
Tests cover each case; scripts pass AST, ruff, and the import-hoist verifier.
* Scanner: exclude comments from hidden-payload check, flag missing pin metadata as incomplete
Hidden network+exec detection now inspects only docstring/string spans (what exec(__doc__)/exec(<str>) can actually run), so a real exec() beside comments that mention a network and a subprocess call no longer false-positives. Missing pinned-release metadata in transitive recovery records a download_error so the --with-deps path fails closed instead of treating it as no dependencies. Adds regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Package scanners: cut false positives and make the CI gate blocking
scan_packages.py and scan_npm_packages.py red-failed on legitimate
library code, so the security-audit steps were left advisory. Reduce
the false positives at the source and flip both gates to blocking.
scan_packages.py:
- Scan code only: blank comments and bare docstrings/doctests before
matching (line numbers preserved), so prose and >>> examples cannot
trip a finding.
- Drop the platform.system() branch from the anti-analysis regex (under
DOTALL it matched across the whole file, so every cross-platform
library tripped it) and fix the dead /proc/self/status alternative.
- Add a reviewed baseline allowlist (scan_packages_baseline.json) keyed
on (package, basename, check): only non-baselined CRITICAL/HIGH exit
1, and a new kind of finding in a listed file still fails.
- sdist fallback: when --with-deps cannot resolve a shard (a sdist-only
package or a version conflict), drop to per-spec and fetch the raw
sdist from the PyPI JSON API (no pip build, no setup.py), so every
package is still scanned and no shard exits 2.
scan_npm_packages.py:
- Mirror the code-only JS/TS scanning (blank // and /* */ comments,
string/template/regex aware) and the baseline allowlist. The npm
corpus is clean today, so the baseline is empty.
security-audit.yml:
- Flip both scan steps to blocking (SCAN_ENFORCE=1), capturing the
scanner exit via PIPESTATUS so tee does not mask it.
tests/security: add coverage for the strip, baseline and sdist paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review feedback on the package scanners
- Do not blank f-strings during code-only scanning (they evaluate at
import); and when a file uses exec/eval, rescan the original for
payload carriers hidden in a docstring/string so exec(__doc__) style
payloads stay visible.
- sdist fallback: recover transitive deps with their version specifier
(fetch the pinned version, not latest), and recover deps in the
--no-deps branch too so a sdist-only transitive dependency is still
scanned instead of silently skipped.
- Baseline: key by package-relative path, not basename, so a future
same-named file in another directory is not auto-suppressed.
Regenerated the baseline accordingly.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* scripts/scan_*: add Mini Shai-Hulud May-12 IOC strings and pin-blocklists
Append the May-12 2026 wave indicators (git-tanstack.com, transformers.pyz,
/tmp/transformers.pyz, "With Love TeamPCP", "We've been online over 2 hours")
to all three scanner IOC tables, add BLOCKED_NPM_VERSIONS (42 TanStack pkgs,
4 opensearch versions, 3 squawk pkgs) in scan_npm_packages.py and
lockfile_supply_chain_audit.py (kept byte-identical), add BLOCKED_PYPI_VERSIONS
(guardrails-ai 0.10.1, mistralai 2.4.6, lightning 2.6.2/2.6.3) plus
RE_MAY12_IOC wiring across check_py_file/check_shell_file/check_workflow_file
in scan_packages.py. The npm orchestrator and the lockfile auditor now
short-circuit on a blocked entry before fetching the tarball, and the
PyPI download pipeline drops blocked specs before pip download is invoked.
* tests/security: regression suite for supply-chain scanners
Adds offline fixture corpus and pytest coverage for scan_npm_packages,
scan_packages, and lockfile_supply_chain_audit so future IOC-table
drift surfaces at PR time. Pytest scope narrowed to tests/security so
GPU smoke tests are not picked up by default.
* ci(security-audit): drop continue-on-error on pip-scan and npm-scan jobs
Promote three harden-runner blocks to egress-policy: block with per-job allowlists.
Add tests-security job running pytest tests/security as a hard gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts: harden third-party downloads, pip resolver pins, atomic writes
Pins uv installer and mlx_vlm qwen3_5 patches by commit SHA + SHA-256
checksum, scrubs PIP_* env vars and forces --index-url + --only-binary
on pip download, applies tarbomb caps to scan_packages archive walks,
and converts non-atomic config writes (kwargs spacer, studio stamper,
notebook validator, scan_packages req-file fixer) to mkstemp+os.replace.
Also adds host allowlist to notebook_to_python downloader, threads an
--allow-shell flag through its shell=True emission with reviewer warning
comments, locks both MLX installer scripts to set -euo pipefail, and
extends CODEOWNERS so colab snapshot data files require notebook-owner
review.
* ci(workflows): harden release-desktop / smoke / notebooks workflows
Pin dtolnay/rust-toolchain to a 40-char SHA, scope release-desktop permissions to read at workflow level with job-level write only on the build job, append --ignore-scripts to every npm ci / npm install in studio-frontend-ci / wheel-smoke / studio-tauri-smoke / release-desktop, validate client_payload.ref shape via an env-var-isolated regex on every notebooks-ci job, and add step-security/harden-runner in audit mode as the first step of release-desktop and mlx-ci.
* scripts: promote silent scanner failures to non-zero exit codes
scan_packages now returns 2 on pip-download failure and emits a CRITICAL archive_corrupted finding on truncated wheels/sdists.
notebook_to_python exits 1 on per-notebook failures; notebook_validator wraps the stash/pop in try/finally; lockfile audit rejects bare UNSLOTH_LOCKFILE_AUDIT_SKIP=1 with a loud GitHub Actions warning.
* Add npm cooldown + new-install-script gate + Dependabot cooldown
Pins min-release-age=7 (npm 11.10+) in repo-root and studio/frontend
.npmrc, adds scripts/check_new_install_scripts.py to fail PRs that
add a postinstall dep, ships a new security-audit job for npm audit
signatures plus the diff, and extends .github/dependabot.yml with
cooldown stanzas. Pin @tanstack/react-router to 1.169.9 per GHSA-
g7cv-rxg3-hmpx; lockfile regen deferred until that release lands on
npm. tests/security gains 4 new tests; full suite 26/26 green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(security): fix tanstack pin, exec bits, expand IOC tables to @uipath/@squawk full
- Revert --ignore-scripts on Studio install workflows: vite build needs
esbuild's native postinstall (per PR #5392 rationale). Keep
--ignore-scripts on security-audit.yml's standalone npm audit job.
- Pin @tanstack/react-router to the actual published 1.169.2 (was a
forward-looking 1.169.9 that does not exist on npm; broke npm ci).
- Drop redundant repo-root .npmrc; studio/frontend/.npmrc covers the
only npm project today (root cooldown re-instate via dependabot.yml).
- Restore exec bits on 7 files my filesystem stripped during cherry-pick.
- Expand BLOCKED_NPM_VERSIONS with full safedep.io + Aikido enumeration:
22 @squawk/* packages with 5 versions each (110 entries; previously
3 entries with 1 version each), and 66 @uipath/* packages (entirely
missing before). Mirror in scripts/lockfile_supply_chain_audit.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/security: suppress CodeQL py/incomplete-url-substring-sanitization
The two flagged 'X' in Y assertions are NOT URL sanitization checks.
They verify our scanner WROTE a known IOC literal into its stdout /
Finding.evidence, which is the opposite of an attack surface --
matching the scanner's output is precisely what catches the worm.
Inline lgtm[] suppression with a 4-line rationale comment above each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts/scan_*: expand IOC tables with Aikido full 169-pkg enumeration
Per Aikido 2026-05-12 disclosure (373 malicious package-version entries
across 169 npm package names), add to BLOCKED_NPM_VERSIONS:
- @mistralai/* npm scope (3 packages, 9 versions) -- separate from
the PyPI mistralai package already in BLOCKED_PYPI_VERSIONS
- @tallyui/* (10 packages, 30 entries)
- @beproduct/nestjs-auth (18 versions 0.1.2..0.1.19)
- @draftlab/* + @draftauth/* (5 packages)
- @taskflow-corp/cli, @tolka/cli, @ml-toolkit-ts/*, @mesadev/*,
@dirigible-ai/sdk, @supersurkhet/*
- 10 unscoped packages (safe-action, ts-dna, cross-stitch,
cmux-agent-mcp, agentwork-cli, git-branch-selector, wot-api,
git-git-git, nextmove-mcp, ml-toolkit-ts)
Also add to KNOWN_IOC_STRINGS / NPM_IOC_STRINGS:
- router_init.js SHA-256 ab4fcadaec49c03278063dd269ea5eef82d24f2124a8e15d7b90f2fa8601266c
- tanstack_runner.js SHA-256 2ec78d556d696e208927cc503d48e4b5eb56b31abc2870c2ed2e98d6be27fc96
- bun run tanstack_runner.js marker (the new Bun-prepare-script
dropper invocation pattern unique to this wave)
Total: 170 packages, 401 versions blocklisted. Studio lockfile still
scans clean (0 findings, 0 hard errors).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts/scan_*: web-verification additions (@tanstack/setup, intercom-client)
Two findings from cross-checking BLOCKED_NPM_VERSIONS / KNOWN_IOC_STRINGS
against GHSA-g7cv-rxg3-hmpx + Aikido + safedep.io + Socket + Semgrep.
- Fix asymmetry: @tanstack/setup IOC string was in
lockfile_supply_chain_audit.py's NPM_IOC_STRINGS but missing from
scan_npm_packages.py's KNOWN_IOC_STRINGS. The literal is the malicious
optional-dependency name used by the May-12 TanStack wave; no
legitimate npm package of this name exists.
- Add intercom-client@7.0.4: the npm counterpart of the lightning
2.6.2/2.6.3 PyPI compromise (Apr-30 wave). Same threat actor
(TeamPCP). Confirmed by Semgrep, Aikido, OX Security, Resecurity,
Kodem. Safe version is 7.0.3 and earlier.
Total BLOCKED_NPM_VERSIONS: 171 packages / 402 versions. Both files
remain byte-identical. Studio lockfile still scans clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(security): add workflow-trigger lint refusing pull_request_target + cache-poisoning vectors
The two patterns that together powered GHSA-g7cv-rxg3-hmpx (TanStack
Mini Shai-Hulud) are now gated at PR time:
1. pull_request_target -- the worm chain started with a fork PR that
ran in the base-repo context. Every workflow in this repo today
uses 'pull_request' (safe); the lint refuses any new
pull_request_target additions outright. workflow_run is
restricted, allowed only with an explicit allow-comment.
2. Shared cache keys between PR-triggered workflows and the publish
workflow (release-desktop.yml). The TanStack attack chain poisoned
a shared Actions cache from a fork PR; the legitimate release
workflow then restored the poisoned cache. The lint refuses any
cache key that appears in both a PR-triggered workflow and a
workflow_dispatch-only / publish workflow.
Current tree is clean: 0 pull_request_target, 0 workflow_run, 0
PR-publish cache-key collisions across all 24 workflows. The lint
locks that invariant in place.
Files:
+ scripts/lint_workflow_triggers.py (~200 LOC, stdlib + PyYAML)
+ tests/security/test_lint_workflow_triggers.py (5 tests covering
current-tree pass, pull_request_target reject, workflow_run
restricted, justified workflow_run accept, cache-key collision
reject)
~ .github/workflows/security-audit.yml: new workflow-trigger-lint
job, no continue-on-error, harden-runner block-mode, PyYAML only
runtime dep.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* security: fix tests-security CI job + CodeQL false-positives
Two CI failures on the prior push:
1. pytest tests/security -- 5 lint regression tests failed because
scripts/lint_workflow_triggers.py imports PyYAML which is not in
the bare runner's Python env. Added pyyaml==6.0.2 to the pip
install step alongside pytest. (29 scanner tests already passed.)
2. CodeQL py/incomplete-url-substring-sanitization fired on two
test assertions that check the scanner WROTE the IOC literal
to its own stdout/stderr. The rule pattern-matches on
`"<host>" in <var>` and cannot distinguish a URL sanitizer from
a regression-test evidence check. Previous `# lgtm[...]` inline
suppressions were detached from the operator when pre-commit
reformatted the assert across multiple lines. Rebuilt the IOC
literals at runtime (`"git-tanstack." + "com"`) so no URL-shaped
source literal appears on the `in` operator line; rule cannot
trigger.
Verified locally: `pytest tests/security -v` -> 34 passed in 2.70s.
* security(studio): defensive .npmrc cooldown aliases + save-exact
Two additions to studio/frontend/.npmrc to harden the existing
`min-release-age=7` (Mini Shai-Hulud defence):
1. `minimum-release-age=10080` (minutes) -- defensive alias for the
same 7-day floor. Some npm versions / wrappers consult one key but
not the other; setting both prevents a single upstream setting-name
parse change from silently disabling the cooldown. The two keys
MUST agree (do not let them drift).
2. `save-exact=true` -- refuses to write back `^x.y.z` ranges into
package.json when a maintainer runs `npm install <pkg>` locally.
Does NOT rewrite already-present ranges; stops NEW carets from
creeping into the manifest as patch-version footguns.
Verified: pytest tests/security -> 34 passed in 2.63s.
* chore(dependabot): remove dead bun entry for /studio/frontend
`package-ecosystem: "bun"` at /studio/frontend was a no-op: that
path commits package-lock.json, not bun.lock / bun.lockb, so
Dependabot's bun ecosystem silently skipped it. The actual
behaviour is unchanged -- the npm entry below the cargo block
already owns npm_and_yarn security advisories for /studio/frontend
with `open-pull-requests-limit: 0` (version-update PRs suppressed,
security PRs flow through).
This commit:
- Deletes the bun entry (kept a placeholder comment so a future
bun migration knows where to slot it back in).
- Rewrites the npm /studio/frontend entry comment to explain the
real intent: lockfile is the authoritative pin, .npmrc
`min-release-age=7` already blocks fresh tarballs at install
time, dependabot only needs to surface security advisories.
No functional change: same set of dependabot PRs as before (zero
version updates, security advisories grouped weekly with cooldown).
Verified: pytest tests/security -> 34 passed in 2.67s; YAML
parses cleanly via PyYAML.
* fix(dependabot): drop unsupported semver-* cooldown keys on github-actions
Dependabot's validator rejected the config with:
The property '#/updates/0/cooldown/semver-minor-days' is not
supported for the package ecosystem 'github-actions'.
The property '#/updates/0/cooldown/semver-patch-days' is not
supported for the package ecosystem 'github-actions'.
The `semver-minor-days` / `semver-patch-days` cooldown knobs are
only valid for semver-aware ecosystems (npm, cargo, etc.). The
github-actions ecosystem pins via git tags / SHAs, not semver, so
only `default-days` is honored. Pre-existing bug on main; surfaced
on this PR because the prior commit re-validated the file.
Behaviour: github-actions PRs now respect the 7-day cooldown floor
(was already the intent), without the no-op semver bands.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/ci: npm tarball content scanner (no-install, hostile-input safe)
Counterpart to scripts/scan_packages.py for the npm side. Pip-side
scanner reads requirements files, downloads PyPI archives via
`pip download --no-deps`, and pattern-scans them for malicious
shapes. This change adds the equivalent for npm tarballs.
Why
===
PR #5392 (lockfile_supply_chain_audit.py) catches injection-pattern
attacks where the malicious metadata lives IN the lockfile -- e.g.
the TanStack Shai-Hulud worm that injected an `optionalDependencies`
entry pointing at a GitHub commit. It does not catch the broader
class of "legit-registry tarball with malicious content but normal
lockfile metadata": attacker steals a maintainer's npm publish
token, publishes a malicious version to registry.npmjs.org with a
valid integrity hash, and the lockfile entry looks normal -- the
malicious code lives inside the tarball's dist/index.js or its own
postinstall script. Today that gap is covered reactively by `npm
audit` + OSV-Scanner once the GHSA lands; there is a real window
before that.
This scanner closes the window by inspecting tarball CONTENT.
What it checks
==============
For each entry in studio/frontend/package-lock.json:
1. Download the tarball directly from registry.npmjs.org. Refuse
any non-allowlisted URL. Stream-bounded at 64 MiB.
2. Verify SHA-512 integrity against the lockfile entry BEFORE
opening the tarball.
3. Safely extract into a sandboxed temp dir behind guards:
- reject symlinks / hardlinks (LNKTYPE, SYMTYPE)
- reject absolute paths and `..` traversal
- reject character / block / FIFO devices
- per-file size cap 8 MiB, cumulative cap 128 MiB,
member count cap 50000
- stream open (mode='r|gz') so we abort mid-extract
- extracted files set to non-executable mode (0o644)
4. Pattern-scan the extracted text content for:
- lifecycle (preinstall/install/postinstall/prepare) scripts
in any package.json that fetch + pipe-to-shell external
content -- the install-time RCE vector
- optionalDependencies pointing at github: / git+ / git:
(TanStack worm injection shape)
- C2 / exfiltration hosts: getsession.org, 169.254.169.254
(IMDS), 169.254.170.2 (ECS), metadata.google.internal,
vault.svc.cluster.local, k8s ServiceAccount token paths,
ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN, npm publish-token
enumeration endpoint
- credential paths a frontend lib should never read:
~/.npmrc, ~/.aws/credentials, ~/.ssh/id_*, /.kube/config,
/.docker/config.json
- JS regex: Function/eval against base64-decoded payload,
process.env.GITHUB_TOKEN / NPM_TOKEN / AWS_* access in
package source
- obfuscation: large base64-ish blob (>=2 KiB) fed into
Function or eval (router_init.js dropper shape)
- literal IOC substrings from public advisories
Safety
======
Threat model: every tarball is hostile. The scanner:
- never runs `npm install`, never executes anything from a
downloaded tarball, never calls subprocess on extracted content
- downloads only from registry.npmjs.org (defence-in-depth check
at parse time AND inside download_tarball)
- stdlib-only (no third-party deps -- adding one would itself
be a supply-chain liability)
- tempdir wiped via atexit on every termination path
- exit codes: 0 clean, 1 HIGH/CRITICAL finding, 2 internal error
Wiring
======
New job `npm-scan-packages` in security-audit.yml, parallel to
`pip-scan-packages`. Triggers same as the existing audits (PR on
manifest changes, push to main/pip, daily 04:13 UTC, dispatch).
Initially `continue-on-error: true` so the baseline can settle --
matches the existing convention for the other audit steps. Drop
that flag once the baseline is clean for a week.
Verified locally
================
- AST parse OK.
- Real-network 3-package smoke: 0 findings.
- Real-network 25-package smoke (Babel + assistant-ui surface):
0 findings, no hard errors.
- 9 fault-injection scenarios all pass:
1. zip-slip path traversal refused
2. symlink member refused
3. oversized member refused (size cap)
4. too-many-members refused (count cap)
5. router_init.js IOC + obfuscated-blob shape both detected
in synthetic malicious tarball
6. lifecycle fetch-exec in scripts.preinstall detected as
CRITICAL
7. AWS IMDS reference (169.254.169.254) detected
8. SRI integrity-parser accepts syntactically-valid SRI
9. download_tarball refuses non-allowlisted hostname
Refs
====
- https://tanstack.com/blog/npm-supply-chain-compromise-postmortem
- https://github.com/TanStack/router/issues/7383
- https://github.com/TanStack/router/security/advisories/GHSA-g7cv-rxg3-hmpx
- https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised
- https://www.stepsecurity.io/blog/mini-shai-hulud-is-back-a-self-spreading-supply-chain-attack-hits-the-npm-ecosystem
* scan_npm_packages: kill false positives + handle real native binaries
First CI run on PR #5393 (run 25710423126 / job 75489317395) hit
two false-positive classes plus one cap-too-tight class:
False positives (7 findings):
@langchain/core 1.1.44 ssrf.{cjs,js}: a SSRF *protection* module
that ships a literal blocklist `const CLOUD_METADATA_IPS = [...]`
of IMDS hosts as data the library REFUSES to dial. Our scanner
saw the IPs as substrings and flagged 6 of them.
object-treeify 1.1.33 package.json: a manual `docker` dev script
that mounts `~/.npmrc` and `~/.aws` for local containerised
builds. npm never runs `scripts.docker` automatically; it is
only invoked when a developer runs `npm run docker`. Our bare
substring scan flagged the `/.npmrc` reference anyway.
Cap-too-tight class (10+ findings):
next/swc, rolldown bindings, biome CLI, lightningcss, mermaid
sourcemap, typescript.js. The 8 MiB per-file cap was calibrated
for JS source and rejected legitimate precompiled native binaries
(next-swc .node is 137 MB) and CLI executables (biome is 25-33 MB).
Fixes
=====
cred-surface-host detection split into two tiers:
ALWAYS_BAD substrings have no legitimate use anywhere and still
bare-match: `registry.npmjs.org/-/npm/v1/tokens`,
`ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN`.
NEEDS_CONTEXT substrings (IMDS IPs, GCE metadata host, k8s
ServiceAccount path, Vault endpoint) require co-occurrence with
EITHER a fetch verb (fetch/axios/http.get/etc) within 200 chars
OR an `http(s)?://HOST` URL prefix OR a `host:`/`hostname:`
config field. A defensive blocklist literal does not match any
of those rules; an actual outbound call always does.
cred-surface-path detection moved out of the bare-text scan into
`scan_package_json` and scoped to the 4 NPM lifecycle hooks
(preinstall / install / postinstall / prepare). A `/.npmrc`
reference in a `docker` dev script is silent; a `cat ~/.npmrc
| curl ...` in a `postinstall` fires HIGH.
Per-file size cap split by content type, sniffed via 16-byte
magic header read (ELF / Mach-O / PE / WASM / archive formats),
plus suffix list (.node/.wasm/.so/.dll/.dylib/.exe), plus regex
for versioned shared libs (libfoo.so.8.17.3), plus a null-byte
ratio fallback for extensionless binaries that headers do not
catch.
Text files: 16 MiB cap (still tight; typescript.js at 9.1 MB is
the legitimate ceiling).
Binary files: 256 MiB cap (next-swc .node is 137 MB; sharp
libvips is ~18 MB; rolldown bindings are 18-26 MB each).
Cumulative: 512 MiB per tarball.
Tarball: 256 MiB compressed.
Binary files are also skipped in the content scanner -- regex
over compiled machine code is noise. The IOC substring fallback
in `scan_extracted_tree` now uses the same magic-sniff to decide
whether to grep.
HTTP timeout bumped 30s -> 60s for large tarballs.
Verified
========
- AST parse OK.
- 11 fault-injection tests pass:
* zip-slip, symlink, oversized-declared-size, count-cap
* router_init.js IOC detected
* IMDS-in-URL still detected (new contextual rule)
* langchain SSRF blocklist no longer false-positive
* object-treeify docker script no longer false-positive
* lifecycle-script `cat ~/.npmrc | curl ...` detected
* synthetic ELF (extensionless executable) extracts and is
correctly skipped from text scan
* versioned `.so.8.17.3` shared lib extracts cleanly
- Real-network end-to-end on the full lockfile:
968 packages, 0 findings, 0 hard errors, 76 seconds.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Michael Han