Security audit fails on main with 1 unsuppressed CRITICAL:
CRITICAL C2 polling/beaconing loop detected
Package: fastapi
File: fastapi/routing.py
The same file and check are already baselined, but the entry is keyed on a
digest of the matched code, so fastapi 0.140.0 rewriting the block reopened it.
That is the baseline working as intended, not a stale pin, so the new code
needs its own review rather than a regenerated file.
The flagged block is the SSE keepalive inserter:
async def _keepalive_inserter() -> None:
async with send_keepalive, receive_stream:
try:
while True:
try:
with anyio.fail_after(_PING_INTERVAL):
data = await receive_stream.receive()
await send_keepalive.send(data)
except TimeoutError:
await send_keepalive.send(KEEPALIVE_COMMENT)
except anyio.EndOfStream:
pass
It forwards one in-memory anyio stream to another and emits a keepalive comment
when the read times out. No socket, no outbound host, no fetched command, and it
terminates on EndOfStream. The heuristic matches it on the shape alone, a loop
with a timeout and a send, so it is a false positive.
Adds that one entry. The existing fastapi entry stays, since the requirement is
unpinned and an older resolve still needs it.
Co-authored-by: danielhanchen <unslothai@gmail.com>
* fix: pin torchcodec for torch 2.10 and warn on ABI mismatch
Add unsloth[audio] extra with torchcodec>=0.10.0,<0.11.0 and emit a
clear warning when installed torchcodec minors disagree with torch
(unslothai/unsloth#7225).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(packaging): address Codex review on torchcodec/torch 2.10 compat (#7299)
- Postpone annotations so import_fixes loads on Python 3.9
- Align TORCH_TORCHCODEC matrix with upstream (2.9: 0.8/0.9, 2.8: 0.6/0.7)
- Fix mismatch hint upper bound (<0.11.0) and gate audio-torch210 suggestion
- Split audio extra per torch minor; gate torch210 pin behind python>=3.10
- Bundle audio-torch210 only in *-torch2100 install extras
* fix(security): refresh openai CRITICAL scan baseline hashes (#7299)
openai package code drift reopened five CRITICAL findings in the
extras pip-scan-packages shard (C2 loop body hashes + IMDS/network
evidence). Update the reviewed allowlist evidence/hashes so CI gates
on new findings only, not benign SDK churn.
* chore: retrigger CI after baseline refresh (#7299)
* chore: touch scan baseline comment to retrigger security audit (#7299)
* Guard torchcodec version parsing so bad version strings cannot break import
* Bundle audio pin into intel-gpu-torch210 and guard the mismatch warning
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
The pip scan-packages extras shard has been red on main because openai
2.47.0 changed the code inside five previously baselined findings, so
their evidence hashes no longer matched the allowlist. The hf-stack
shard was about to go red the same way: unsloth-zoo 2026.7.5 changed
two baselined test files.
All seven reopened findings were re-verified against the exact resolved
archives before re-baselining:
- openai/_base_client.py: while True in SyncPage.iter_pages, the
pagination iterator.
- openai/auth/_workload.py: Azure IMDS and GCP metadata token
providers for the documented workload identity federation feature.
- openai/resources/{beta/responses,realtime,responses}: while True in
websocket __aiter__ event loops; the loop bodies gained reconnect
handling in 2.47.0, which is what shifted the hashes.
- unsloth-zoo tests/test_vision_collator_audio.py: asserts that an
inline /tmp/a.wav path is passed through by the audio collator.
- unsloth-zoo tests/test_gemma4_forced_float32_ple_dtype.py:
compile()/exec() of the project's own generated Gemma4 PLE cast
helper source in tests.
No existing entries were removed. All three shards now exit 0 locally
against the same requirement sets CI uses.
The enforcing pip scan-packages hf-stack shard fails on two CRITICAL
staged-dropper findings in unsloth-zoo test files:
tests/test_mlx_save_export_regressions.py and
tests/test_vision_collator_audio.py. Both are false positives: the
combination heuristic matches a /tmp path literal alongside unrelated
subprocess/import references in the same file, but those are mocked test
fixtures (monkeypatch.setattr on subprocess, asserted /tmp path strings),
not droppers. Add both to the reviewed allowlist so the gate stops
red-failing on legitimate test code. The scan then exits 0 on both the
hf-stack shard and a direct unsloth-zoo scan.
The pip scan-packages studio shard is red on main and on every open PR:
the baselined fastapi finding (the benign SSE keepalive `while True:`
loop in fastapi/routing.py, reviewed and suppressed long ago) records
its evidence at L586 with the span digest of the fastapi release current
at baseline time. The latest fastapi shifts that loop to L587 and its
span digest with it, so the evidence hash no longer matches and the
scanner reports the finding as new, failing the shard with one
unsuppressed CRITICAL.
Re-reviewed the flagged code in the current release before refreshing:
L587 is the same keepalive loop inside the streaming response machinery,
not a beacon. Only the one entry's evidence and evidence_hash change.
Verified with the scanner itself: `scan_packages.py fastapi
--no-baseline` reproduces the exact CI evidence string, and with the
updated baseline the same scan exits 0 with the finding suppressed as
1 CRITICAL baselined.
The supply-chain scan gates on non-baselined CRITICAL/HIGH findings. A newer
sentencepiece release reindented the stdout/stderr fd-redirect helper in
sentencepiece/__init__.py (the os.dup2 pair the heuristic flags as a
reverse/bind-shell pattern), moving it from L1221/L1226 to L772/L777 and
changing its leading indentation.
The baseline key is (package, package-relative file, check, evidence_hash),
where evidence_hash is over the matched code with the L<NN>: markers stripped
but the code's own indentation preserved. The reindent therefore changed the
hash (bba233.. -> 65b5a11c..), so the existing entry no longer suppressed the
finding and it resurfaced as a blocking CRITICAL in the hf-stack and studio
scan legs.
Add the new indentation variant to the allowlist. The calls are sentencepiece
redirecting stdout/stderr file descriptors to capture its C++ logs, not a
shell; no socket or networking is involved. The old L1221 entry is kept so
both versions stay covered.
New releases of huggingface-hub (1.23.0) and openai (2.45.0) shifted or
added polling loops that the C2 polling/beaconing check flags, failing
all three pip scan-packages shards (studio 1, hf-stack 1, extras 3 new
CRITICAL findings) org-wide including on main.
Regenerated with scan_packages.py --write-baseline per CI shard (same
shard-to-requirements mapping and --with-deps as security-audit.yml)
and merged. All entries were manually reviewed at the resolved versions:
- huggingface-hub hf_api.py: create_repo 409-concurrency retry loop
body changed in 1.23.0; refreshed evidence hash. The loop POSTs to
the canonical Hub endpoint and retries only on a specific conflict
error. Benign client retry.
- openai beta/threads/runs/runs.py: create_and_poll run-status helper
refactored in 2.45.0 (Assistants deprecation annotations); refreshed
evidence hash. Documented polling helper against api.openai.com.
- openai beta/responses/responses.py: new beta websocket client whose
__aiter__ yields server events until the connection closes. New
entry; standard event-stream iterator, not beaconing.
- openai resources/responses/responses.py: evidence line number
refreshed only, hash unchanged.
The two dropped entries are the pre-refactor hashes of the same two
loops above; they no longer occur at the resolved versions. Verified
locally: all three shards exit 0 with 0 unsuppressed CRITICAL/HIGH
(hf-stack 120, studio 151, extras 99 suppressed).
* scripts: refresh scan_packages allowlist baseline
Regenerate scripts/scan_packages_baseline.json against the current
resolved dependency set so the blocking pip scan-packages gate matches
what the scanner now finds. Refreshes evidence hashes for benign
findings whose code shifted lines (unsloth-zoo mlx loader, gguf/mlx
test /tmp fixtures) and adds two mainstream-library entries that were
newly surfaced (torch inductor codecache base64+subprocess compile
cache, torch testing common_utils socket import). Stale entries whose
matching code changed and no longer triggers are dropped.
All entries remain CRITICAL/HIGH findings manually judged benign;
matched on (package, file, check, evidence_hash).
* ci(security-audit): re-run scan when the allowlist baseline changes
The security-audit pull_request trigger listed the scanners but not
their allowlist baselines, so a baseline-only edit never re-ran the
scan that consumes it. A refreshed baseline could therefore merge
without CI confirming its evidence hashes match what the scanner finds.
Add scan_packages_baseline.json and scan_npm_packages_baseline.json to
the paths filter so baseline changes are validated on their own PR.
* Studio: exclude mlx-lm 0.31.3 (broke gemma4/qwen3_5 QK-norm load)
mlx-lm 0.31.3 regressed the QK-norm archs: its strict load_weights rejects the
q_norm/k_norm tensors with "Received N parameters not in model", so gemma4 and
qwen3_5 checkpoints fail to load. Studio installs the MLX stack unpinned at
latest, which pulls 0.31.3. Verified on a real macos-14 runner: gemma4 fails to
load on 0.31.3 but loads and generates coherently on 0.31.2 and on git-main
(future 0.31.4). See mlx-lm #1242.
Exclude just that release (!=0.31.3) in the installer and the self-heal floor so
--upgrade still resolves to the newest good build, and treat an already-installed
0.31.3 as unsatisfied so the self-heal replaces it.
* Studio MLX: cover fresh-install path + robust bad-version compare
Address PR review:
- Fresh install.sh (Apple Silicon) runs the base 'uv pip install unsloth' with
SKIP_STUDIO_BASE=1, skipping the guarded MLX-stack step, so transitive
resolution could still pull mlx-lm 0.31.3. install.sh already exports
UV_OVERRIDE -> overrides-darwin-arm64.txt before that install, so exclude
mlx-lm 0.31.3 there too; this also strengthens the self-heal (same override).
- Match the known-bad version with parsed packaging.Version so 0.31.3 == 0.31.3.0
(trailing-zero normalization) instead of raw string equality.
* Studio: exclude mlx-lm 0.31.3 on the fresh Apple Silicon install too
The overrides file only applies via UV_OVERRIDE when it exists relative to the
script, which is not true for a curl-piped install, and the guarded MLX step in
install_python_stack.py is skipped there (SKIP_STUDIO_BASE=1). So the base
install could still resolve the transitive mlx-lm to the broken 0.31.3. Append
mlx-lm!=0.31.3 to the base install on Apple Silicon (empty elsewhere), so the
fresh path pins away from 0.31.3 without waiting for the runtime self-heal.
* Studio: exclude mlx-lm 0.31.3 on the migrated install; keep the >=0.22.0 floor
The with-deps migrated install did not append ${_MLX_LM_EXCLUDE_ARG:-}, so a
curl-piped Apple Silicon migration (no repo overrides file, UV_OVERRIDE unset)
could resolve mlx-lm 0.31.3 transitively. Append the exclusion there, matching
the fresh install path. The no-torch migration is left alone since --no-deps
never resolves mlx-lm (same as the fresh no-torch path).
Also restore the >=0.22.0 floor in overrides-darwin-arm64.txt: a uv override
replaces the transitive constraint, so a bare !=0.31.3 could let the resolver
drop below the supported minimum that mlx_repair.py enforces at runtime.
* Triage huggingface_hub 1.22.0 / fastapi / multiprocess scanner false positives
The scan-packages gate red-failed on all three shards after transitive deps
bumped. Every new CRITICAL is a benign false positive, verified against upstream:
- huggingface_hub 1.22.0 added _sandbox.py for the remote HF sandbox feature.
Its job-startup bootstrap string (fetch sbx-server into the container /tmp and
exec it) and the SandboxPool host-reservation loop trip the staged-dropper and
C2-loop heuristics; that script runs inside a remote HF container, not on the
user machine. The bump also re-hashed the already-reviewed benign polling loops
in hf_api.py and utils/_http.py. The PyPI artifact is byte-identical to the
official v1.22.0 tag.
- fastapi 0.139.0 routing.py re-hashed the websocket keepalive while-True loop;
byte-identical to upstream 0.139.0.
- multiprocess 0.70.19 forkserver.py and tests/__init__.py re-hashed the AF_UNIX
fork-server IPC and fd-inheritance tests; genuine uqfoundation release, local
IPC not network.
Added 7 reviewed allowlist entries (no blind regenerate). All three shards
(hf-stack, studio, extras) exit 0 locally.
* Tighten mlx-lm 0.31.3 exclusion comments
* Trim mlx-lm 0.31.3 exclusion comments
The pip scan-packages gate (SCAN_ENFORCE=1) blocks on non-baselined
CRITICAL/HIGH findings. Recent upstream releases of transitive
dependencies added new files/loops that trip the pattern scanner, so all
three shards (extras, hf-stack, studio) red-failed on legitimate library
code. Add the 7 reviewed findings to scripts/scan_packages_baseline.json.
Each entry is genuine upstream code from the official PyPI archive:
- huggingface-hub huggingface_hub/_sandbox.py (staged dropper + C2 loop):
the HF Jobs sandbox bootstrap string and its host-pool reservation
loop. New in huggingface_hub 1.x (pulled via huggingface_hub>=0.34.0).
- huggingface-hub huggingface_hub/hf_api.py, utils/_http.py (C2 loop):
standard polling / retry while True loops.
- fastapi fastapi/routing.py (C2 loop): websocket receive loop.
- fastmcp-slim fastmcp/cli/apps_dev.py (fs enum + network): the FastMCP
dev CLI (PrefectHQ) making httpx/socket calls.
- cffi cffi/_cffi_gen_src.py (compile + exec): cffi generating and
running C extension source, its core purpose.
Additive only: no existing baseline entry is changed or removed. Verified
by re-running the scanner over the full closure on Python 3.12.13 (the CI
interpreter); it now exits 0 with only MEDIUM findings remaining.
* scan_packages: key baseline on matched-code hash
The baseline matched on (package, package-relative file, check), which
excluded the matched code, so a future finding of the same check in the
same file was suppressed regardless of what the code did. A malicious
future version of an already-baselined package could place a payload in
the same file under the same check and pass the enforcing gate.
Key the baseline on a hash of the matched code too. The hash is over the
deduped, sorted set of matched spans with L<NN>: line markers stripped, so
version bumps, line shifts and match reordering stay stable while new or
changed flagged code reopens the finding. Version is left out of the key so
routine dependency bumps do not reopen every entry. The hash is capped and
recomputable from the stored evidence.
Regenerate scan_packages_baseline.json against the current dependency set;
the hf-stack, studio and extras scan shards pass enforcing (no active
CRITICAL or HIGH).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: refresh baseline for newer unsloth-zoo release
A newer unsloth-zoo published after the first regenerate added
tests/test_mlx_save_export_regressions.py, a benign test fixture
(temporary_location="/tmp/ignored") that trips the /tmp dropper check.
Regenerate the hf-stack shard against the current set so the entry is
allowlisted; studio and extras are unchanged.
* scan_packages: harden baseline loading against malformed JSON
Guard against a non-dict top-level baseline and non-dict entries so a
corrupt or hand-edited allowlist warns and fails closed instead of
crashing with AttributeError, and treat an explicit evidence: null as
empty.
* scan_packages: hash the full match set, keep indentation, strip only the marker
Address the evidence-hash review feedback:
- Capture every matching line, not the first three, so a payload appended
after existing matches in a baselined file and check reopens the finding
instead of riding the sample.
- Preserve leading indentation so a flagged line moved out of a guarded block
reads as changed.
- Strip only each span's prefix up to the first L<NN>: marker, so an L<NN>:
inside the matched code is kept and a change to it reopens the finding.
Evidence and its hash are stored in full and stay recomputable from the stored
field. Regenerate the baseline; hf-stack, studio and extras pass enforcing with
no active CRITICAL or HIGH.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind baseline evidence to full matched code
Address review feedback on the evidence-hash baseline key:
- Split evidence only on real span delimiters (" | " before an L<NN>:
marker, or a newline), so a bitwise-or or union type in matched code
is no longer split apart into separate spans.
- Record matched lines in full (drop the 160-char per-line cap) and
record every distinct multiline match, so code appended past the cap
or a second cross-line match reopens the finding instead of riding the
first one.
- Give the large-JS-bundle and .pth base64-blob findings a content
digest instead of empty or prefix-only evidence, and record all .pth
import lines, so a changed bundle, blob or import no longer inherits a
baselined empty or truncated key.
- Warn when a loaded baseline has entries without evidence_hash so a
legacy baseline is regenerated rather than silently degraded.
Regenerate scripts/scan_packages_baseline.json against the current dep
set and add regression tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: harden multiline and duplicate evidence handling
Follow-up hardening so the evidence hash tracks the full matched code:
- For DOTALL patterns that match across lines, record every line the match
spans (not just the start line), so a change on a continuation line (the
URL inside a baselined C2 loop, a swapped credential path) reopens the
finding. A pathological greedy span is bounded to its head line plus a
digest of the rest.
- Keep duplicate spans in the canonical evidence so a second identical
matched line in a new code path changes the key instead of deduping away.
- Anchor the evidence prefix to strip only a genuine leading label or
line-number marker, leaving a marker-like "L<NN>:" inside raw .pth code
intact.
- Make the legacy-baseline warning explicit that entries without an
evidence_hash reopen rather than suppress under a coarse key.
Regenerate scripts/scan_packages_baseline.json (same finding set; entries
for same-file repeated checks are now tracked separately) and add tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind every combo and large finding to its full content
Close the remaining asymmetric-evidence gaps so a changed payload cannot
ride a reviewed baseline entry:
- Digest a capped multiline span from the code without line markers, so a
pure line shift stays stable while a continuation-line change reopens.
- Give the "Unusually large executable .pth" finding a content digest
instead of keying on byte size and import-line count alone.
- Record both contributing signals for the JS credential+network stealer,
the shell credential+network and persistence-hook combos, and the hidden
network+exec docstring payload, so changing the network/exec side reopens.
- Allow punctuation in an evidence label prefix so a "network+exec:" label
is stripped and line shifts do not change the key.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* scan_packages: bind remaining Python combos; key npm baseline on evidence
Python scanner: the openssl+key, anti-analysis, DNS-exfil and base64+exec+blob
combos recorded only one contributing signal, so a changed payload on the other
side could ride a reviewed baseline entry. Each now binds every co-occurring
signal (and the blob is digested, since it can sit on a separate line from the
decode call).
npm scanner: scan_npm_packages.py keyed its allowlist on (package, path,
pattern) only, the same coarse-key bypass the Python scanner just closed. Add an
evidence hash to the key (schema v3, fail-closed on older baselines) and store
full evidence. The committed baseline stays empty by design.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_npm_packages: bind full blob evidence and harden baseline loader
Follow-up on the npm evidence-hash key:
- _evidence now records every match and, when a snippet is truncated for
display, appends a digest of the full match. The obfuscated-blob key was
hashing only the truncated first-match snippet, so a changed payload tail or
an appended blob in the same package/file/pattern could ride a reviewed entry.
- _load_baseline guards that the root is an object, entries is a list, and each
entry is a dict before reading it, so a malformed baseline warns and fails
closed instead of raising AttributeError.
Add tests for a changed blob tail reopening the key and for malformed entries.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: symmetric baseline-loader guards; bind npm outbound host context
- Python _load_baseline now rejects a non-list "entries" with a warning instead
of raising TypeError, matching the npm loader.
- npm cred-surface-host (outbound) records the host with its URL path / fetch
call / host config, so a changed outbound path, headers or body reopens the
key rather than riding the bare host literal.
Add tests for both.
* scan_npm_packages: migrate v2 baselines and bind host-config outbound context
- _load_baseline now migrates schema v2 entries by recomputing the evidence
hash from stored evidence (with a legacy warning), matching the Python
loader, instead of discarding them; only pre-v2 basename schemas are rejected.
- The cred-surface-host (outbound) host-config branch now captures the whole
line (path, headers, body), so a changed outbound payload on the same
hostname line reopens the key instead of riding the bare host snippet.
Add tests for v2 migration and the host-config context binding.
* scan packages: bind PEM key bodies and npm windowed evidence to baseline keys
scan_packages: embedded-key findings now pin the full PEM block (BEGIN..END)
via a content digest, so a key body swapped under the same marker reopens the
finding instead of riding the unchanged BEGIN line. Single-line and DER keys
were already bound by their full matched line; marker-only references with no
END block (validation header lists) are unaffected, so the committed baseline
is unchanged.
scan_npm_packages: _evidence now digests the full containing line whenever the
shown snippet is only a window into it (short match on a long line, or a
truncated payload), so a changed payload tail outside the display window
reopens the key. The npm baseline is empty, so this changes no suppressions.
Adds regression tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan packages: bind multi-line evidence and every blob to baseline keys
_extract_evidence now extends each single-line match over its bracket
continuations, so a multi-line call binds its argument lines and a changed
URL or body on a continuation line reopens the key. After the per-line pass it
also records cross-line matches the scan cannot otherwise see (a DOTALL regex,
or a multi-line construct appended under a check that already had a one-line
match), so an appended multiline payload reopens instead of riding the key.
_blob_digest hashes every large base64 blob (not just the first) for the
base64+exec finding and the .pth large-blob finding, so an appended or swapped
second encoded payload reopens; single-blob files keep the same digest.
scan_npm_packages _evidence digests the full logical line (the matched line
plus its bracket-continuation lines), so a multi-line fetch's option and header
lines bind and a changed payload on a following line reopens the outbound key.
Regenerated the Python baseline: same package/file/check set, 24 entries pick
up the wider multi-line evidence. Adds regression tests for each case.
* scan packages: stop giant greedy spans from binding a whole-file digest
When a greedy DOTALL pattern (reverse shell socket...subprocess, C2 loop) has
its anchor tokens far apart, the match span covers the whole file. Digesting
that span bound thousands of unrelated lines, so the evidence hash drifted on
any edit between the anchors (a dependency bump reshuffling the file), which
made a baselined finding reopen on an upstream release. The multiline pass now
skips an oversized span when the per-line pass already bound the signal lines,
so the evidence is the stable matched lines; a genuinely appended multi-line
construct stays under the cap and is still recorded.
Regenerated the Python baseline against Python 3.12 (the version the scan CI
shards run) so the resolved dependency set matches CI. Same package/file/check
set. Adds a regression test.
* scan packages: tighten evidence binding (order, string brackets, span size)
Address review follow-ups on the evidence extraction:
- _canon_evidence keeps discovery (line) order instead of sorting. Line-shift
stability already comes from stripping the L<NN>: markers, so order stays
significant and reordering matched lines (a multi-line call's arguments)
reopens the finding.
- _logical_line_end (Python) and _logical_line_text (npm) blank string literals
before counting brackets, so a ) inside a string argument does not close the
logical line early and drop later argument lines.
- The oversized-span skip now only drops a giant whole-file bridge (over 60
lines); a genuinely appended multi-line construct is recorded so its payload
reopens, rather than riding an existing one-line match.
- npm _logical_line_text binds the enclosing bracket group, so a host-config
object whose { is on a prior line binds its path/headers/body lines.
Regenerated the Python baseline (Python 3.12, matching the scan CI shards):
same package/file/check set. Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan npm packages: normalize and bound the logical-line digest
- _evidence whitespace-normalizes the logical line before digesting (matching
_evidence_hash), so a formatter-only reindent of the bound continuation lines
does not change the sha256 suffix and reopen an unchanged finding.
- _logical_line_text follows a bracket group to its close up to a hard 200-line
cap (digest input only), so a config object longer than the backward window
still binds its whole tail instead of silently truncating.
Adds regression tests. npm baseline is empty, so no regeneration is needed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: cap single-line evidence and widen npm opener window
Cap each rendered evidence line at 200 chars in scan_packages.py: a long
or minified one-line file is shown as a bounded prefix plus a sha256 of the
full line, so a packed payload cannot dump unbounded content into the CI
logs or baseline while a change past the cutoff still changes the digest
and reopens the finding. Mirrors how the npm scanner bounds its snippets.
Widen the npm backward opener window (_MAX_CONT_LINES 12 to 200, symmetric
with the forward cap) so a host deep inside a large options object binds
the whole object, not just its own line; a changed path, header, or body on
any property reopens.
Regenerate the Python baseline with Python 3.12: only the protobuf
nspkg.pth and unsloth-zoo compiler.py evidence change, both from the new
line cap; the package/file/check key set is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: bind all host contexts, deep call continuations, far-back npm openers
Three fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: measure the forward bracket-group cap from the matched
line (idx + _MAX_GROUP_LINES) instead of the opener, so an opener found near
the widened backward limit no longer consumes the forward budget and drops
the path, headers, or body that follow the host.
scan_npm_packages.py: _outbound_host_evidence now records every outbound
context form for a host (URL, fetch-context, host-config), claiming each
non-overlapping match in form order, so a separate host-config request added
beside an already-baselined URL changes the evidence and reopens the key.
The common single-context case keeps its existing snippet.
scan_packages.py: follow a matched Python call over its continuations up to a
separate _MAX_CALL_LINES (40), decoupled from the 12-line display threshold,
so a multi-line requests.post( binds its whole argument list in the digest
and a changed body deep in the call reopens; bounded so a miscounted bracket
cannot swallow unrelated code. No baseline change: the current dependency set
has no matched call that closes between 13 and 40 lines, confirmed by a
Python 3.12 regenerate that produced a byte-identical baseline.
* scan: clamp npm depth, pin large bundles, follow backslash and bound .pth dump
Four fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: clamp the backward opener scan at depth 0 so a leading
unmatched closer (a preceding block whose opener is outside the backward
window) no longer drives depth negative and masks the real enclosing opener
that follows; a host-config object after such a block now binds and a changed
path reopens.
scan_packages.py: a large JS bundle now pins its whole content even when
another JS heuristic already fired. The bundle digest was only added when no
other finding existed; it is now appended to every finding's evidence on a
large bundle, so an unchanged obfuscation signature no longer lets changed
payload elsewhere ride the matched-line key.
scan_packages.py: _logical_line_end follows explicit backslash line
continuations, so a call split with a backslash before its parenthesis binds
the continuation line (URL/body) instead of returning at the zero-depth API
line.
scan_packages.py: the catch-all .pth import evidence is bounded through
_cap_line (prefix plus a digest of every line) so a large .pth of benign
imports cannot dump the whole member into the logs or baseline while an
appended or swapped import still reopens.
Baseline regenerated with Python 3.12: key set unchanged; one entry
(unsloth-zoo compiler.py) gains the backslash-continued banner lines now
bound by the continuation fix.
* scan: handle multi-line strings, lifecycle bodies, and de-quadratic evidence
Addresses a review round plus a performance audit of the evidence extractor.
Correctness (fail-closed):
- Bind the UNION of the single-line-blanked and multi-line-blanked bracket spans
in both scanners. The multi-line view blanks a triple-quoted Python string or a
backtick template literal that spans lines, so a `)` inside such a string no
longer closes the enclosing call early and drop later arguments. The single-line
view still counts a payload embedded INSIDE a string, so a dropper that hides a
call in a string keeps its argument lines bound. Taking the larger span never
shrinks the binding below either view, avoiding a fail-open regression.
- cred-env-in-lifecycle now pins the whole lifecycle script body via a digest, so
a changed non-token line (e.g. adding a curl exfil beside the token reference)
reopens, not just a change on the token line.
Performance / DoS (the scanner runs on attacker-controlled package files up to the
64 MiB / 16 MiB member caps, with no per-file time budget):
- _extract_evidence precomputes newline offsets once and maps match offsets with
bisect, removing the O(matches) whole-file content.count per match that made the
finditer fallback quadratic (a crafted minified file went from ~13 s/MiB and
hours at the cap to linear).
- npm _index_text splits and string-blanks the file once per evidence call instead
of per match (was O(matches x file) time and allocation).
- Bound evidence output: _MAX_EVIDENCE_SPANS (Python) and _MAX_EVIDENCE_MATCHES
(npm) fold the remainder into a digest so a file with thousands of matches cannot
build a multi-megabyte evidence/baseline blob while an added/removed match past
the cap still changes the key.
- _outbound_host_evidence caps matches per form and bounds the overlap claim so a
host repeated many times cannot make it quadratic.
No baseline change: a Python 3.12 regenerate is byte-identical (the union equals the
legacy single-line span for every current dependency file; the cap thresholds sit
above the largest real entry), so these are forward-looking hardening with no drift.
* scan: count all overflow matches, bind their context, blank JS regex literals
Follow-ups on the evidence output caps from the previous commit.
- _outbound_host_evidence no longer truncates each pattern's match iterator with
islice; it iterates every match and runs the overlap dedup only while the
display list is below the cap (so claimed stays bounded and the check is O(cap)
per match, not quadratic), folding every match past the cap into the overflow
digest. A host context beyond the 64th is counted again, so it reopens.
- The overflow digest (both scanners, via a shared _overflow_digest) binds each
overflow match's logical-line context, not just the regex match text, so a
changed payload on an over-cap line reopens even with the matched token
unchanged.
- The multi-line JS blanked view now blanks regex-literal bodies (tracking the
previous significant char for regex-vs-division and char classes for a literal
`/` inside `[...]`), so a `)` inside `/)/` no longer closes an outbound call
early. The bound span is the union of the single-line and multi-line views, so
an imperfect regex decision only ever grows the span, never shrinks it.
- The Python overflow digest canonicalizes spans (strips L<NN>: markers via
_canon_evidence) before hashing, restoring line-shift stability for the
over-cap region.
No baseline change: the overflow branches only trigger above the per-finding caps
(above the largest real entry), and the npm baseline is empty, so a Python 3.12
regenerate is byte-identical.
* scan: refresh baseline for ipython interactiveshell.py span drift
A newer ipython release changed the filesystem-enumeration span in
IPython/core/interactiveshell.py, so its content digest no longer matched the
baselined evidence and the studio scan shard flagged it as a non-baselined
CRITICAL. Regenerated with Python 3.12: only the ipython entry's evidence_hash
changes; the package/file/check key set is unchanged, and a studio enforcing
spot-check exits 0.
* Bound scanner evidence memory: stream overflow spans and cap lifecycle baseline size
scan_packages.py: _extract_evidence no longer materializes a rendered span
per match before slicing at the display cap. Once out holds _MAX_EVIDENCE_SPANS
spans, further spans fold straight into a running digest, so a minified or
padded file with hundreds of thousands of matching lines keeps memory bounded
to the display cap instead of the match count. The fold reproduces
_canon_evidence(" | ".join(overflow)) byte for byte, so the overflow digest and
every baseline key are unchanged.
scan_npm_packages.py: lifecycle-fetch-exec and cred-path-in-lifecycle stored the
entire install script body as evidence, so --write-baseline on a package with a
multi-MiB lifecycle script bloated the baseline JSON. Both now store a bounded
matched snippet plus a body-sha256 digest, matching cred-env-in-lifecycle. The
digest still binds the whole body, so a change to any line reopens the finding.
Adds tests for the streamed overflow bound and the bounded-but-reopens lifecycle
evidence. Baseline unchanged (byte-identical Python evidence; npm baseline empty).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make npm bracket-group scan order-aware so a same-line close-then-open binds
_scan_group counted brackets with a per-line net (opens minus closes), which
collapses intra-line order: a line that closes a prior block and then opens the
host-config object on the same line, e.g. `}); const opts = {`, nets to <= 0, so
the trailing `{` was dropped and the group started at the hostname line. A
changed path/headers on the following lines then hashed to the same evidence and
could ride an existing baseline key.
Replace the net count with an order-aware (L, R) reduction per line (L closers
needing an opener to the left, R openers needing a closer to the right) and apply
it in order in both the backward and forward scans, clamping stray closers at 0.
The trailing opener now stays visible so the whole object binds and a changed
payload reopens. Per-line cost is unchanged (one C-level bracket findall), so the
existing outbound-host evidence is byte-identical on all prior shapes; only the
previously-dropped same-line case changes. Adds a regression test for it.
* Harden scanner evidence: bound memory and bind Python call tails fail-closed
Five fixes across both scanners, none of which change the committed baseline (a
full regen of all three pip shards produced a byte-identical 185-key set).
scan_npm_packages.py: _evidence and _outbound_host_evidence collected every regex
match into a list before applying the 64-match display cap, so a text file under
the size cap that repeats a cheap signal (such as NPM_TOKEN) millions of times
could allocate a huge list of re.Match objects and stall or OOM before the
overflow digest ran. They now stream from finditer and fold overflow as matches
arrive via a shared _fold_overflow_match helper, byte-identical to the prior
digest.
scan_packages.py:
- _extract_evidence kept inserting every unique over-cap span into the seen set
even after it stopped appending to the display list, so a generated file with
millions of one-line matches still grew that set unbounded. It now tracks spans
only while filling the display list (per-line spans are unique by line number,
so dropping them past the cap cannot miss a dedup).
- _scan_line_end counted brackets with a per-line net, so a continued statement
that closes on the same line it opens a flagged call (a leading "]" before
"requests.post(") had the call's open paren cancelled and bound only the opener
line. It now applies brackets in order via _bracket_lr (leading closers clamp at
0), matching the npm bracket fix.
- a single-quoted string continued by a trailing backslash was not tracked across
lines, so a close paren inside the continued string on the next line closed the
call early; _blank_code_strings now carries the continuation.
- a call with more argument lines than the soft cap was hashed only through the
cap, so a changed data=/headers tail past it stayed suppressed; a closing call
is now followed to its real close under a 200-line hard limit (a never-closing
opener still stops at the 40-line soft cap so it cannot swallow the file).
Adds regression tests for each. npm baseline is empty; the Python baseline is
unchanged (verified byte-identical by regenerating all three shards).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bind giant DOTALL span anchors and add context to constant IOC evidence
Two fail-closed gaps where a changed payload could keep the same evidence hash
and stay suppressed by the baseline.
scan_packages.py: a giant greedy DOTALL span (a cross-line IOC match bridging
more than 60 lines, e.g. RE_TEMP_EXEC matching a /tmp line and a much-later
subprocess line) was dropped entirely once the per-line pass had any match, so an
appended cross-line payload -- a new /tmp line plus a later subprocess line that
share no single line, so the per-line pass never binds them -- produced the same
evidence and rode the key. The span is no longer dropped: it is bound by its head
and tail anchor lines plus a digest over just those (no line numbers, so a pure
line shift is stable). An added or moved anchor reopens the finding, while churn
in the bridged interior stays stable, so this does not reintroduce whole-file
drift. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry
such a span and are refreshed; a full three-shard regen confirmed only those two
keys change.
scan_npm_packages.py: known-ioc-string and cred-surface-host (always-bad) recorded
only the bare needle/host as evidence, so a reviewed tarball that kept the IOC
string while altering the adjacent fetch/exfil body produced an identical key.
They now bind matched-line context: known-ioc-string via the matched line and its
bracket-group continuation, cred-surface-host (always-bad) via the outbound call
context (path/headers/body, falling back to the bare host when not in an outbound
call). A changed payload on the same call now reopens.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
only the two giant-span entries.
* Hash giant-span interiors, bind exec/eval trigger, JS content, intra-literal whitespace
Four fail-closed gaps where a changed payload could keep the same evidence hash.
scan_packages.py:
- A giant bridged DOTALL span was bound only by its head and tail anchors, so a
cross-line payload inserted into the bridged interior between unchanged outer
anchors kept the same key. The whole span content is now digested (via _render),
so any interior change reopens; a pure line shift stays stable because the digest
is over the markerless code. Two baseline entries (multiprocess test, unsloth-zoo
scanner file) carry such a span; with full-interior binding, multiprocess
resolved at two versions across shards now yields two distinct entries where the
anchor digest had collapsed them into one.
- The exec/eval-with-hidden-payload findings omitted the visible exec/eval line
that makes the hidden string executable, so flipping a harmless eval("1+1") to
exec(__doc__) kept the same key while arming the payload. The trigger line from
the real-code view is now bound into the evidence.
- check_js_file extracted evidence with the Python-string-aware extractor, which
does not blank JS backtick template literals, so a template containing a close
paren closed a call's bracket span early and omitted later option/body lines. The
full file content digest is now pinned to every JS finding (not just large
bundles), binding the whole call.
scan_npm_packages.py: the evidence canon collapsed all whitespace via split(),
erasing whitespace inside JS string literals along with harmless indentation, so a
changed request body 'a b' -> 'a b' kept the same key. A new _canon_preserve_strings
collapses whitespace only OUTSIDE string literals (reindent-stable) while preserving
it INSIDE single/double/backtick literals (intra-payload edits reopen). Used for the
evidence hash and the logical-line digests.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
the two giant-span entries and adds the second multiprocess version's entry.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
check_js_file put the bundle's KB size inside the finding's check label, which is
part of the baseline match key (package, file, check). When tensorboard's
projector_binary.js grew from 1918 KB to 1933 KB, the reviewed baseline entry stopped
matching and the benign HIGH resurfaced, red-failing the studio and extras
scan-packages shards. Move the size into the evidence field (shown for review, not
matched) and keep the check label constant, then update the one tensorboard baseline
entry to the size-agnostic label. The finding is suppressed again and will not
re-break when the bundle grows by a few KB. Scanner self-tests pass unchanged.
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: drop OpenEnv and unused ExecuTorch/open_spiel install deps
* Studio: drop 8 more unused install deps from extras
* Studio: restore tomli<3.11 for kernels; tidy dep-cleanup comments and tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refresh scan-packages baseline for scipy _external + unsloth-zoo tests
scipy moved its vendored array_api_compat from scipy/_lib to
scipy/_external, so the four allowlisted array_api_compat __init__.py
entries stopped matching and resurfaced as unsuppressed CRITICAL
"Downloads and executes remote code" findings on all three pip
scan-packages shards (extras, hf-stack, studio). Add the _external
paths next to the existing _lib ones so both scipy layouts stay covered.
Allowlist two unsloth-zoo test-file false positives now present in the
hf-stack shard: tests/test_mlx_save_export_regressions.py (writes to
/tmp dropper) and tests/test_mlx_trainer_internals.py (obfuscation plus
exec/eval).
Drop nine stale entries for packages removed from the Studio
requirements and no longer in any shard closure (evaluate, pytest,
hypothesis, kgb, langid), confirmed absent via with-deps resolution of
all three shards.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Package scanners: cut false positives and make the CI gate blocking
scan_packages.py and scan_npm_packages.py red-failed on legitimate
library code, so the security-audit steps were left advisory. Reduce
the false positives at the source and flip both gates to blocking.
scan_packages.py:
- Scan code only: blank comments and bare docstrings/doctests before
matching (line numbers preserved), so prose and >>> examples cannot
trip a finding.
- Drop the platform.system() branch from the anti-analysis regex (under
DOTALL it matched across the whole file, so every cross-platform
library tripped it) and fix the dead /proc/self/status alternative.
- Add a reviewed baseline allowlist (scan_packages_baseline.json) keyed
on (package, basename, check): only non-baselined CRITICAL/HIGH exit
1, and a new kind of finding in a listed file still fails.
- sdist fallback: when --with-deps cannot resolve a shard (a sdist-only
package or a version conflict), drop to per-spec and fetch the raw
sdist from the PyPI JSON API (no pip build, no setup.py), so every
package is still scanned and no shard exits 2.
scan_npm_packages.py:
- Mirror the code-only JS/TS scanning (blank // and /* */ comments,
string/template/regex aware) and the baseline allowlist. The npm
corpus is clean today, so the baseline is empty.
security-audit.yml:
- Flip both scan steps to blocking (SCAN_ENFORCE=1), capturing the
scanner exit via PIPESTATUS so tee does not mask it.
tests/security: add coverage for the strip, baseline and sdist paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review feedback on the package scanners
- Do not blank f-strings during code-only scanning (they evaluate at
import); and when a file uses exec/eval, rescan the original for
payload carriers hidden in a docstring/string so exec(__doc__) style
payloads stay visible.
- sdist fallback: recover transitive deps with their version specifier
(fetch the pinned version, not latest), and recover deps in the
--no-deps branch too so a sdist-only transitive dependency is still
scanned instead of silently skipped.
- Baseline: key by package-relative path, not basename, so a future
same-named file in another directory is not auto-suppressed.
Regenerated the baseline accordingly.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han