Recent supply-chain incidents that scan_packages would have missed:
- PyTorch Lightning 2.6.x: payload in _runtime/router_runtime.js
(14.8 MB), persistence via .claude/settings.json SessionStart
and .vscode/tasks.json folderOpen
- npm chalk/debug + Shai-Hulud: hex-var obfuscation, window.ethereum
Web3 hijack, .github/workflows/shai-hulud.yml repo takeover,
trufflehog credential exfil
- elementary-data 0.23.3: token harvesters with embedded gh{p,o,s}_
and AKIA regexes
- litellm 1.82.7: also covered by existing patterns, but anyone on
`>=` got it during the 40-min exposure window
- langchain-core CVE-2025-68664 / n8n CVE-2025-68668 / marimo
CVE-2026-39987: first-party design flaws, not malicious-author
scan_packages.py:
- Six new regexes: RE_DEV_TOOL_HIJACK, RE_TOKEN_REGEX,
RE_JS_OBFUSCATION, RE_WEB3_HIJACK, RE_WORKFLOW_INJECT,
RE_SHELL_DROPPER.
- Three new checkers: check_js_file, check_shell_file,
check_workflow_file. scan_archive now routes .js/.mjs/.cjs/.ts
to the JS checker, .sh/.bash to the shell checker, and
.github/workflows/*.yml to the workflow checker.
- JS checker fires CRITICAL on hex-var obfuscation OR Web3 hijack
OR (token regex + network) OR workflow-injection signature; HIGH
on a >100 KB JS bundle inside a Python wheel (the Lightning tell).
- Smoke-tested: every new pattern matches its canonical positive
and rejects four legitimate-looking false-positive baits.
security-audit.yml:
- OSV-Scanner step: cross-ecosystem advisory check (PyPI + npm
+ cargo) from one binary. OSV's feed is a superset of GitHub-
Advisory; catches CVEs that haven't propagated yet (e.g.
langchain-core was on OSV before GitHub Advisory).
- Semgrep step: p/supply-chain + p/python + p/javascript +
p/security-audit packs catch first-party logic bugs (CVEs 7/9/10
above) that pattern scanning never sees.
- Lockfile pin verifier: warns on every non-`==` spec in
requirements/*.txt. Currently surfaces 104 unpinned specs as
informational baseline; tighten to blocking once the baseline
is curated.
All new steps continue-on-error initially; they surface findings to
the workflow summary + advisory-audit-logs artifact.
Three advisory-DB lookups previously spun up three separate runners.
All three are fast lockfile-driven checks (pip-audit ~1m37s, npm audit
~12s, cargo audit ~24s) and the runner-setup overhead dominates each.
Run them sequentially on a single runner with python + node + rust
toolchains pre-installed; total wall clock comes out roughly the same
(~3 min) but with one PR check instead of three.
Each step keeps continue-on-error: true so a finding in one toolchain
does not suppress the others. Logs land in a single advisory-audit-logs
artifact (pip + npm + cargo + the filtered req set).
Heavy job stays separate: pip-scan-packages remains the 3-shard matrix
that downloads + pattern-scans the full PyPI transitive closure (~6
min/shard, in parallel). Conflating that into the advisory job would
bloat the runner image and serialize a 6 min job behind a 30 s one.
Previous run took ~10+ minutes because each requirements file ran
its own --with-deps resolve serially, and the six files all share
~70% of their transitive set (transformers, peft, accelerate land
in three of them). Net effect: the same 200+ archives downloaded and
pattern-scanned three times in series.
Two changes:
1. Within a shard, feed every -r file to ONE scan_packages call so
pip's resolver intersects version constraints once and yields
a single deduped transitive set.
2. Across shards, run three matrix jobs in parallel:
- hf-stack: unsloth-deps + no-torch-runtime (pyproject extras)
- studio: studio + overrides + extras-no-deps
- extras: extras (heavy openai-whisper / scikit-learn stack)
Wall clock now bounded by the slowest shard rather than the
sum, dropping ~10 min to ~3-5 min.
Each shard uploads its own artifact (scan-packages-log-<id>) so log
correlation stays clean. fail-fast: false so one shard's findings
don't suppress the others.
Last push surfaced two silent failures:
1. pip-audit aborted on openai-whisper. The package's setup.py
imports pkg_resources, which the isolated build env's modern
setuptools no longer ships by default. Because we passed every
-r file in one invocation, that single build failure killed the
audit for ALL files (the run reported success only because
continue-on-error swallowed exit 1).
2. scan_packages --with-deps aborted on the first git+ spec it
hit (triton-kernels.txt's git+https://github.com/triton-lang
/triton.git, plus OpenEnv in extras-no-deps.txt). Same
all-or-nothing behaviour: the entire transitive scan reported
"0 archives downloaded" and "all clean" -- meaning we silently
scanned nothing.
Fixes:
- Build a filtered audit-reqs/ tree first. Each Studio requirements
file is copied with `git+` lines stripped (replaced with a
`# [security-audit] skipped` marker so the exclusion is auditable
in the artifact). Pure git refs are out of scope for both pip-
audit (CVE DB only knows PyPI versions) and scan_packages (it
inspects PyPI archives, not git HEADs).
- Run pip-audit per-file in a loop. One bad file no longer takes
out the whole audit.
- Pin setuptools<78 + wheel into pip's isolated build env via
PIP_CONSTRAINT, so legacy setup.py packages (openai-whisper) can
still emit metadata for the resolver.
- Run scan_packages per-file too, with the same git+ filter and a
skip for files that are empty after filtering (triton-kernels.txt
becomes a comments-only file and would otherwise spam the log
with `--help`).
Net effect: pip-audit now actually emits CVE findings (we know the
default branch carries 17), and scan_packages downloads + pattern-
scans the full transitive closure of every PyPI-only requirements
file plus unsloth's pyproject deps.
The previous Security audit only covered Studio's backend requirements.
The unsloth pip package itself ships its own dep set via pyproject.toml
(typer/pydantic/pyyaml/nest-asyncio core, plus the huggingfacenotorch
extras: transformers/peft/accelerate/trl/datasets/diffusers/etc.) -- a
malicious upload to any of those would slip past us today. Build a
combined dep list from pyproject.toml + the six Studio requirements
files and feed it to both pip-audit and scan_packages.
Add scan_packages.py at scripts/scan_packages.py so the scanner ships
with the repo and CI does not depend on a network fetch at job time.
Pass --with-deps to scan_packages so the pre-install pattern scan
walks the full transitive closure -- supply-chain attacks usually land
several hops down (litellm 1.82.7 was a dep of a dep for most users;
top-level-only scanning would have missed it).
No installation in either job. pip-audit's -r mode resolves through
PyPI metadata, scan_packages downloads sdist/wheel archives raw and
inspects them without running install hooks. An attacker who has
compromised a transitive dep cannot execute code in this workflow.
Three Priority-1 follow-ups from the lint review.
Lint CI gains two non-blocking gates that surface drift without
blocking merges (the same shape as the existing format-drift step):
- codespell: typo catcher across source / comments / docs. Skips
lockfiles, generated assets, binary artefacts, LICENSE files.
ignore-words-list pulls out short identifiers and PyTorch
idioms (parm/parms, ans, hist, etc.) the default dictionary
would flag. Local run finds 16 real typos to fix in a follow-up.
- shellcheck: catches subtle shell bugs `bash -n` doesn't see --
unquoted expansions, useless cat, `[[ ]]` command substitution,
etc. SC1090 + SC2034 muted because install/setup scripts
legitimately source runtime paths and use export-only
assignments. Critical-path coverage: install.sh, setup.sh,
tests/sh/.
Both pinned for reproducibility (codespell>=2.3,<3 in pip,
shellcheck via apt-get). Both surface findings in PR annotations
without failing the run; drop continue-on-error after the cleanup
PRs land.
New workflow: Security audit. Runs `pip-audit` against the same
dep set Studio's backend pytest matrix installs, so we audit what
the runtime actually loads (not what pyproject.toml's transitive
resolution might pull in differently). Triggers:
- PRs touching requirements / pyproject.toml,
- push to main / pip,
- nightly @ 04:13 UTC (off-the-hour to dodge cron rush),
- workflow_dispatch.
The default branch already carries 17 known vulnerabilities per
the dependabot banner, so a hard gate today would block every PR
on a baseline we have not triaged. Non-blocking; full table goes
to GITHUB_STEP_SUMMARY for grep-ability and a 30-day artefact for
historical comparison.
The custom AST anti-pattern scan I prototyped was dropped: every
class of CPU-import-time bug we hit in this PR (bitsandbytes,
torchvision, _cuda_getCurrentRawStream, DEVICE_COUNT==0 stream
init) is already caught by the Repo tests (CPU) job exercising
the actual import on a CPU torch wheel. Restating the rule
in AST form would only add noise.
