ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

studio/ci: npm tarball content scanner (no-install, hostile-input safe) (#5393)

Browse source 
* studio/ci: npm tarball content scanner (no-install, hostile-input safe)

Counterpart to scripts/scan_packages.py for the npm side. Pip-side
scanner reads requirements files, downloads PyPI archives via
`pip download --no-deps`, and pattern-scans them for malicious
shapes. This change adds the equivalent for npm tarballs.

Why
===

PR #5392 (lockfile_supply_chain_audit.py) catches injection-pattern
attacks where the malicious metadata lives IN the lockfile -- e.g.
the TanStack Shai-Hulud worm that injected an `optionalDependencies`
entry pointing at a GitHub commit. It does not catch the broader
class of "legit-registry tarball with malicious content but normal
lockfile metadata": attacker steals a maintainer's npm publish
token, publishes a malicious version to registry.npmjs.org with a
valid integrity hash, and the lockfile entry looks normal -- the
malicious code lives inside the tarball's dist/index.js or its own
postinstall script. Today that gap is covered reactively by `npm
audit` + OSV-Scanner once the GHSA lands; there is a real window
before that.

This scanner closes the window by inspecting tarball CONTENT.

What it checks
==============

For each entry in studio/frontend/package-lock.json:

  1. Download the tarball directly from registry.npmjs.org. Refuse
     any non-allowlisted URL. Stream-bounded at 64 MiB.
  2. Verify SHA-512 integrity against the lockfile entry BEFORE
     opening the tarball.
  3. Safely extract into a sandboxed temp dir behind guards:
       - reject symlinks / hardlinks (LNKTYPE, SYMTYPE)
       - reject absolute paths and `..` traversal
       - reject character / block / FIFO devices
       - per-file size cap 8 MiB, cumulative cap 128 MiB,
         member count cap 50000
       - stream open (mode='r|gz') so we abort mid-extract
       - extracted files set to non-executable mode (0o644)
  4. Pattern-scan the extracted text content for:
       - lifecycle (preinstall/install/postinstall/prepare) scripts
         in any package.json that fetch + pipe-to-shell external
         content -- the install-time RCE vector
       - optionalDependencies pointing at github: / git+ / git:
         (TanStack worm injection shape)
       - C2 / exfiltration hosts: getsession.org, 169.254.169.254
         (IMDS), 169.254.170.2 (ECS), metadata.google.internal,
         vault.svc.cluster.local, k8s ServiceAccount token paths,
         ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN, npm publish-token
         enumeration endpoint
       - credential paths a frontend lib should never read:
         ~/.npmrc, ~/.aws/credentials, ~/.ssh/id_*, /.kube/config,
         /.docker/config.json
       - JS regex: Function/eval against base64-decoded payload,
         process.env.GITHUB_TOKEN / NPM_TOKEN / AWS_* access in
         package source
       - obfuscation: large base64-ish blob (>=2 KiB) fed into
         Function or eval (router_init.js dropper shape)
       - literal IOC substrings from public advisories

Safety
======

Threat model: every tarball is hostile. The scanner:

  - never runs `npm install`, never executes anything from a
    downloaded tarball, never calls subprocess on extracted content
  - downloads only from registry.npmjs.org (defence-in-depth check
    at parse time AND inside download_tarball)
  - stdlib-only (no third-party deps -- adding one would itself
    be a supply-chain liability)
  - tempdir wiped via atexit on every termination path
  - exit codes: 0 clean, 1 HIGH/CRITICAL finding, 2 internal error

Wiring
======

New job `npm-scan-packages` in security-audit.yml, parallel to
`pip-scan-packages`. Triggers same as the existing audits (PR on
manifest changes, push to main/pip, daily 04:13 UTC, dispatch).
Initially `continue-on-error: true` so the baseline can settle --
matches the existing convention for the other audit steps. Drop
that flag once the baseline is clean for a week.

Verified locally
================

  - AST parse OK.
  - Real-network 3-package smoke: 0 findings.
  - Real-network 25-package smoke (Babel + assistant-ui surface):
    0 findings, no hard errors.
  - 9 fault-injection scenarios all pass:
      1. zip-slip path traversal refused
      2. symlink member refused
      3. oversized member refused (size cap)
      4. too-many-members refused (count cap)
      5. router_init.js IOC + obfuscated-blob shape both detected
         in synthetic malicious tarball
      6. lifecycle fetch-exec in scripts.preinstall detected as
         CRITICAL
      7. AWS IMDS reference (169.254.169.254) detected
      8. SRI integrity-parser accepts syntactically-valid SRI
      9. download_tarball refuses non-allowlisted hostname

Refs
====

  - https://tanstack.com/blog/npm-supply-chain-compromise-postmortem
  - https://github.com/TanStack/router/issues/7383
  - https://github.com/TanStack/router/security/advisories/GHSA-g7cv-rxg3-hmpx
  - https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised
  - https://www.stepsecurity.io/blog/mini-shai-hulud-is-back-a-self-spreading-supply-chain-attack-hits-the-npm-ecosystem

* scan_npm_packages: kill false positives + handle real native binaries

First CI run on PR #5393 (run 25710423126 / job 75489317395) hit
two false-positive classes plus one cap-too-tight class:

False positives (7 findings):

  @langchain/core 1.1.44 ssrf.{cjs,js}: a SSRF *protection* module
    that ships a literal blocklist `const CLOUD_METADATA_IPS = [...]`
    of IMDS hosts as data the library REFUSES to dial. Our scanner
    saw the IPs as substrings and flagged 6 of them.

  object-treeify 1.1.33 package.json: a manual `docker` dev script
    that mounts `~/.npmrc` and `~/.aws` for local containerised
    builds. npm never runs `scripts.docker` automatically; it is
    only invoked when a developer runs `npm run docker`. Our bare
    substring scan flagged the `/.npmrc` reference anyway.

Cap-too-tight class (10+ findings):

  next/swc, rolldown bindings, biome CLI, lightningcss, mermaid
  sourcemap, typescript.js. The 8 MiB per-file cap was calibrated
  for JS source and rejected legitimate precompiled native binaries
  (next-swc .node is 137 MB) and CLI executables (biome is 25-33 MB).

Fixes
=====

  cred-surface-host detection split into two tiers:

    ALWAYS_BAD substrings have no legitimate use anywhere and still
    bare-match: `registry.npmjs.org/-/npm/v1/tokens`,
    `ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN`.

    NEEDS_CONTEXT substrings (IMDS IPs, GCE metadata host, k8s
    ServiceAccount path, Vault endpoint) require co-occurrence with
    EITHER a fetch verb (fetch/axios/http.get/etc) within 200 chars
    OR an `http(s)?://HOST` URL prefix OR a `host:`/`hostname:`
    config field. A defensive blocklist literal does not match any
    of those rules; an actual outbound call always does.

  cred-surface-path detection moved out of the bare-text scan into
  `scan_package_json` and scoped to the 4 NPM lifecycle hooks
  (preinstall / install / postinstall / prepare). A `/.npmrc`
  reference in a `docker` dev script is silent; a `cat ~/.npmrc
  | curl ...` in a `postinstall` fires HIGH.

  Per-file size cap split by content type, sniffed via 16-byte
  magic header read (ELF / Mach-O / PE / WASM / archive formats),
  plus suffix list (.node/.wasm/.so/.dll/.dylib/.exe), plus regex
  for versioned shared libs (libfoo.so.8.17.3), plus a null-byte
  ratio fallback for extensionless binaries that headers do not
  catch.

    Text files: 16 MiB cap (still tight; typescript.js at 9.1 MB is
    the legitimate ceiling).
    Binary files: 256 MiB cap (next-swc .node is 137 MB; sharp
    libvips is ~18 MB; rolldown bindings are 18-26 MB each).
    Cumulative: 512 MiB per tarball.
    Tarball: 256 MiB compressed.

  Binary files are also skipped in the content scanner -- regex
  over compiled machine code is noise. The IOC substring fallback
  in `scan_extracted_tree` now uses the same magic-sniff to decide
  whether to grep.

  HTTP timeout bumped 30s -> 60s for large tarballs.

Verified
========

  - AST parse OK.
  - 11 fault-injection tests pass:
      * zip-slip, symlink, oversized-declared-size, count-cap
      * router_init.js IOC detected
      * IMDS-in-URL still detected (new contextual rule)
      * langchain SSRF blocklist no longer false-positive
      * object-treeify docker script no longer false-positive
      * lifecycle-script `cat ~/.npmrc | curl ...` detected
      * synthetic ELF (extensionless executable) extracts and is
        correctly skipped from text scan
      * versioned `.so.8.17.3` shared lib extracts cleanly
  - Real-network end-to-end on the full lockfile:
      968 packages, 0 findings, 0 hard errors, 76 seconds.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-05-11 20:37:05 -07:00 • committed by GitHub
commit e27cc0ab08
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 1281 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

80
.github/workflows/security-audit.yml vendored
View file

@ -57,6 +57,7 @@ on:
- 'studio/src-tauri/Cargo.lock'
- 'pyproject.toml'
- 'scripts/scan_packages.py'
- 'scripts/scan_npm_packages.py'
- '.github/workflows/security-audit.yml'
push:
branches: [main, pip]
@ -815,3 +816,82 @@ jobs:
logs-scan-packages-${{ matrix.shard.id }}.txt
audit-reqs/
retention-days: 30
# ─────────────────────────────────────────────────────────────────────
# npm: pre-install tarball content scan.
# ─────────────────────────────────────────────────────────────────────
npm-scan-packages:
# Counterpart to pip-scan-packages for the npm side. Reads
# studio/frontend/package-lock.json, downloads each resolved
# tarball DIRECTLY from registry.npmjs.org (never via `npm
# install` -- no lifecycle scripts ever run), verifies the
# lockfile integrity hash, unpacks each tarball into a sandboxed
# temp dir behind size / count / path-escape / symlink guards,
# and pattern-scans the extracted file contents for the
# signatures common to npm supply-chain attacks:
#
# - lifecycle (preinstall / install / postinstall / prepare)
# scripts in any package.json that fetch + execute external
# code,
# - C2 / exfiltration hosts (getsession.org, AWS IMDS,
# Kubernetes ServiceAccount token paths, GitHub Actions OIDC,
# HashiCorp Vault endpoints),
# - credential-stealing references (.npmrc, .aws/credentials,
# GITHUB_TOKEN / NPM_TOKEN in JS sources),
# - known IOC filenames (router_init.js, tanstack_runner.js,
# router_runtime.js),
# - obfuscation shapes (Function/eval against base64 blobs).
#
# Threat model: every tarball is hostile. Safety guarantees are
# documented at scripts/scan_npm_packages.py top-of-file. The
# script is stdlib-only so adding it does not increase the
# transitive supply-chain surface.
name: npm scan-packages (Studio frontend tarballs)
runs-on: ubuntu-latest
timeout-minutes: 30
needs: []
steps:
- name: Harden runner (egress audit)
uses: step-security/harden-runner@a5ad31d6a139d249332a2605b85202e8c0b78450 # v2.19.1
with:
egress-policy: audit
disable-sudo: true
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.12'
- name: Sanity-check scan_npm_packages.py
run: |
test -f scripts/scan_npm_packages.py
python3 -c "import ast; ast.parse(open('scripts/scan_npm_packages.py').read())"
- name: Scan npm tarballs (declared + transitive, no install)
# The script exits 1 on HIGH/CRITICAL findings; we capture the
# full log and surface it in the step summary either way. It
# never runs `npm install`, never executes anything from a
# downloaded tarball, and only fetches from registry.npmjs.org.
# Initially non-blocking so the baseline can settle; drop
# continue-on-error once the baseline is clean for a week.
continue-on-error: true
run: |
set -o pipefail
LOG=logs-scan-npm.txt
python3 scripts/scan_npm_packages.py 2>&1 | tee "$LOG"
{
echo "## scan_npm_packages"
echo
echo '### Findings (tail)'
echo '```'
tail -300 "$LOG"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: scan-npm-packages-log
path: logs-scan-npm.txt
retention-days: 30

1201
scripts/scan_npm_packages.py Executable file
View file

File diff suppressed because it is too large Load diff