* Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths
Follow-up hardening on the now-blocking scanners so the enforcing gate cannot
report clean while a malicious artifact goes unscanned.
scan_packages.py
- Hidden payload: also flag a network call AND an os/subprocess exec that live
only in a blanked docstring/string of an exec/eval file (the fetch-then-run
shape of an exec(__doc__) dropper). Either alone in real code was already
covered; hidden together they are the payload.
- Pinned releases fail closed: _release_files no longer falls back to the latest
artifact when a pinned version is missing or empty, so a yanked/bad pin is an
error instead of a different file being scanned in its place.
- requires_dist is read from the pinned release's metadata, not the project-level
(latest) document, so a sdist-only pin follows its own dependency tree.
- Environment markers are evaluated (PEP 508) instead of dropping any marker that
merely contains the word extra, so default-true markers like extra != 'dev' are
kept; conservative fallback keeps a dep on any uncertainty.
- Transitive recovery is a depth-bounded worklist: a wheel dependency whose own
child is sdist-only is fetched (--no-deps) and scanned, then its children are
recovered in turn, rather than being silently skipped.
scan_npm_packages.py
- Baseline keys use the package-relative path instead of the basename, so the
same basename in a different directory is not over-suppressed.
Tests cover each case; full scripts pass AST and ruff checks.
* Address review: tighten marker scope, decoy-proof the dropper check, fail closed on missing pin metadata
- Markers: keep any dep whose marker can hold on another install target
(sys_platform == 'win32', python_version == '3.13'); only drop a marker that
depends solely on extra and is false with no extra. A scanner runs on one
target but must cover code installed on others. Pure-extra markers are
evaluated against default_environment() with extra unset.
- Hidden dropper: the network+exec docstring check now inspects the removed
(blanked) span directly, so a benign visible network or subprocess call cannot
mask a payload that still lives in a docstring. Carrier checks stay
blanked-only (an in-code carrier is already caught by the normal check), so
corpus findings are unchanged.
- requires_dist: a pinned version whose own metadata cannot be fetched recovers
nothing rather than substituting the latest release's dependency tree.
- Transitive recovery: the last-ditch direct-sdist branch also chases the
recovered package's declared deps, matching the other branches.
- npm baseline: schema bumped to v2 (package-relative keys); a pre-v2 baseline
with entries is ignored (fail closed) instead of mis-applying basename keys.
Tests cover each case; scripts pass AST, ruff, and the import-hoist verifier.
* Scanner: exclude comments from hidden-payload check, flag missing pin metadata as incomplete
Hidden network+exec detection now inspects only docstring/string spans (what exec(__doc__)/exec(<str>) can actually run), so a real exec() beside comments that mention a network and a subprocess call no longer false-positives. Missing pinned-release metadata in transitive recovery records a download_error so the --with-deps path fails closed instead of treating it as no dependencies. Adds regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix scan_packages.py --fix crash on download_packages() tuple return
`download_packages()` returns `(results, download_errors)`, but the two
`--fix`-path call sites still treated the return value as the bare results
list. `find_safe_version` did `downloaded = download_packages(...)` followed
by `if not downloaded:` (always false: a 2-tuple is truthy) and
`for _, archive_path in downloaded:`, which unpacked the results list into
two variables -> ValueError in the normal single-archive `--no-deps` case.
`_run_fix` indexed `downloaded[0][1]`, i.e. the second archive of the results
list instead of the first archive's path -> IndexError. So `--fix` crashed
exactly when a CRITICAL finding needed remediation. The main scan path already
unpacks the tuple; this aligns the two `--fix` sites with it.
Adds CPU-only regression tests for both sites.
Closes#6412
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Update scripts/scan_packages.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update scripts/scan_packages.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Reduce and tighten comments and docstrings in tests
Shorten verbose comments and docstrings across the test suite without
changing any test logic. Remove narration that restates the next line,
collapse long module and test docstrings to a single line, and drop banner
separators. Keep regression context (issue and PR references, run ids),
skip reasons, mocking and timing rationale, license headers, lint and type
directives, and commented-out code.
Comments and docstrings only: an AST signature check confirms no code,
assertions, or string literals changed, and the suite byte-compiles cleanly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Package scanners: cut false positives and make the CI gate blocking
scan_packages.py and scan_npm_packages.py red-failed on legitimate
library code, so the security-audit steps were left advisory. Reduce
the false positives at the source and flip both gates to blocking.
scan_packages.py:
- Scan code only: blank comments and bare docstrings/doctests before
matching (line numbers preserved), so prose and >>> examples cannot
trip a finding.
- Drop the platform.system() branch from the anti-analysis regex (under
DOTALL it matched across the whole file, so every cross-platform
library tripped it) and fix the dead /proc/self/status alternative.
- Add a reviewed baseline allowlist (scan_packages_baseline.json) keyed
on (package, basename, check): only non-baselined CRITICAL/HIGH exit
1, and a new kind of finding in a listed file still fails.
- sdist fallback: when --with-deps cannot resolve a shard (a sdist-only
package or a version conflict), drop to per-spec and fetch the raw
sdist from the PyPI JSON API (no pip build, no setup.py), so every
package is still scanned and no shard exits 2.
scan_npm_packages.py:
- Mirror the code-only JS/TS scanning (blank // and /* */ comments,
string/template/regex aware) and the baseline allowlist. The npm
corpus is clean today, so the baseline is empty.
security-audit.yml:
- Flip both scan steps to blocking (SCAN_ENFORCE=1), capturing the
scanner exit via PIPESTATUS so tee does not mask it.
tests/security: add coverage for the strip, baseline and sdist paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review feedback on the package scanners
- Do not blank f-strings during code-only scanning (they evaluate at
import); and when a file uses exec/eval, rescan the original for
payload carriers hidden in a docstring/string so exec(__doc__) style
payloads stay visible.
- sdist fallback: recover transitive deps with their version specifier
(fetch the pinned version, not latest), and recover deps in the
--no-deps branch too so a sdist-only transitive dependency is still
scanned instead of silently skipped.
- Baseline: key by package-relative path, not basename, so a future
same-named file in another directory is not auto-suppressed.
Regenerated the baseline accordingly.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* ci: add advisory lockfile supply-chain audit
Adds a fast, focused workflow that scans every checked-in npm and
cargo lockfile on PRs touching one. Default behaviour is advisory:
only public indicator-of-compromise strings, versions on the public
known-malicious list, and structurally broken lockfiles fail the
build. Structural anomalies (missing integrity hashes, non-default
registry, etc.) surface as :⚠️: annotations without gating
merges, so reviewers see the audit result inline on every PR
without changing the existing install behaviour.
Also commits the two missing npm lockfiles the audit needs:
studio/package-lock.json (Tauri CLI holder for desktop release)
and studio/backend/core/data_recipe/oxc-validator/package-lock.json
(oxc-parser runtime for the data-recipe validator). studio/setup.sh,
studio/setup.ps1, build.sh, and pyproject.toml are intentionally
left alone so the existing install path keeps working unchanged.
Audit script behaviour:
default mode -> exits 1 only on blocked-known-malicious,
known-ioc-string, malformed-lockfile,
missing-lockfile, unreadable-lockfile, or
missing-toml-parser
--strict -> promotes every finding to blocking (opt-in)
Adds a try/except around lockfile reads so a permissions error
prints a finding instead of crashing CI with a raw traceback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test(security): update cargo regression test for advisory mode
`scripts/lockfile_supply_chain_audit.py` now classifies
`non-registry-cargo-source` as an advisory finding by default
(returns exit 0 with a `:⚠️:` annotation) rather than
unconditionally blocking with exit 1. Update the existing
`test_malicious_cargo_lockfile_refused` to pass --strict so it
keeps verifying the "refuse to install" behavior it is named for,
and add a second test that pins the default-mode behavior:
advisory finding emitted, exit code 0.
* audit: escape Finding for GH Actions annotations
`:⚠️:` and `::error::` workflow commands truncate the
annotation message at the first newline unless the message is
%-encoded per the workflow-commands spec. Since `Finding.__str__`
returns three lines (kind+path, package, detail), the package
and detail fields were being dropped from the GitHub Actions UI.
Add a `_gha_escape()` helper that applies the spec'd escapes
(`%` -> `%25`, then `\r` -> `%0D`, then `\n` -> `%0A`; the `%`
replacement must happen first so the subsequent escapes are not
double-encoded), wrap every Finding rendered into a workflow
command with it, and pin both the helper and the end-to-end
single-line emission with two new regression tests.
Caught by gemini-code-assist on PR #5604.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* scripts/scan_*: add Mini Shai-Hulud May-12 IOC strings and pin-blocklists
Append the May-12 2026 wave indicators (git-tanstack.com, transformers.pyz,
/tmp/transformers.pyz, "With Love TeamPCP", "We've been online over 2 hours")
to all three scanner IOC tables, add BLOCKED_NPM_VERSIONS (42 TanStack pkgs,
4 opensearch versions, 3 squawk pkgs) in scan_npm_packages.py and
lockfile_supply_chain_audit.py (kept byte-identical), add BLOCKED_PYPI_VERSIONS
(guardrails-ai 0.10.1, mistralai 2.4.6, lightning 2.6.2/2.6.3) plus
RE_MAY12_IOC wiring across check_py_file/check_shell_file/check_workflow_file
in scan_packages.py. The npm orchestrator and the lockfile auditor now
short-circuit on a blocked entry before fetching the tarball, and the
PyPI download pipeline drops blocked specs before pip download is invoked.
* tests/security: regression suite for supply-chain scanners
Adds offline fixture corpus and pytest coverage for scan_npm_packages,
scan_packages, and lockfile_supply_chain_audit so future IOC-table
drift surfaces at PR time. Pytest scope narrowed to tests/security so
GPU smoke tests are not picked up by default.
* ci(security-audit): drop continue-on-error on pip-scan and npm-scan jobs
Promote three harden-runner blocks to egress-policy: block with per-job allowlists.
Add tests-security job running pytest tests/security as a hard gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts: harden third-party downloads, pip resolver pins, atomic writes
Pins uv installer and mlx_vlm qwen3_5 patches by commit SHA + SHA-256
checksum, scrubs PIP_* env vars and forces --index-url + --only-binary
on pip download, applies tarbomb caps to scan_packages archive walks,
and converts non-atomic config writes (kwargs spacer, studio stamper,
notebook validator, scan_packages req-file fixer) to mkstemp+os.replace.
Also adds host allowlist to notebook_to_python downloader, threads an
--allow-shell flag through its shell=True emission with reviewer warning
comments, locks both MLX installer scripts to set -euo pipefail, and
extends CODEOWNERS so colab snapshot data files require notebook-owner
review.
* ci(workflows): harden release-desktop / smoke / notebooks workflows
Pin dtolnay/rust-toolchain to a 40-char SHA, scope release-desktop permissions to read at workflow level with job-level write only on the build job, append --ignore-scripts to every npm ci / npm install in studio-frontend-ci / wheel-smoke / studio-tauri-smoke / release-desktop, validate client_payload.ref shape via an env-var-isolated regex on every notebooks-ci job, and add step-security/harden-runner in audit mode as the first step of release-desktop and mlx-ci.
* scripts: promote silent scanner failures to non-zero exit codes
scan_packages now returns 2 on pip-download failure and emits a CRITICAL archive_corrupted finding on truncated wheels/sdists.
notebook_to_python exits 1 on per-notebook failures; notebook_validator wraps the stash/pop in try/finally; lockfile audit rejects bare UNSLOTH_LOCKFILE_AUDIT_SKIP=1 with a loud GitHub Actions warning.
* Add npm cooldown + new-install-script gate + Dependabot cooldown
Pins min-release-age=7 (npm 11.10+) in repo-root and studio/frontend
.npmrc, adds scripts/check_new_install_scripts.py to fail PRs that
add a postinstall dep, ships a new security-audit job for npm audit
signatures plus the diff, and extends .github/dependabot.yml with
cooldown stanzas. Pin @tanstack/react-router to 1.169.9 per GHSA-
g7cv-rxg3-hmpx; lockfile regen deferred until that release lands on
npm. tests/security gains 4 new tests; full suite 26/26 green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(security): fix tanstack pin, exec bits, expand IOC tables to @uipath/@squawk full
- Revert --ignore-scripts on Studio install workflows: vite build needs
esbuild's native postinstall (per PR #5392 rationale). Keep
--ignore-scripts on security-audit.yml's standalone npm audit job.
- Pin @tanstack/react-router to the actual published 1.169.2 (was a
forward-looking 1.169.9 that does not exist on npm; broke npm ci).
- Drop redundant repo-root .npmrc; studio/frontend/.npmrc covers the
only npm project today (root cooldown re-instate via dependabot.yml).
- Restore exec bits on 7 files my filesystem stripped during cherry-pick.
- Expand BLOCKED_NPM_VERSIONS with full safedep.io + Aikido enumeration:
22 @squawk/* packages with 5 versions each (110 entries; previously
3 entries with 1 version each), and 66 @uipath/* packages (entirely
missing before). Mirror in scripts/lockfile_supply_chain_audit.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests/security: suppress CodeQL py/incomplete-url-substring-sanitization
The two flagged 'X' in Y assertions are NOT URL sanitization checks.
They verify our scanner WROTE a known IOC literal into its stdout /
Finding.evidence, which is the opposite of an attack surface --
matching the scanner's output is precisely what catches the worm.
Inline lgtm[] suppression with a 4-line rationale comment above each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts/scan_*: expand IOC tables with Aikido full 169-pkg enumeration
Per Aikido 2026-05-12 disclosure (373 malicious package-version entries
across 169 npm package names), add to BLOCKED_NPM_VERSIONS:
- @mistralai/* npm scope (3 packages, 9 versions) -- separate from
the PyPI mistralai package already in BLOCKED_PYPI_VERSIONS
- @tallyui/* (10 packages, 30 entries)
- @beproduct/nestjs-auth (18 versions 0.1.2..0.1.19)
- @draftlab/* + @draftauth/* (5 packages)
- @taskflow-corp/cli, @tolka/cli, @ml-toolkit-ts/*, @mesadev/*,
@dirigible-ai/sdk, @supersurkhet/*
- 10 unscoped packages (safe-action, ts-dna, cross-stitch,
cmux-agent-mcp, agentwork-cli, git-branch-selector, wot-api,
git-git-git, nextmove-mcp, ml-toolkit-ts)
Also add to KNOWN_IOC_STRINGS / NPM_IOC_STRINGS:
- router_init.js SHA-256 ab4fcadaec49c03278063dd269ea5eef82d24f2124a8e15d7b90f2fa8601266c
- tanstack_runner.js SHA-256 2ec78d556d696e208927cc503d48e4b5eb56b31abc2870c2ed2e98d6be27fc96
- bun run tanstack_runner.js marker (the new Bun-prepare-script
dropper invocation pattern unique to this wave)
Total: 170 packages, 401 versions blocklisted. Studio lockfile still
scans clean (0 findings, 0 hard errors).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scripts/scan_*: web-verification additions (@tanstack/setup, intercom-client)
Two findings from cross-checking BLOCKED_NPM_VERSIONS / KNOWN_IOC_STRINGS
against GHSA-g7cv-rxg3-hmpx + Aikido + safedep.io + Socket + Semgrep.
- Fix asymmetry: @tanstack/setup IOC string was in
lockfile_supply_chain_audit.py's NPM_IOC_STRINGS but missing from
scan_npm_packages.py's KNOWN_IOC_STRINGS. The literal is the malicious
optional-dependency name used by the May-12 TanStack wave; no
legitimate npm package of this name exists.
- Add intercom-client@7.0.4: the npm counterpart of the lightning
2.6.2/2.6.3 PyPI compromise (Apr-30 wave). Same threat actor
(TeamPCP). Confirmed by Semgrep, Aikido, OX Security, Resecurity,
Kodem. Safe version is 7.0.3 and earlier.
Total BLOCKED_NPM_VERSIONS: 171 packages / 402 versions. Both files
remain byte-identical. Studio lockfile still scans clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(security): add workflow-trigger lint refusing pull_request_target + cache-poisoning vectors
The two patterns that together powered GHSA-g7cv-rxg3-hmpx (TanStack
Mini Shai-Hulud) are now gated at PR time:
1. pull_request_target -- the worm chain started with a fork PR that
ran in the base-repo context. Every workflow in this repo today
uses 'pull_request' (safe); the lint refuses any new
pull_request_target additions outright. workflow_run is
restricted, allowed only with an explicit allow-comment.
2. Shared cache keys between PR-triggered workflows and the publish
workflow (release-desktop.yml). The TanStack attack chain poisoned
a shared Actions cache from a fork PR; the legitimate release
workflow then restored the poisoned cache. The lint refuses any
cache key that appears in both a PR-triggered workflow and a
workflow_dispatch-only / publish workflow.
Current tree is clean: 0 pull_request_target, 0 workflow_run, 0
PR-publish cache-key collisions across all 24 workflows. The lint
locks that invariant in place.
Files:
+ scripts/lint_workflow_triggers.py (~200 LOC, stdlib + PyYAML)
+ tests/security/test_lint_workflow_triggers.py (5 tests covering
current-tree pass, pull_request_target reject, workflow_run
restricted, justified workflow_run accept, cache-key collision
reject)
~ .github/workflows/security-audit.yml: new workflow-trigger-lint
job, no continue-on-error, harden-runner block-mode, PyYAML only
runtime dep.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* security: fix tests-security CI job + CodeQL false-positives
Two CI failures on the prior push:
1. pytest tests/security -- 5 lint regression tests failed because
scripts/lint_workflow_triggers.py imports PyYAML which is not in
the bare runner's Python env. Added pyyaml==6.0.2 to the pip
install step alongside pytest. (29 scanner tests already passed.)
2. CodeQL py/incomplete-url-substring-sanitization fired on two
test assertions that check the scanner WROTE the IOC literal
to its own stdout/stderr. The rule pattern-matches on
`"<host>" in <var>` and cannot distinguish a URL sanitizer from
a regression-test evidence check. Previous `# lgtm[...]` inline
suppressions were detached from the operator when pre-commit
reformatted the assert across multiple lines. Rebuilt the IOC
literals at runtime (`"git-tanstack." + "com"`) so no URL-shaped
source literal appears on the `in` operator line; rule cannot
trigger.
Verified locally: `pytest tests/security -v` -> 34 passed in 2.70s.
* security(studio): defensive .npmrc cooldown aliases + save-exact
Two additions to studio/frontend/.npmrc to harden the existing
`min-release-age=7` (Mini Shai-Hulud defence):
1. `minimum-release-age=10080` (minutes) -- defensive alias for the
same 7-day floor. Some npm versions / wrappers consult one key but
not the other; setting both prevents a single upstream setting-name
parse change from silently disabling the cooldown. The two keys
MUST agree (do not let them drift).
2. `save-exact=true` -- refuses to write back `^x.y.z` ranges into
package.json when a maintainer runs `npm install <pkg>` locally.
Does NOT rewrite already-present ranges; stops NEW carets from
creeping into the manifest as patch-version footguns.
Verified: pytest tests/security -> 34 passed in 2.63s.
* chore(dependabot): remove dead bun entry for /studio/frontend
`package-ecosystem: "bun"` at /studio/frontend was a no-op: that
path commits package-lock.json, not bun.lock / bun.lockb, so
Dependabot's bun ecosystem silently skipped it. The actual
behaviour is unchanged -- the npm entry below the cargo block
already owns npm_and_yarn security advisories for /studio/frontend
with `open-pull-requests-limit: 0` (version-update PRs suppressed,
security PRs flow through).
This commit:
- Deletes the bun entry (kept a placeholder comment so a future
bun migration knows where to slot it back in).
- Rewrites the npm /studio/frontend entry comment to explain the
real intent: lockfile is the authoritative pin, .npmrc
`min-release-age=7` already blocks fresh tarballs at install
time, dependabot only needs to surface security advisories.
No functional change: same set of dependabot PRs as before (zero
version updates, security advisories grouped weekly with cooldown).
Verified: pytest tests/security -> 34 passed in 2.67s; YAML
parses cleanly via PyYAML.
* fix(dependabot): drop unsupported semver-* cooldown keys on github-actions
Dependabot's validator rejected the config with:
The property '#/updates/0/cooldown/semver-minor-days' is not
supported for the package ecosystem 'github-actions'.
The property '#/updates/0/cooldown/semver-patch-days' is not
supported for the package ecosystem 'github-actions'.
The `semver-minor-days` / `semver-patch-days` cooldown knobs are
only valid for semver-aware ecosystems (npm, cargo, etc.). The
github-actions ecosystem pins via git tags / SHAs, not semver, so
only `default-days` is honored. Pre-existing bug on main; surfaced
on this PR because the prior commit re-validated the file.
Behaviour: github-actions PRs now respect the 7-day cooldown floor
(was already the intent), without the no-op semver bands.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han