1696a15cf8
Recent supply-chain incidents that scan_packages would have missed:
- PyTorch Lightning 2.6.x: payload in _runtime/router_runtime.js
(14.8 MB), persistence via .claude/settings.json SessionStart
and .vscode/tasks.json folderOpen
- npm chalk/debug + Shai-Hulud: hex-var obfuscation, window.ethereum
Web3 hijack, .github/workflows/shai-hulud.yml repo takeover,
trufflehog credential exfil
- elementary-data 0.23.3: token harvesters with embedded gh{p,o,s}_
and AKIA regexes
- litellm 1.82.7: also covered by existing patterns, but anyone on
`>=` got it during the 40-min exposure window
- langchain-core CVE-2025-68664 / n8n CVE-2025-68668 / marimo
CVE-2026-39987: first-party design flaws, not malicious-author
scan_packages.py:
- Six new regexes: RE_DEV_TOOL_HIJACK, RE_TOKEN_REGEX,
RE_JS_OBFUSCATION, RE_WEB3_HIJACK, RE_WORKFLOW_INJECT,
RE_SHELL_DROPPER.
- Three new checkers: check_js_file, check_shell_file,
check_workflow_file. scan_archive now routes .js/.mjs/.cjs/.ts
to the JS checker, .sh/.bash to the shell checker, and
.github/workflows/*.yml to the workflow checker.
- JS checker fires CRITICAL on hex-var obfuscation OR Web3 hijack
OR (token regex + network) OR workflow-injection signature; HIGH
on a >100 KB JS bundle inside a Python wheel (the Lightning tell).
- Smoke-tested: every new pattern matches its canonical positive
and rejects four legitimate-looking false-positive baits.
security-audit.yml:
- OSV-Scanner step: cross-ecosystem advisory check (PyPI + npm
+ cargo) from one binary. OSV's feed is a superset of GitHub-
Advisory; catches CVEs that haven't propagated yet (e.g.
langchain-core was on OSV before GitHub Advisory).
- Semgrep step: p/supply-chain + p/python + p/javascript +
p/security-audit packs catch first-party logic bugs (CVEs 7/9/10
above) that pattern scanning never sees.
- Lockfile pin verifier: warns on every non-`==` spec in
requirements/*.txt. Currently surfaces 104 unpinned specs as
informational baseline; tighten to blocking once the baseline
is curated.
All new steps continue-on-error initially; they surface findings to
the workflow summary + advisory-audit-logs artifact.
572 lines
26 KiB
YAML
572 lines
26 KiB
YAML
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
|
|
|
# Multi-language supply-chain audit. Triggers:
|
|
# - PRs touching any dependency manifest (Python / npm / Cargo) or
|
|
# this workflow file,
|
|
# - push to main / pip,
|
|
# - nightly @ 04:13 UTC so newly-published advisories surface even
|
|
# when no PR opens,
|
|
# - workflow_dispatch for ad-hoc invocations.
|
|
#
|
|
# Two jobs:
|
|
# - advisory-audit: one runner that runs pip-audit + npm audit +
|
|
# cargo audit back-to-back. All three are
|
|
# advisory-DB lookups -- fast, lockfile-driven,
|
|
# no archive download. Setting up the python /
|
|
# node / rust toolchains on one runner and
|
|
# running the three commands serially is
|
|
# cheaper than spinning up three runners.
|
|
# - pip-scan-packages: 3-shard matrix that downloads + pattern-scans
|
|
# every PyPI archive in the transitive closure.
|
|
# This is the expensive job (~6 min/shard,
|
|
# running in parallel) and it must stay
|
|
# independent so a CVE-DB hit in advisory-audit
|
|
# does not block the supply-chain pattern scan
|
|
# (or vice versa).
|
|
#
|
|
# All steps are non-blocking initially. The default branch already
|
|
# carries a known-vuln backlog (the dependabot banner shows 17 today,
|
|
# pip-audit catches 2 more, npm/cargo will catch their own); a hard
|
|
# gate now would block every PR on a baseline we have not triaged.
|
|
# As each baseline closes, drop continue-on-error per step.
|
|
#
|
|
# Dependency coverage:
|
|
# - unsloth core (pyproject.toml [project.dependencies])
|
|
# - unsloth `huggingfacenotorch` extras (the canonical install path
|
|
# for fine-tuning users; pulls transformers / peft / accelerate /
|
|
# trl / datasets / diffusers / sentence-transformers / etc.)
|
|
# - all six Studio backend requirements files
|
|
# - Studio frontend (npm) and Tauri shell (cargo)
|
|
# Each Python step builds a filtered dep list from pyproject.toml +
|
|
# requirements/*.txt before auditing. We do NOT install any of these
|
|
# -- pip-audit resolves through PyPI metadata, scan_packages.py
|
|
# downloads sdist/wheel archives and inspects them without running
|
|
# install hooks, so an attacker who has compromised a transitive dep
|
|
# cannot execute code in this workflow.
|
|
|
|
name: Security audit
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- 'studio/backend/requirements/**'
|
|
- 'studio/frontend/package.json'
|
|
- 'studio/frontend/package-lock.json'
|
|
- 'studio/src-tauri/Cargo.toml'
|
|
- 'studio/src-tauri/Cargo.lock'
|
|
- 'pyproject.toml'
|
|
- 'scripts/scan_packages.py'
|
|
- '.github/workflows/security-audit.yml'
|
|
push:
|
|
branches: [main, pip]
|
|
schedule:
|
|
- cron: '13 4 * * *' # 04:13 UTC daily, off the cron rush
|
|
workflow_dispatch:
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
# Combined advisory-DB audit: pip-audit + npm audit + cargo audit
|
|
# all on one runner. Each step is continue-on-error so a finding in
|
|
# one toolchain does not suppress the others.
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
advisory-audit:
|
|
name: advisory audit (pip + npm + cargo)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.12'
|
|
cache: 'pip'
|
|
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '22'
|
|
cache: 'npm'
|
|
cache-dependency-path: studio/frontend/package-lock.json
|
|
|
|
- uses: dtolnay/rust-toolchain@stable
|
|
|
|
- uses: swatinem/rust-cache@v2
|
|
with:
|
|
workspaces: studio/src-tauri -> target
|
|
|
|
- name: Install pip-audit + cargo-audit
|
|
# cargo-audit pulls advisories from the RustSec advisory-db on
|
|
# first run and caches them under ~/.cargo/advisory-db. Pin
|
|
# --locked so the version we install matches Cargo.lock
|
|
# determinism.
|
|
# npm audit is bundled with the node toolchain, no install.
|
|
run: |
|
|
python -m pip install --upgrade pip 'pip-audit>=2.7'
|
|
cargo install --locked --version '^0.21' cargo-audit
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Python: pip-audit
|
|
# ─────────────────────────────────────────────────────────────
|
|
- name: Build filtered Python requirements set
|
|
# Two transforms:
|
|
# (1) Generate audit-reqs/unsloth-deps.txt from pyproject.toml
|
|
# so pip-audit sees the unsloth pip package's own dep set
|
|
# (core + huggingfacenotorch extras: transformers / peft /
|
|
# accelerate / trl / datasets / diffusers /
|
|
# sentence-transformers / huggingface_hub / hf_transfer /
|
|
# etc.).
|
|
# (2) Copy each studio/backend/requirements/*.txt into
|
|
# audit-reqs/ with `git+` lines stripped. pip-audit's `-r`
|
|
# mode does a dry-run resolve against PyPI metadata; a
|
|
# `git+https://...` spec forces it to clone, which is
|
|
# both slow and outside the threat model (we audit
|
|
# PyPI-served archives; a git ref is whatever HEAD says
|
|
# on the runner). A comment line is left in place so the
|
|
# skipped specs are obvious in the artifact.
|
|
# The `huggingface` extra is `huggingfacenotorch` plus torch /
|
|
# torchvision / triton, deliberately skipped: Studio backend
|
|
# already pins a torch and the +cu* / +cpu local-version tags
|
|
# trip up the PyPI resolver in `-r` mode.
|
|
run: |
|
|
mkdir -p audit-reqs
|
|
python <<'PY' > audit-reqs/unsloth-deps.txt
|
|
import tomllib
|
|
with open("pyproject.toml", "rb") as f:
|
|
d = tomllib.load(f)
|
|
core = d["project"]["dependencies"]
|
|
extras = d["project"]["optional-dependencies"]["huggingfacenotorch"]
|
|
print("# Auto-generated from pyproject.toml by security-audit.yml.")
|
|
print("# core deps + huggingfacenotorch extras.")
|
|
for spec in core + extras:
|
|
print(spec)
|
|
PY
|
|
for f in studio.txt extras.txt extras-no-deps.txt \
|
|
no-torch-runtime.txt overrides.txt triton-kernels.txt; do
|
|
python <<PY > "audit-reqs/$f"
|
|
src = "studio/backend/requirements/$f"
|
|
with open(src) as fh:
|
|
for line in fh:
|
|
stripped = line.strip()
|
|
before_comment = stripped.split("#", 1)[0]
|
|
if "git+" in before_comment:
|
|
print(f"# [security-audit] skipped git+ spec: {stripped}")
|
|
continue
|
|
print(line.rstrip("\n"))
|
|
PY
|
|
done
|
|
|
|
- name: pip-audit (declared Python deps, no install)
|
|
# `-r requirements.txt` resolves the requirements through pip's
|
|
# dependency resolver against PyPI metadata and audits the
|
|
# resolved tree without ever executing setup.py / install
|
|
# hooks. Way faster than installing the full Studio runtime
|
|
# and -- critically -- safer: an attacker who has compromised
|
|
# a transitive dep cannot run code in this job.
|
|
#
|
|
# extras.txt + extras-no-deps.txt have legacy setup.py
|
|
# packages (notably openai-whisper) whose setup.py imports
|
|
# `pkg_resources`, which the isolated build env's current
|
|
# setuptools no longer ships. PIP_CONSTRAINT pins an older
|
|
# setuptools into the build env so those builds resolve.
|
|
# Per-file loop so one bad file doesn't take out the whole
|
|
# audit.
|
|
continue-on-error: true
|
|
env:
|
|
PIP_CONSTRAINT: ${{ github.workspace }}/audit-reqs/build-constraints.txt
|
|
run: |
|
|
set +e
|
|
cat > audit-reqs/build-constraints.txt <<'CONSTRAINTS'
|
|
setuptools<78
|
|
wheel
|
|
CONSTRAINTS
|
|
: > logs-pip-audit.txt
|
|
for f in unsloth-deps studio extras extras-no-deps \
|
|
no-torch-runtime overrides triton-kernels; do
|
|
if ! grep -qE '^[^#[:space:]]' "audit-reqs/$f.txt"; then
|
|
echo "[security-audit] $f.txt has no PyPI specs after git+ filter, skipping" \
|
|
| tee -a logs-pip-audit.txt
|
|
continue
|
|
fi
|
|
echo "::group::pip-audit -r audit-reqs/$f.txt"
|
|
{
|
|
echo
|
|
echo "=== $f ==="
|
|
pip-audit -r "audit-reqs/$f.txt" --format=columns
|
|
echo "=== end $f (rc=$?) ==="
|
|
} 2>&1 | tee -a logs-pip-audit.txt
|
|
echo "::endgroup::"
|
|
done
|
|
{
|
|
echo "## pip-audit (Python)"
|
|
echo
|
|
echo '### Coverage'
|
|
echo '- unsloth core + `huggingfacenotorch` extras (pyproject.toml)'
|
|
echo '- studio/backend/requirements/{studio,extras,extras-no-deps,no-torch-runtime,overrides,triton-kernels}.txt'
|
|
echo '- `git+` specs are stripped before audit (out of scope: we audit PyPI archives)'
|
|
echo
|
|
echo '### Findings'
|
|
echo '```'
|
|
cat logs-pip-audit.txt
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# npm: Studio frontend
|
|
# ─────────────────────────────────────────────────────────────
|
|
- name: npm audit (Studio frontend)
|
|
# `npm audit` resolves the lockfile through the npmjs.com
|
|
# advisory DB. `--audit-level=high` filters the noise floor
|
|
# to only HIGH and CRITICAL. We do NOT pass --omit=dev: a
|
|
# malicious dev-only dep can still steal secrets from a CI
|
|
# runner, so dev deps need to be in the audit surface.
|
|
continue-on-error: true
|
|
working-directory: studio/frontend
|
|
run: |
|
|
set +e
|
|
npm audit --audit-level=high | tee ../../logs-npm-audit.txt
|
|
# Always also write the full JSON for grep-ability.
|
|
npm audit --json > ../../logs-npm-audit.json || true
|
|
{
|
|
echo "## npm audit (Studio frontend)"
|
|
echo
|
|
echo '```'
|
|
tail -200 ../../logs-npm-audit.txt
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# cargo: Studio Tauri shell
|
|
# ─────────────────────────────────────────────────────────────
|
|
- name: cargo audit (Studio Tauri)
|
|
# `--deny warnings` would make the job fail on any advisory.
|
|
# Keep non-blocking initially; drop continue-on-error after
|
|
# the baseline closes.
|
|
continue-on-error: true
|
|
working-directory: studio/src-tauri
|
|
run: |
|
|
set +e
|
|
cargo audit | tee ../../logs-cargo-audit.txt
|
|
{
|
|
echo "## cargo audit (Studio Tauri)"
|
|
echo
|
|
echo '```'
|
|
tail -200 ../../logs-cargo-audit.txt
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# OSV-Scanner: cross-ecosystem advisory DB (PyPI + npm + cargo)
|
|
# ─────────────────────────────────────────────────────────────
|
|
- name: OSV-Scanner (PyPI + npm + cargo, cross-ecosystem advisories)
|
|
# OSV's advisory feed is a superset of GitHub-Advisory + RustSec
|
|
# + npm advisories; running it alongside the per-ecosystem audit
|
|
# tools catches CVEs that haven't propagated to the per-ecosystem
|
|
# DBs yet (e.g. langchain-core CVE-2025-68664 was on OSV before
|
|
# GitHub Advisory). Single binary, one transitive resolver, all
|
|
# three lockfile types in one pass. Non-blocking until baselines
|
|
# close.
|
|
continue-on-error: true
|
|
run: |
|
|
set +e
|
|
curl -fsSL -o /tmp/osv-scanner.tar.gz \
|
|
https://github.com/google/osv-scanner/releases/download/v2.0.2/osv-scanner_linux_amd64.tar.gz
|
|
tar -xzf /tmp/osv-scanner.tar.gz -C /tmp osv-scanner
|
|
/tmp/osv-scanner --version
|
|
/tmp/osv-scanner scan source \
|
|
--lockfile=studio/frontend/package-lock.json \
|
|
--lockfile=studio/src-tauri/Cargo.lock \
|
|
--lockfile=requirements.txt:audit-reqs/unsloth-deps.txt \
|
|
--lockfile=requirements.txt:audit-reqs/studio.txt \
|
|
--lockfile=requirements.txt:audit-reqs/no-torch-runtime.txt \
|
|
--lockfile=requirements.txt:audit-reqs/overrides.txt \
|
|
--lockfile=requirements.txt:audit-reqs/extras.txt \
|
|
--lockfile=requirements.txt:audit-reqs/extras-no-deps.txt \
|
|
--format=table 2>&1 | tee logs-osv-scanner.txt
|
|
{
|
|
echo "## OSV-Scanner (cross-ecosystem)"
|
|
echo
|
|
echo '```'
|
|
tail -200 logs-osv-scanner.txt
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Semgrep: design-flaw detection (catches what regex-pattern
|
|
# scanning of malicious authors cannot — first-party logic bugs
|
|
# like langchain-core CVE-2025-68664 dumps/dumpd injection,
|
|
# n8n CVE-2025-68668 _pyodide.eval_code sandbox escape, marimo
|
|
# CVE-2026-39987 unauth WebSocket).
|
|
# ─────────────────────────────────────────────────────────────
|
|
- name: Semgrep (supply-chain + python rule packs)
|
|
continue-on-error: true
|
|
run: |
|
|
set +e
|
|
python -m pip install --quiet 'semgrep>=1.95'
|
|
semgrep --version
|
|
semgrep scan \
|
|
--config p/supply-chain \
|
|
--config p/python \
|
|
--config p/javascript \
|
|
--config p/security-audit \
|
|
--severity ERROR --severity WARNING \
|
|
--metrics off \
|
|
--timeout 120 \
|
|
studio/backend unsloth scripts \
|
|
2>&1 | tee logs-semgrep.txt
|
|
{
|
|
echo "## Semgrep (supply-chain + python + javascript rules)"
|
|
echo
|
|
echo '```'
|
|
tail -200 logs-semgrep.txt
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Lockfile pin verifier. The litellm 1.82.7 attack window was
|
|
# ~40 minutes; anyone resolving with `>=` got the malicious
|
|
# version automatically. Flag every spec in the requirements
|
|
# files that does not pin to an exact `==` (or `@` for git
|
|
# refs, or `===` for arbitrary equality). Warning-only for now;
|
|
# graduate to blocking once the baseline is clean.
|
|
# ─────────────────────────────────────────────────────────────
|
|
- name: Lockfile pin verifier (Python requirements)
|
|
continue-on-error: true
|
|
run: |
|
|
python <<'PY' | tee logs-pin-verifier.txt
|
|
import re
|
|
from pathlib import Path
|
|
|
|
# Specs that look like `pkg==1.2.3` or `pkg @ git+...` or
|
|
# bare comments / -r lines are pinned-or-not-applicable.
|
|
PINNED = re.compile(r"^\s*[A-Za-z0-9_.\-]+\s*(?:===|==)\s*[^,;]+\s*$")
|
|
GIT_OR_URL = re.compile(r"^\s*[A-Za-z0-9_.\-]+\s*@\s*(?:git\+|https?://)")
|
|
|
|
unpinned = []
|
|
for f in sorted(Path("studio/backend/requirements").glob("*.txt")):
|
|
for i, raw in enumerate(f.read_text().splitlines(), 1):
|
|
line = raw.strip()
|
|
if not line or line.startswith("#") or line.startswith("-"):
|
|
continue
|
|
spec = line.split("#", 1)[0].strip().split(";", 1)[0].strip()
|
|
if not spec:
|
|
continue
|
|
if "git+" in spec or PINNED.match(spec) or GIT_OR_URL.match(spec):
|
|
continue
|
|
unpinned.append((str(f), i, line))
|
|
|
|
print(f"::group::Lockfile pin status")
|
|
if unpinned:
|
|
print(f"WARN: {len(unpinned)} non-`==` specs across requirements/*.txt")
|
|
print("(litellm 1.82.7 wave hit anyone on `>=`; tighten when feasible.)")
|
|
for f, i, line in unpinned[:80]:
|
|
print(f" {f}:{i}: {line}")
|
|
if len(unpinned) > 80:
|
|
print(f" ... and {len(unpinned) - 80} more")
|
|
else:
|
|
print("OK: every spec is exact-pinned.")
|
|
print("::endgroup::")
|
|
PY
|
|
{
|
|
echo "## Lockfile pin verifier"
|
|
echo
|
|
echo '```'
|
|
cat logs-pin-verifier.txt
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: advisory-audit-logs
|
|
path: |
|
|
logs-pip-audit.txt
|
|
logs-npm-audit.txt
|
|
logs-npm-audit.json
|
|
logs-cargo-audit.txt
|
|
logs-osv-scanner.txt
|
|
logs-semgrep.txt
|
|
logs-pin-verifier.txt
|
|
audit-reqs/
|
|
retention-days: 30
|
|
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
# Python: pre-install package scan (no install, no execution)
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
pip-scan-packages:
|
|
# Downloads each declared dep WITHOUT installing it and inspects
|
|
# the archive contents for known malicious patterns: weaponized
|
|
# .pth files, credential stealers, obfuscated payloads,
|
|
# install-time droppers, suspicious subprocess / network /
|
|
# base64-blob combinations.
|
|
#
|
|
# This is the kind of check that would have caught:
|
|
# - litellm 1.82.7 / 1.82.8 (March 2026, supply-chain compromise)
|
|
# - the typo-squat campaign against PyTorch Lightning
|
|
# before either landed in the install path. pip-audit only knows
|
|
# about CVE-published vulnerabilities, so it does NOT see novel
|
|
# malicious uploads. scan_packages.py runs deterministic regex
|
|
# pattern matching, no LLM calls.
|
|
#
|
|
# `--with-deps` makes the scan transitive: every package the
|
|
# declared set resolves to gets fetched and pattern-scanned, not
|
|
# just the top-level pins. Resolving the full transitive closure
|
|
# of the unsloth + Studio dep tree downloads several hundred
|
|
# archives, hence the longer timeout.
|
|
#
|
|
# Sharded across runners for wall-clock parallelism. Each shard
|
|
# runs scan_packages.py once with --with-deps so its own slice
|
|
# benefits from pip's deduped transitive resolve. Shard
|
|
# composition tries to balance load:
|
|
# - hf-stack: pyproject extras + no-torch-runtime
|
|
# (~150 archives, transformers/peft/accelerate/...)
|
|
# - studio: FastAPI/Studio backend + overrides + extras-no-deps
|
|
# (~150 archives, smaller scientific stack)
|
|
# - extras: the heavy openai-whisper / scikit-learn / librosa
|
|
# stack (~250 archives, dominant cost)
|
|
# triton-kernels.txt is git+-only, fully skipped.
|
|
name: ${{ matrix.shard.name }}
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 25
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
shard:
|
|
- name: 'pip scan-packages :: hf-stack'
|
|
id: hf-stack
|
|
files: 'unsloth-deps no-torch-runtime'
|
|
- name: 'pip scan-packages :: studio'
|
|
id: studio
|
|
files: 'studio overrides extras-no-deps'
|
|
- name: 'pip scan-packages :: extras'
|
|
id: extras
|
|
files: 'extras'
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.12'
|
|
cache: 'pip'
|
|
|
|
- name: Install scan_packages.py runtime deps
|
|
# scan_packages.py imports requests + packaging at runtime to
|
|
# talk to PyPI's JSON API and to parse version specifiers. We
|
|
# do not install the packages it scans -- those are downloaded
|
|
# raw and inspected without ever touching `pip install`.
|
|
run: python -m pip install --upgrade pip requests packaging
|
|
|
|
- name: Build filtered requirements set
|
|
# Mirrors the advisory-audit job's input transform: pyproject.toml
|
|
# extraction + git+ stripping. scan_packages.py downloads
|
|
# PyPI archives without building, so it tolerates legacy
|
|
# setup.py packages (no resolver dry-run); but `--with-deps`
|
|
# delegates resolution to a single `pip download` call that
|
|
# cannot satisfy `git+` specs without git operations, so we
|
|
# strip them here too.
|
|
run: |
|
|
mkdir -p audit-reqs
|
|
python <<'PY' > audit-reqs/unsloth-deps.txt
|
|
import tomllib
|
|
with open("pyproject.toml", "rb") as f:
|
|
d = tomllib.load(f)
|
|
core = d["project"]["dependencies"]
|
|
extras = d["project"]["optional-dependencies"]["huggingfacenotorch"]
|
|
print("# Auto-generated from pyproject.toml by security-audit.yml.")
|
|
print("# core deps + huggingfacenotorch extras.")
|
|
for spec in core + extras:
|
|
print(spec)
|
|
PY
|
|
for f in studio.txt extras.txt extras-no-deps.txt \
|
|
no-torch-runtime.txt overrides.txt triton-kernels.txt; do
|
|
python <<PY > "audit-reqs/$f"
|
|
src = "studio/backend/requirements/$f"
|
|
with open(src) as fh:
|
|
for line in fh:
|
|
stripped = line.strip()
|
|
before_comment = stripped.split("#", 1)[0]
|
|
if "git+" in before_comment:
|
|
print(f"# [security-audit] skipped git+ spec: {stripped}")
|
|
continue
|
|
print(line.rstrip("\n"))
|
|
PY
|
|
done
|
|
|
|
- name: Sanity-check scan_packages.py
|
|
# The scanner lives at scripts/scan_packages.py in this repo
|
|
# so we don't depend on a network fetch at job time.
|
|
run: |
|
|
test -f scripts/scan_packages.py
|
|
head -3 scripts/scan_packages.py
|
|
grep -q "Standalone pre-install package scanner" scripts/scan_packages.py
|
|
|
|
- name: Scan declared + transitive Python deps
|
|
# scan_packages.py exits 1 on CRITICAL/HIGH findings, 0 on
|
|
# clean. We swallow the exit because the baseline isn't
|
|
# triaged yet; surface the findings in the workflow summary.
|
|
# Drop continue-on-error after the first clean run on main.
|
|
#
|
|
# `--with-deps` walks PyPI metadata to enumerate every
|
|
# transitive dep the declared set would install, then scans
|
|
# them all. Without this flag, we'd only catch a malicious
|
|
# *direct* dep -- and supply-chain attacks usually land
|
|
# several hops down (litellm 1.82.7 was a dep of a dep for
|
|
# most users).
|
|
#
|
|
# This step runs once per matrix shard. Within a shard, every
|
|
# -r file is fed to a single `pip download` call so pip
|
|
# intersects version constraints and yields a deduped
|
|
# transitive set (no point fetching the same transformers
|
|
# wheel five times). Across shards we accept some redundant
|
|
# downloads in exchange for wall-clock parallelism.
|
|
continue-on-error: true
|
|
env:
|
|
SHARD_FILES: ${{ matrix.shard.files }}
|
|
run: |
|
|
set +e
|
|
mkdir -p logs
|
|
LOG="logs-scan-packages-${{ matrix.shard.id }}.txt"
|
|
echo "::group::shard ${{ matrix.shard.id }} input files"
|
|
REQ_ARGS=()
|
|
for f in $SHARD_FILES; do
|
|
if grep -qE '^[^#[:space:]]' "audit-reqs/$f.txt"; then
|
|
echo " + audit-reqs/$f.txt"
|
|
REQ_ARGS+=( -r "audit-reqs/$f.txt" )
|
|
else
|
|
echo " - audit-reqs/$f.txt (empty after git+ filter, skipping)"
|
|
fi
|
|
done
|
|
echo "::endgroup::"
|
|
if [ ${#REQ_ARGS[@]} -eq 0 ]; then
|
|
echo "[security-audit] shard ${{ matrix.shard.id }}: no PyPI specs, nothing to scan" \
|
|
| tee "$LOG"
|
|
else
|
|
python scripts/scan_packages.py --with-deps "${REQ_ARGS[@]}" \
|
|
2>&1 | tee "$LOG"
|
|
fi
|
|
{
|
|
echo "## scan_packages :: shard ${{ matrix.shard.id }}"
|
|
echo
|
|
echo "### Files in this shard"
|
|
for f in $SHARD_FILES; do echo "- audit-reqs/$f.txt"; done
|
|
echo
|
|
echo '### Findings (tail)'
|
|
echo '```'
|
|
tail -200 "$LOG"
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: scan-packages-log-${{ matrix.shard.id }}
|
|
path: |
|
|
logs-scan-packages-${{ matrix.shard.id }}.txt
|
|
audit-reqs/
|
|
retention-days: 30
|