From f76e05a81df1dffa9606f4dddfdf6ab3db153f48 Mon Sep 17 00:00:00 2001 From: Daniel Han Date: Wed, 6 May 2026 22:47:01 +0000 Subject: [PATCH] CI(security): per-file audit, strip git+, pin setuptools in build env Last push surfaced two silent failures: 1. pip-audit aborted on openai-whisper. The package's setup.py imports pkg_resources, which the isolated build env's modern setuptools no longer ships by default. Because we passed every -r file in one invocation, that single build failure killed the audit for ALL files (the run reported success only because continue-on-error swallowed exit 1). 2. scan_packages --with-deps aborted on the first git+ spec it hit (triton-kernels.txt's git+https://github.com/triton-lang /triton.git, plus OpenEnv in extras-no-deps.txt). Same all-or-nothing behaviour: the entire transitive scan reported "0 archives downloaded" and "all clean" -- meaning we silently scanned nothing. Fixes: - Build a filtered audit-reqs/ tree first. Each Studio requirements file is copied with `git+` lines stripped (replaced with a `# [security-audit] skipped` marker so the exclusion is auditable in the artifact). Pure git refs are out of scope for both pip- audit (CVE DB only knows PyPI versions) and scan_packages (it inspects PyPI archives, not git HEADs). - Run pip-audit per-file in a loop. One bad file no longer takes out the whole audit. - Pin setuptools<78 + wheel into pip's isolated build env via PIP_CONSTRAINT, so legacy setup.py packages (openai-whisper) can still emit metadata for the resolver. - Run scan_packages per-file too, with the same git+ filter and a skip for files that are empty after filtering (triton-kernels.txt becomes a comments-only file and would otherwise spam the log with `--help`). Net effect: pip-audit now actually emits CVE findings (we know the default branch carries 17), and scan_packages downloads + pattern- scans the full transitive closure of every PyPI-only requirements file plus unsloth's pyproject deps. --- .github/workflows/security-audit.yml | 192 ++++++++++++++++++++------- 1 file changed, 142 insertions(+), 50 deletions(-) diff --git a/.github/workflows/security-audit.yml b/.github/workflows/security-audit.yml index c8eb62663a..247e84258c 100644 --- a/.github/workflows/security-audit.yml +++ b/.github/workflows/security-audit.yml @@ -75,25 +75,30 @@ jobs: - name: Install pip-audit (only) run: python -m pip install --upgrade pip 'pip-audit>=2.7' - - name: Build unsloth-deps.txt from pyproject.toml - # tomllib is in the stdlib on 3.11+. We extract: - # - [project.dependencies] (the four core deps) - # - [project.optional-dependencies].huggingfacenotorch - # (the no-torch fine-tuning bundle; pulls transformers / - # peft / accelerate / trl / datasets / diffusers / - # sentence-transformers / huggingface_hub / hf_transfer - # and friends -- the most realistic surface area for an - # unsloth user) + - name: Build filtered requirements set + # Two transforms: + # (1) Generate audit-reqs/unsloth-deps.txt from pyproject.toml + # so pip-audit sees the unsloth pip package's own dep set + # (core + huggingfacenotorch extras: transformers / peft / + # accelerate / trl / datasets / diffusers / + # sentence-transformers / huggingface_hub / hf_transfer / + # etc.). + # (2) Copy each studio/backend/requirements/*.txt into + # audit-reqs/ with `git+` lines stripped. pip-audit's `-r` + # mode does a dry-run resolve against PyPI metadata; a + # `git+https://...` spec forces it to clone, which is + # both slow and outside the threat model (we audit + # PyPI-served archives; a git ref is whatever HEAD says + # on the runner). A comment line is left in place so the + # skipped specs are obvious in the artifact. # The `huggingface` extra is `huggingfacenotorch` plus torch / - # torchvision / triton, which we deliberately skip here: the - # Studio backend requirements already pin a torch and - # pip-audit on Studio's matrix-resolved set audits the same - # ground. We avoid re-resolving torch from PyPI metadata - # because the +cu* / +cpu local-version tags trip up the - # resolver in `-r` mode. + # torchvision / triton, deliberately skipped: Studio backend + # already pins a torch and the +cu* / +cpu local-version tags + # trip up the PyPI resolver in `-r` mode. run: | - python <<'PY' > unsloth-deps.txt - import tomllib, sys + mkdir -p audit-reqs + python <<'PY' > audit-reqs/unsloth-deps.txt + import tomllib with open("pyproject.toml", "rb") as f: d = tomllib.load(f) core = d["project"]["dependencies"] @@ -103,9 +108,31 @@ jobs: for spec in core + extras: print(spec) PY - echo "::group::unsloth-deps.txt" - cat unsloth-deps.txt + for f in studio.txt extras.txt extras-no-deps.txt \ + no-torch-runtime.txt overrides.txt triton-kernels.txt; do + python < "audit-reqs/$f" + import re + src = "studio/backend/requirements/$f" + with open(src) as fh: + for line in fh: + stripped = line.strip() + # Skip pure git+ specs but leave a marker so the + # exclusion is auditable. + before_comment = stripped.split("#", 1)[0] + if "git+" in before_comment: + print(f"# [security-audit] skipped git+ spec: {stripped}") + continue + print(line.rstrip("\n")) + PY + done + echo "::group::audit-reqs/unsloth-deps.txt" + cat audit-reqs/unsloth-deps.txt echo "::endgroup::" + for f in audit-reqs/*.txt; do + echo "::group::$f" + cat "$f" + echo "::endgroup::" + done - name: Audit declared deps via -r (no install) # `-r requirements.txt` resolves the requirements through pip's @@ -114,28 +141,52 @@ jobs: # hooks. Way faster than installing the full Studio runtime # and -- critically -- safer: an attacker who has compromised # a transitive dep cannot run code in this job. - # Audits unsloth core + huggingfacenotorch extras alongside - # every committed Studio requirements file so cross-file - # conflicts and cross-stack vulns surface together. + # + # extras.txt + extras-no-deps.txt are audited separately + # (`continue-on-error` per step) because some of their members + # ship legacy setup.py scripts that the resolver tries to + # build at metadata-collection time. openai-whisper's setup.py + # imports `pkg_resources`, which the isolated build env's + # current setuptools no longer ships. PIP_CONSTRAINT pins an + # older setuptools into the build env so those builds resolve; + # if that still fails, the step continues so the rest of the + # audit completes. continue-on-error: true + env: + # Pin setuptools into pip's isolated build envs. This fixes + # the "ModuleNotFoundError: No module named 'pkg_resources'" + # raised when openai-whisper's setup.py imports it. + PIP_CONSTRAINT: ${{ github.workspace }}/audit-reqs/build-constraints.txt run: | set +e - pip-audit \ - -r unsloth-deps.txt \ - -r studio/backend/requirements/studio.txt \ - -r studio/backend/requirements/extras.txt \ - -r studio/backend/requirements/extras-no-deps.txt \ - -r studio/backend/requirements/no-torch-runtime.txt \ - -r studio/backend/requirements/overrides.txt \ - -r studio/backend/requirements/triton-kernels.txt \ - --format=columns \ - | tee logs-pip-audit.txt + cat > audit-reqs/build-constraints.txt <<'CONSTRAINTS' + setuptools<78 + wheel + CONSTRAINTS + : > logs-pip-audit.txt + for f in unsloth-deps studio extras extras-no-deps \ + no-torch-runtime overrides triton-kernels; do + if ! grep -qE '^[^#[:space:]]' "audit-reqs/$f.txt"; then + echo "[security-audit] $f.txt has no PyPI specs after git+ filter, skipping" \ + | tee -a logs-pip-audit.txt + continue + fi + echo "::group::pip-audit -r audit-reqs/$f.txt" + { + echo + echo "=== $f ===" + pip-audit -r "audit-reqs/$f.txt" --format=columns + echo "=== end $f (rc=$?) ===" + } 2>&1 | tee -a logs-pip-audit.txt + echo "::endgroup::" + done { echo "## pip-audit" echo echo '### Coverage' echo '- unsloth core + `huggingfacenotorch` extras (pyproject.toml)' echo '- studio/backend/requirements/{studio,extras,extras-no-deps,no-torch-runtime,overrides,triton-kernels}.txt' + echo '- `git+` specs are stripped before audit (out of scope: we audit PyPI archives)' echo echo '### Findings' echo '```' @@ -149,7 +200,7 @@ jobs: name: pip-audit-log path: | logs-pip-audit.txt - unsloth-deps.txt + audit-reqs/ retention-days: 30 # ───────────────────────────────────────────────────────────────────── @@ -193,20 +244,41 @@ jobs: # raw and inspected without ever touching `pip install`. run: python -m pip install --upgrade pip requests packaging - - name: Build unsloth-deps.txt from pyproject.toml - # Same extraction as pip-audit. Kept inline (rather than a - # shared composite action) to keep this workflow file - # self-contained and reviewable. + - name: Build filtered requirements set + # Mirrors the pip-audit job's input transform: pyproject.toml + # extraction + git+ stripping. scan_packages.py downloads + # PyPI archives without building, so it tolerates legacy + # setup.py packages (no resolver dry-run); but `--with-deps` + # delegates resolution to a single `pip download` call that + # cannot satisfy `git+` specs without git operations, so we + # strip them here too. run: | - python <<'PY' > unsloth-deps.txt + mkdir -p audit-reqs + python <<'PY' > audit-reqs/unsloth-deps.txt import tomllib with open("pyproject.toml", "rb") as f: d = tomllib.load(f) core = d["project"]["dependencies"] extras = d["project"]["optional-dependencies"]["huggingfacenotorch"] + print("# Auto-generated from pyproject.toml by security-audit.yml.") + print("# core deps + huggingfacenotorch extras.") for spec in core + extras: print(spec) PY + for f in studio.txt extras.txt extras-no-deps.txt \ + no-torch-runtime.txt overrides.txt triton-kernels.txt; do + python < "audit-reqs/$f" + src = "studio/backend/requirements/$f" + with open(src) as fh: + for line in fh: + stripped = line.strip() + before_comment = stripped.split("#", 1)[0] + if "git+" in before_comment: + print(f"# [security-audit] skipped git+ spec: {stripped}") + continue + print(line.rstrip("\n")) + PY + done - name: Sanity-check scan_packages.py # The scanner lives at scripts/scan_packages.py in this repo @@ -228,26 +300,46 @@ jobs: # *direct* dep -- and supply-chain attacks usually land # several hops down (litellm 1.82.7 was a dep of a dep for # most users). + # + # We invoke per-file rather than passing every -r at once. + # `--with-deps` collapses all -r files into a single + # `pip download` call internally; if any file's resolver run + # fails, the whole batch returns 0 archives. Per-file keeps a + # bad file from blanking the entire scan. continue-on-error: true run: | set +e - python scripts/scan_packages.py \ - --with-deps \ - -r unsloth-deps.txt \ - -r studio/backend/requirements/studio.txt \ - -r studio/backend/requirements/extras.txt \ - -r studio/backend/requirements/extras-no-deps.txt \ - -r studio/backend/requirements/no-torch-runtime.txt \ - -r studio/backend/requirements/overrides.txt \ - -r studio/backend/requirements/triton-kernels.txt \ - 2>&1 | tee logs-scan-packages.txt + : > logs-scan-packages.txt + for f in unsloth-deps studio extras extras-no-deps \ + no-torch-runtime overrides triton-kernels; do + # Skip files whose only content is comments / blanks + # (e.g. triton-kernels.txt after git+ stripping). The + # scanner exits 2 + prints help on an empty input, + # which would just spam the log. + if ! grep -qE '^[^#[:space:]]' "audit-reqs/$f.txt"; then + echo "::group::scan_packages skipped: audit-reqs/$f.txt (empty after filter)" + echo "[security-audit] $f.txt has no PyPI specs after git+ filter, skipping" \ + | tee -a logs-scan-packages.txt + echo "::endgroup::" + continue + fi + echo "::group::scan_packages.py -r audit-reqs/$f.txt --with-deps" + { + echo + echo "=== $f ===" + python scripts/scan_packages.py --with-deps -r "audit-reqs/$f.txt" + echo "=== end $f (rc=$?) ===" + } 2>&1 | tee -a logs-scan-packages.txt + echo "::endgroup::" + done { echo "## scan_packages (pre-install, transitive)" echo echo '### Coverage' echo '- unsloth core + `huggingfacenotorch` extras (pyproject.toml)' echo '- studio/backend/requirements/{studio,extras,extras-no-deps,no-torch-runtime,overrides,triton-kernels}.txt' - echo '- transitive closure via `--with-deps`' + echo '- transitive closure via `--with-deps`, scanned per-file' + echo '- `git+` specs are stripped (out of scope: we scan PyPI archives)' echo echo '### Findings (tail)' echo '```' @@ -261,7 +353,7 @@ jobs: name: scan-packages-log path: | logs-scan-packages.txt - unsloth-deps.txt + audit-reqs/ retention-days: 30 # ─────────────────────────────────────────────────────────────────────