Last security-audit run revealed 4 step-level errors hidden by
continue-on-error (the job reported pass but each fix is real):
1. OSV-Scanner curl 404 -> tar exit 2. v2.x ships a raw binary
(`osv-scanner_linux_amd64`), not a tarball. Drop tar -xzf,
curl -o the binary directly + chmod +x.
2. cargo audit `parse error: TOML parse error at line 5 col 8`
on RUSTSEC-2026-0073.md. cargo-audit 0.21 doesn't parse the
CVSS 4.0 schema used in 2026 advisories. Bump pin to ^0.22.
3. TruffleHog `flag 'no-update' cannot be repeated`. The
trufflesecurity/trufflehog action passes --no-update
internally already; remove our duplicate from extra_args.
4. cyclonedx-py `unrecognized arguments: --schema-version 1.6
--outfile ...`. cyclonedx-bom 4.x renamed to `--sv` for spec
version and `-o` for the output file.
Plus pin every remaining mutable-ref action to a 40-char SHA. The
new GHA pinning verifier flagged 4 third-party + 40 first-party
mutable refs; this commit pins all 44 to the latest SHA *within
the existing major version* (no auto-upgrades). Mappings:
actions/checkout @v4 -> 34e114876b... (v4.3.1)
actions/setup-node @v4 -> 49933ea528... (v4.4.0)
actions/setup-python @v5 -> a26af69be9... (v5.6.0)
actions/stale @v10 -> b5d41d4e1d... (v10.2.0)
actions/upload-artifact @v4 -> ea165f8d65... (v4.6.2)
actions/cache @v4 -> 0057852bfa... (v4.3.0)
swatinem/rust-cache @v2 -> 23869a5bd6... (v2.9.1)
dtolnay/rust-toolchain @stable-> 29eef336d9... (stable @ 2026-05-07)
44 pins applied across 11 workflow files. The pin verifier now
reports zero unpinned `uses:`. Dependabot's github-actions
ecosystem (already configured in .github/dependabot.yml) will
auto-bump these SHAs in weekly batches.
This closes the same attack class that hit litellm 1.82.7: an
attacker who hijacks a tag (as in the aquasecurity/trivy-action
March 2026 incident) cannot redirect our workflows because we no
longer follow tag refs.
Trivy was the initial entry point for the litellm 1.82.7/8 supply-
chain compromise (March 2026):
Late Feb: attacker exploited a misconfigured pull_request_target in
Trivy's CI -> stole the aqua-bot PAT.
Mar 19: attacker force-rewrote 76 of 77 tags in
aquasecurity/trivy-action (and all 7 in setup-trivy) to
point at malicious commits. Anyone using a tag ref
(`@v0`, `@v0.69.4`, `@latest`) auto-pulled the trojan.
Mar 24: litellm's CI ran the trojaned Trivy unpinned -> the
payload exfiltrated PYPI_PUBLISH from the runner ->
attackers published the malicious litellm wheels.
A security scanner has the same broad runtime read access as
deployment tooling -- by design. That's exactly what made it the
ideal pivot. Our prior `aquasecurity/trivy-action@v0.36.0` was a tag
ref, the same shape that hit litellm, and Aqua's remediation does
not eliminate the meta-attack class (next compromise restarts the
clock). Removing rather than re-pinning.
Coverage we lose, and how we backfill:
- cross-ecosystem CVE: already covered by OSV-Scanner (NVD + GHSA
+ GitLab + RustSec feeds).
- secret detection: already covered by TruffleHog + the new
GitHub Actions pinning verifier.
- OS package CVEs: not relevant for a Python package + Tauri
desktop app.
- IaC misconfig (Dockerfile / k8s / Tauri config): the one unique
Trivy value-add. Unfilled for now; revisit with checkov / kics
if/when we ship a Dockerfile or k8s manifests.
Also pinned the two remaining third-party actions to commit SHAs
(was a tag ref, the exact thing the GHA pinning verifier flagged):
- step-security/harden-runner: a5ad31d (= v2.19.1)
- trufflesecurity/trufflehog: 17456f8 (= v3.95.2)
Dependabot's github-actions ecosystem will auto-bump these SHAs.
Refs: https://docs.litellm.ai/blog/security-update-march-2026https://www.microsoft.com/en-us/security/blog/2026/03/24/detecting-investigating-defending-against-trivy-supply-chain-compromise/
Goes after the residual gaps from the supply-chain incident audit.
Each addition targets a real attack class that prior layers couldn't
catch:
1. step-security/harden-runner (audit mode) on every job. eBPF
egress firewall on the runner -- if scan_packages misses a
payload, harden-runner's audit log records every host the
malicious archive dialed. Audit mode initially so we observe
the legitimate egress profile before promoting to block.
2. Trivy filesystem scan (vuln + misconfig + secret). Hits NVD +
GHSA + GitLab + Aqua Vuln DB and also catches Dockerfile / k8s /
Tauri / shell IaC misconfigs that pip-audit + OSV don't see.
3. TruffleHog secret-leak scan on PR diffs. --only-verified so we
only flag tokens the source provider confirmed are live; runs
base..head on PRs and full repo on push. Catches accidental API
key commits that the Lint CI's grep-based codespell check
cannot. checkout fetch-depth: 0 so the diff range exists.
4. CycloneDX SBOM generation as artifact. Per-requirements file
plus a project-level SBOM from pyproject.toml. Lets downstream
consumers audit our wheel contents (the ML supply-chain SBOM gap
is a known industry-wide problem; meets half of NTIA SBOM mins).
5. GitHub Actions pinning verifier. Reports every `uses: foo@v4`
or `@main` mutable ref. tj-actions/changed-files (Mar 2025) hit
anyone using non-SHA pins. Currently surfaces 4 third-party
unpinned refs (dtolnay/rust-toolchain, swatinem/rust-cache) and
40 first-party (`actions/*`); informational baseline, tighten
once we're ready. Dependabot's github-actions ecosystem
auto-bumps SHA pins, so the maintenance cost is zero.
6. Hash-pin verifier. Reports how many == specs would gain from
`--hash=sha256:` entries. Currently 11 == pins, 0 with hash.
Roadmap step: `uv pip compile --generate-hashes` then
`pip install --require-hashes`. Hash-locked installs would have
refused a republished litellm 1.82.7 even at the same version
string.
7. Custom Semgrep rules at .semgrep/unsloth-rules.yml. Seven rules
for the *specific shape* of recent ML-stack CVEs we'd otherwise
re-introduce ourselves: langchain-core deserialize-roundtrip
(CVE-2025-68664), n8n private-pyodide-eval (CVE-2025-68668),
marimo websocket-no-auth (CVE-2026-39987), litellm
popen-with-network-stdin, Shai-Hulud workflow-write,
pickle-from-network, shell=True with f-string interpolation.
dependabot.yml: extend to pip + cargo ecosystems so security
advisories on Python deps and the Tauri shell auto-generate update
PRs alongside the github-actions / bun / npm ones.
All new steps continue-on-error initially; findings land in
GITHUB_STEP_SUMMARY plus the advisory-audit-logs artifact.
Recent supply-chain incidents that scan_packages would have missed:
- PyTorch Lightning 2.6.x: payload in _runtime/router_runtime.js
(14.8 MB), persistence via .claude/settings.json SessionStart
and .vscode/tasks.json folderOpen
- npm chalk/debug + Shai-Hulud: hex-var obfuscation, window.ethereum
Web3 hijack, .github/workflows/shai-hulud.yml repo takeover,
trufflehog credential exfil
- elementary-data 0.23.3: token harvesters with embedded gh{p,o,s}_
and AKIA regexes
- litellm 1.82.7: also covered by existing patterns, but anyone on
`>=` got it during the 40-min exposure window
- langchain-core CVE-2025-68664 / n8n CVE-2025-68668 / marimo
CVE-2026-39987: first-party design flaws, not malicious-author
scan_packages.py:
- Six new regexes: RE_DEV_TOOL_HIJACK, RE_TOKEN_REGEX,
RE_JS_OBFUSCATION, RE_WEB3_HIJACK, RE_WORKFLOW_INJECT,
RE_SHELL_DROPPER.
- Three new checkers: check_js_file, check_shell_file,
check_workflow_file. scan_archive now routes .js/.mjs/.cjs/.ts
to the JS checker, .sh/.bash to the shell checker, and
.github/workflows/*.yml to the workflow checker.
- JS checker fires CRITICAL on hex-var obfuscation OR Web3 hijack
OR (token regex + network) OR workflow-injection signature; HIGH
on a >100 KB JS bundle inside a Python wheel (the Lightning tell).
- Smoke-tested: every new pattern matches its canonical positive
and rejects four legitimate-looking false-positive baits.
security-audit.yml:
- OSV-Scanner step: cross-ecosystem advisory check (PyPI + npm
+ cargo) from one binary. OSV's feed is a superset of GitHub-
Advisory; catches CVEs that haven't propagated yet (e.g.
langchain-core was on OSV before GitHub Advisory).
- Semgrep step: p/supply-chain + p/python + p/javascript +
p/security-audit packs catch first-party logic bugs (CVEs 7/9/10
above) that pattern scanning never sees.
- Lockfile pin verifier: warns on every non-`==` spec in
requirements/*.txt. Currently surfaces 104 unpinned specs as
informational baseline; tighten to blocking once the baseline
is curated.
All new steps continue-on-error initially; they surface findings to
the workflow summary + advisory-audit-logs artifact.
Three advisory-DB lookups previously spun up three separate runners.
All three are fast lockfile-driven checks (pip-audit ~1m37s, npm audit
~12s, cargo audit ~24s) and the runner-setup overhead dominates each.
Run them sequentially on a single runner with python + node + rust
toolchains pre-installed; total wall clock comes out roughly the same
(~3 min) but with one PR check instead of three.
Each step keeps continue-on-error: true so a finding in one toolchain
does not suppress the others. Logs land in a single advisory-audit-logs
artifact (pip + npm + cargo + the filtered req set).
Heavy job stays separate: pip-scan-packages remains the 3-shard matrix
that downloads + pattern-scans the full PyPI transitive closure (~6
min/shard, in parallel). Conflating that into the advisory job would
bloat the runner image and serialize a 6 min job behind a 30 s one.
Previous run took ~10+ minutes because each requirements file ran
its own --with-deps resolve serially, and the six files all share
~70% of their transitive set (transformers, peft, accelerate land
in three of them). Net effect: the same 200+ archives downloaded and
pattern-scanned three times in series.
Two changes:
1. Within a shard, feed every -r file to ONE scan_packages call so
pip's resolver intersects version constraints once and yields
a single deduped transitive set.
2. Across shards, run three matrix jobs in parallel:
- hf-stack: unsloth-deps + no-torch-runtime (pyproject extras)
- studio: studio + overrides + extras-no-deps
- extras: extras (heavy openai-whisper / scikit-learn stack)
Wall clock now bounded by the slowest shard rather than the
sum, dropping ~10 min to ~3-5 min.
Each shard uploads its own artifact (scan-packages-log-<id>) so log
correlation stays clean. fail-fast: false so one shard's findings
don't suppress the others.
Last push surfaced two silent failures:
1. pip-audit aborted on openai-whisper. The package's setup.py
imports pkg_resources, which the isolated build env's modern
setuptools no longer ships by default. Because we passed every
-r file in one invocation, that single build failure killed the
audit for ALL files (the run reported success only because
continue-on-error swallowed exit 1).
2. scan_packages --with-deps aborted on the first git+ spec it
hit (triton-kernels.txt's git+https://github.com/triton-lang
/triton.git, plus OpenEnv in extras-no-deps.txt). Same
all-or-nothing behaviour: the entire transitive scan reported
"0 archives downloaded" and "all clean" -- meaning we silently
scanned nothing.
Fixes:
- Build a filtered audit-reqs/ tree first. Each Studio requirements
file is copied with `git+` lines stripped (replaced with a
`# [security-audit] skipped` marker so the exclusion is auditable
in the artifact). Pure git refs are out of scope for both pip-
audit (CVE DB only knows PyPI versions) and scan_packages (it
inspects PyPI archives, not git HEADs).
- Run pip-audit per-file in a loop. One bad file no longer takes
out the whole audit.
- Pin setuptools<78 + wheel into pip's isolated build env via
PIP_CONSTRAINT, so legacy setup.py packages (openai-whisper) can
still emit metadata for the resolver.
- Run scan_packages per-file too, with the same git+ filter and a
skip for files that are empty after filtering (triton-kernels.txt
becomes a comments-only file and would otherwise spam the log
with `--help`).
Net effect: pip-audit now actually emits CVE findings (we know the
default branch carries 17), and scan_packages downloads + pattern-
scans the full transitive closure of every PyPI-only requirements
file plus unsloth's pyproject deps.
The previous Security audit only covered Studio's backend requirements.
The unsloth pip package itself ships its own dep set via pyproject.toml
(typer/pydantic/pyyaml/nest-asyncio core, plus the huggingfacenotorch
extras: transformers/peft/accelerate/trl/datasets/diffusers/etc.) -- a
malicious upload to any of those would slip past us today. Build a
combined dep list from pyproject.toml + the six Studio requirements
files and feed it to both pip-audit and scan_packages.
Add scan_packages.py at scripts/scan_packages.py so the scanner ships
with the repo and CI does not depend on a network fetch at job time.
Pass --with-deps to scan_packages so the pre-install pattern scan
walks the full transitive closure -- supply-chain attacks usually land
several hops down (litellm 1.82.7 was a dep of a dep for most users;
top-level-only scanning would have missed it).
No installation in either job. pip-audit's -r mode resolves through
PyPI metadata, scan_packages downloads sdist/wheel archives raw and
inspects them without running install hooks. An attacker who has
compromised a transitive dep cannot execute code in this workflow.
Three Priority-1 follow-ups from the lint review.
Lint CI gains two non-blocking gates that surface drift without
blocking merges (the same shape as the existing format-drift step):
- codespell: typo catcher across source / comments / docs. Skips
lockfiles, generated assets, binary artefacts, LICENSE files.
ignore-words-list pulls out short identifiers and PyTorch
idioms (parm/parms, ans, hist, etc.) the default dictionary
would flag. Local run finds 16 real typos to fix in a follow-up.
- shellcheck: catches subtle shell bugs `bash -n` doesn't see --
unquoted expansions, useless cat, `[[ ]]` command substitution,
etc. SC1090 + SC2034 muted because install/setup scripts
legitimately source runtime paths and use export-only
assignments. Critical-path coverage: install.sh, setup.sh,
tests/sh/.
Both pinned for reproducibility (codespell>=2.3,<3 in pip,
shellcheck via apt-get). Both surface findings in PR annotations
without failing the run; drop continue-on-error after the cleanup
PRs land.
New workflow: Security audit. Runs `pip-audit` against the same
dep set Studio's backend pytest matrix installs, so we audit what
the runtime actually loads (not what pyproject.toml's transitive
resolution might pull in differently). Triggers:
- PRs touching requirements / pyproject.toml,
- push to main / pip,
- nightly @ 04:13 UTC (off-the-hour to dodge cron rush),
- workflow_dispatch.
The default branch already carries 17 known vulnerabilities per
the dependabot banner, so a hard gate today would block every PR
on a baseline we have not triaged. Non-blocking; full table goes
to GITHUB_STEP_SUMMARY for grep-ability and a 30-day artefact for
historical comparison.
The custom AST anti-pattern scan I prototyped was dropped: every
class of CPU-import-time bug we hit in this PR (bitsandbytes,
torchvision, _cuda_getCurrentRawStream, DEVICE_COUNT==0 stream
init) is already caught by the Repo tests (CPU) job exercising
the actual import on a CPU torch wheel. Restating the rule
in AST form would only add noise.
