ab2717afe0* Studio: persistent per-user trust_remote_code approval cache The consent gate pins each approval to a content fingerprint (sha256 over every repo .py), but nothing was persisted, so the dialog reappeared on every fresh load of the same unchanged repo. This adds an on-disk, per-user approval cache that lets the gate skip the dialog when the same user reloads the same code, while keeping the safety guarantees intact. Two-tier validation, both must hold or the user is re-prompted: - Commit SHA (cheap, one HfApi.model_info().sha, no download): a match means a byte-identical tree to the approved revision, so the scan/download is skipped. - Content fingerprint (authoritative): used whenever the SHA is unavailable (local path / offline) and always recomputed on a SHA miss. A new or edited .py changes both the SHA and the fingerprint, so it is caught in every mode. Safety: - Keyed per subject; one user's approval never auto-runs code for another. - CRITICAL is never stored or honored (guarded on both write and read), so a hand-edited store cannot smuggle in an auto-approval. - The malware (HF unsafe-file) gate stays unconditional. - Fail-safe: a corrupt store, an unresolvable SHA, or any error degrades to "ask again", never to "auto-approve". UNSLOTH_TRC_APPROVAL_CACHE_DISABLE=1 turns the cache off entirely. New module utils/security/remote_code_approvals.py holds the store (studio_root()/security/remote_code_approvals.json, atomic write, 0600, RLock) plus the SHA resolvers. Recording happens at the single gate chokepoint when the caller supplies the matching fingerprint, so subject is just threaded through inference/training/export (orchestrators, routes, workers). The scan endpoint returns already_approved so the frontend can skip the dialog on a cache hit. Tests: new tests/test_trc_approval_cache.py covers cache miss, SHA-match skip, SHA-moved re-scan, new-file re-consent, CRITICAL never cached (write + forged read), disable flag, subject isolation, combined adapter+base key, corrupt store, and no-subject bypass. Full security suite: 101 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: make the approval cache skip only the prompt, never the scan Codex found that the SHA "no-scan" fast path could run untrusted code without re-consent. Removed it; the gate now always re-scans and the cache only seeds the authoritative fingerprint check, so it can skip the dialog but never the scan. - CRITICAL is hard-blocked on every load (the scan always runs), so a hand-edited store that downgrades a CRITICAL repo's severity can no longer auto-run it (P2: do not trust editable severity for SHA approvals). - The fingerprint covers external auto_map repos, so changed third-party code always re-prompts even when the primary commit SHA is unchanged; there is no longer a SHA path that bypasses the fingerprint (P1: external auto_map repos). - resolve_commit_sha is resolved fresh on every call (no memoization), so a repo whose default branch moves after approval re-prompts instead of reusing a stale cached SHA (P1: revalidate mutable Hub SHAs). The SHA is now only a conservative secondary gate: a fresh resolvable SHA must match the approved revision, else the seed is withheld; a None (local/offline) falls back to the fingerprint. - Approvals record the scanner ruleset version (SCAN_RULES_VERSION); the gate ignores approvals from an older ruleset so reclassified bytes are re-scanned and re-shown instead of silently auto-approved (P2: invalidate on scan-policy change). Tests: test_trc_approval_cache.py rewritten around the prompt-skip semantics (unchanged repo still scans; SHA move / changed code / scanner-version bump / disable flag all re-prompt; forged downgraded severity still blocks CRITICAL). 105 passed with test_consent_gate.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments to be more succinct * Keep run-owner subject out of persisted config; serialize approval writes Threading subject (the run owner's username / API-key id) into the training config meant _sanitize_db_config persisted it into config_json, which training-history GET returns to any authenticated user, leaking who started a run in multi-user installs. Filter subject alongside the token fields; the worker still receives it from the live config. The approval store's RLock only guards one process, but approvals are recorded from separate inference/export/training subprocesses, so concurrent writers could clobber each other on os.replace and drop an approval (re-prompt). Hold a best-effort cross-process file lock around the read-modify-write. * Fail safe on a malformed approval store A store with the right version but a non-dict shape (e.g. a hand-edited "subjects": []) passed _load()'s check, then lookup chained .get() on a list and raised, breaking every remote-code load until the file was removed. Validate that subjects is a dict in _load(), and tolerate a non-dict per-subject entry in lookup/record/forget, so a corrupt store fails safe (re-prompt) instead. * Keep subject out of the MLX W&B run config _run_mlx_training uploads the whole training config to W&B minus a sensitive set that only listed hf_token/wandb_token/s3_config, so the authenticated subject (username / API-key id) was sent to W&B as run config even though DB history already strips it. Add subject to the W&B-sensitive filter, mirroring training._sanitize_db_config. * Tighten the W&B subject-filter comment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
111 lines
3.6 KiB
Python
111 lines
3.6 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Security helpers for the ``trust_remote_code`` boundary.
|
|
|
|
Two orthogonal questions: ``trusted_org.is_trusted_org_repo`` (may we AUTO-enable
|
|
remote code for this name?) and ``remote_code_scan`` (WHAT would run if the user
|
|
opts in?). The load paths try ``trust_remote_code=False`` first and, on the
|
|
transformers "requires trust_remote_code" error, scan the repo's ``auto_map``,
|
|
surface findings + a pinning fingerprint, and require explicit consent before
|
|
retrying with it enabled. Detection (is-vision / version / size) reads raw
|
|
``config.json`` and never enters this flow.
|
|
"""
|
|
|
|
from utils.security.consent import ( # noqa: F401
|
|
RemoteCodeDecision,
|
|
evaluate_remote_code_consent,
|
|
evaluate_remote_code_consent_for_targets,
|
|
)
|
|
from utils.security.file_security import ( # noqa: F401
|
|
FileSecurityDecision,
|
|
evaluate_file_security,
|
|
security_load_subdirs,
|
|
)
|
|
from utils.security.remote_code_scan import ( # noqa: F401
|
|
CRITICAL,
|
|
HIGH,
|
|
MEDIUM,
|
|
Finding,
|
|
RemoteCodeUnscannable,
|
|
ScanResult,
|
|
remote_code_fingerprint,
|
|
repo_remote_code_files,
|
|
scan_remote_code_files,
|
|
)
|
|
from utils.security.trusted_org import is_trusted_org_repo # noqa: F401
|
|
|
|
__all__ = [
|
|
"is_trusted_org_repo",
|
|
"scan_remote_code_files",
|
|
"repo_remote_code_files",
|
|
"RemoteCodeUnscannable",
|
|
"remote_code_fingerprint",
|
|
"should_block_remote_code",
|
|
"evaluate_remote_code_consent",
|
|
"evaluate_remote_code_consent_for_targets",
|
|
"preflight_remote_code_consent",
|
|
"preflight_remote_code_consent_for_targets",
|
|
"evaluate_file_security",
|
|
"security_load_subdirs",
|
|
"FileSecurityDecision",
|
|
"RemoteCodeDecision",
|
|
"ScanResult",
|
|
"Finding",
|
|
"CRITICAL",
|
|
"HIGH",
|
|
"MEDIUM",
|
|
]
|
|
|
|
|
|
def preflight_remote_code_consent(
|
|
model_name: str,
|
|
hf_token = None,
|
|
*,
|
|
trust_remote_code: bool = True,
|
|
approved_fingerprint = None,
|
|
trusted_org = None,
|
|
subject = None,
|
|
) -> "RemoteCodeDecision":
|
|
"""Scan a model's ``auto_map`` for the consent dialog. Thin wrapper over
|
|
``evaluate_remote_code_consent`` defaulting ``trust_remote_code=True`` so the scan
|
|
runs whenever the repo declares custom code; the start routes pass the user's real
|
|
value + approved fingerprint to enforce consent before any state mutation.
|
|
"""
|
|
return evaluate_remote_code_consent(
|
|
model_name,
|
|
hf_token,
|
|
trust_remote_code = trust_remote_code,
|
|
approved_fingerprint = approved_fingerprint,
|
|
trusted_org = trusted_org,
|
|
subject = subject,
|
|
)
|
|
|
|
|
|
def preflight_remote_code_consent_for_targets(
|
|
targets,
|
|
hf_token = None,
|
|
*,
|
|
trust_remote_code: bool = True,
|
|
approved_fingerprint = None,
|
|
subject = None,
|
|
) -> "RemoteCodeDecision":
|
|
"""Preflight consent over multiple repos (a LoRA adapter plus its base) scanned as
|
|
one combined unit with a single pinning fingerprint. Wrapper defaulting
|
|
``trust_remote_code=True``; the load passes the user's real value + fingerprint.
|
|
"""
|
|
return evaluate_remote_code_consent_for_targets(
|
|
targets,
|
|
hf_token,
|
|
trust_remote_code = trust_remote_code,
|
|
approved_fingerprint = approved_fingerprint,
|
|
subject = subject,
|
|
)
|
|
|
|
|
|
def should_block_remote_code(result: "ScanResult") -> bool:
|
|
"""Recommend blocking by default on CRITICAL/HIGH findings. Advisory only: the
|
|
caller still surfaces findings and takes explicit consent.
|
|
"""
|
|
sev = result.max_severity
|
|
return sev in (CRITICAL, HIGH)
|