Do not overstate the python host allowlist, and keep the blocked-command fallback tool-neutral

Two accuracy fixes to the sandbox tool notes:

- The sandbox keeps networking on (no CLONE_NEWNET) and the AST host check only
  inspects literal URL/host arguments, so a dynamically built request to a
  private host (host = '10.0.0.5'; requests.get('http://' + host + '/x')) runs
  with no network_calls violation. The note claimed the python tool can fetch
  only from a fixed allowlist and not arbitrary addresses, advertising a boundary
  the code does not enforce. Reframe it as intent (the python tool is intended to
  fetch from public sources such as github.com, huggingface.co, and pypi.org
  rather than private hosts) without the false only/arbitrary absolute.

- The blocked-network-command message told the model to fetch from Python code
  even when only the terminal is enabled that turn (python absent from the
  schema), which can induce an invalid tool call. _bash_exec has no per-turn
  tool signal, so make the fallback tool-neutral and gated: if a code-execution
  tool is enabled this turn, public files can be fetched from code instead.

Update the note tests to lock in both changes; the bypass note variant already
omits the network sentence and stays consistent.
This commit is contained in:
danielhanchen 2026-07-19 15:40:02 +00:00
commit fda0c11682
2 changed files with 34 additions and 24 deletions

View file

@ -2968,11 +2968,10 @@ _SANDBOX_PATHS_NOTE_INTRO = (
"host are already here."
)
_SANDBOX_PATHS_NOTE_NETWORK = (
" Internet access is limited: the python tool can fetch "
"only from a fixed allowlist of public sites (such as github.com, "
"huggingface.co, and pypi.org), not the user's own machines, private hosts, "
"or arbitrary addresses, and the terminal blocks direct download commands "
"like curl and wget."
" Internet access is limited: the python tool is intended to fetch "
"from public sources such as github.com, huggingface.co, and pypi.org "
"rather than the user's own machines or private hosts, and the terminal "
"blocks direct download commands like curl and wget."
)
_SANDBOX_PATHS_NOTE_TAIL = (
" Documents the user attaches to the chat are retrieved "
@ -2985,7 +2984,8 @@ _SANDBOX_PATHS_NOTE_TAIL = (
"you need are not here, ask the user to provide them or an exact path "
"instead of guessing one."
)
# Default (sandboxed) note: keeps the precise allowlist + curl/wget restriction.
# Default (sandboxed) note: steers the python tool at public sources and keeps
# the curl/wget restriction (without overstating the host check as a hard wall).
_SANDBOX_PATHS_NOTE = (
_SANDBOX_PATHS_NOTE_INTRO + _SANDBOX_PATHS_NOTE_NETWORK + _SANDBOX_PATHS_NOTE_TAIL
)
@ -5914,11 +5914,12 @@ def _bash_exec(
if blocked & _NETWORK_BLOCKED_COMMANDS:
return (
base + " These download commands are blocked by name in "
"the terminal. Public files on allowlisted sites (such as "
"github.com, huggingface.co, or pypi.org) can be fetched "
"from Python code instead. If you need files from another "
"location, ask the user to place them in the working "
"directory or give a path the sandbox can read."
"the terminal. If a code-execution tool is enabled this "
"turn, public files on sites such as github.com, "
"huggingface.co, or pypi.org can be fetched from code "
"instead. If you need files from another location, ask the "
"user to place them in the working directory or give a path "
"the sandbox can read."
)
return base
elif not _harden_parent_against_proc_env_leak():

View file

@ -5,9 +5,10 @@
tool descriptions (#7242).
The note must not misdirect the model with claims that are false for the real
sandbox: (1) the sandbox is not network-isolated -- the python tool can reach an
allowlist of public hosts (github.com / huggingface.co / pypi.org), so it must
not claim it "cannot reach ... remote hosts" at all; (2) a project sandbox is a
sandbox: (1) the sandbox is not network-isolated -- the python tool should fetch
from public hosts (github.com / huggingface.co / pypi.org), so it must not claim
it "cannot reach ... remote hosts" at all, nor overstate the host check as a hard
wall that makes private/arbitrary hosts impossible; (2) a project sandbox is a
shared per-project directory, so a new thread can inherit files from prior
threads -- the note must not claim the workdir "starts empty" or "persists only
for this conversation".
@ -36,13 +37,17 @@ from core.inference.tools import (
def test_note_does_not_claim_full_network_isolation():
lowered = _SANDBOX_PATHS_NOTE.lower()
# The old absolute claim is false for the python tool (allowlisted egress).
# The old absolute claim is false for the python tool (it does reach egress).
assert "cannot reach other machines or remote hosts" not in lowered
# It should instead scope the block to private/arbitrary hosts and name the
# allowlist so the model still fetches valid public URLs.
assert "allowlist" in lowered
# It must not overstate the host check as an enforced hard boundary:
# _sandbox_preexec leaves networking on and the AST check only inspects
# literal hosts, so a dynamically built request to a private host still runs.
assert "only from a fixed allowlist" not in lowered
assert "arbitrary addresses" not in lowered
# It should still steer to public sources so the model fetches valid URLs.
assert "public sources" in lowered
assert "github.com" in lowered and "huggingface.co" in lowered
assert "arbitrary" in lowered or "private" in lowered
assert "private" in lowered
def test_note_does_not_claim_project_sandbox_starts_empty():
@ -78,7 +83,7 @@ def test_note_scopes_network_block_to_the_terminal_not_all_shell_commands():
lowered = _SANDBOX_PATHS_NOTE.lower()
assert "shell network commands are blocked" not in lowered
assert "curl" in lowered and "wget" in lowered
assert "python tool can fetch" in lowered
assert "python tool is intended to fetch" in lowered
def test_note_distinguishes_attachments_from_sandbox_uploads():
@ -91,13 +96,17 @@ def test_note_distinguishes_attachments_from_sandbox_uploads():
assert "attach" in lowered
def test_blocked_network_command_message_points_to_python_egress():
# A blocked curl/wget must not tell the model the whole sandbox is offline: the
# python tool can still fetch allowlisted public hosts.
def test_blocked_network_command_message_gates_code_fallback_on_tool_availability():
# A blocked curl/wget must not tell the model the whole sandbox is offline, and
# must not name a specific tool (e.g. python) as the remedy: when only the
# terminal is enabled the python tool is absent from the schema, so an
# unconditional "fetch it from Python code" instruction invites an invalid tool
# call. The fallback is gated on a code-execution tool being enabled this turn.
msg = _bash_exec("curl https://raw.githubusercontent.com/foo/bar/main/x.py").lower()
assert "blocked command" in msg
assert "cannot reach other machines or remote hosts" not in msg
assert "python" in msg
assert "from python code instead" not in msg
assert "if a code-execution tool is enabled this turn" in msg
assert "github.com" in msg