* fetch bare hostnames as https instead of refusing them
* normalize host:port URLs and route schemeless github repos to the readme API
* only rewrite dotted host:port URLs with in-range ports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* reject relative paths and oversized ports in url normalization
* Match web-fetch ports as ASCII digits so a unicode digit cannot raise
str.isdigit() is True for digit-class characters int() refuses (superscript
two, circled digit one), so _normalize_url_scheme reached int(port) and raised
ValueError out of _fetch_url_raw, which runs before its try block. A
web_search url of "example.com:<superscript two>" surfaced a generic tool
exception instead of the Blocked: message it returned before this branch.
Match the port against an anchored [0-9]{1,5} instead; the five-digit cap that
kept the range check from converting an unbounded integer is now in the
pattern.
* Apply the invalid-port guard to redirect targets too
_fetch_url_raw wraps the initial parsed.port in try/except ValueError, but the
redirect hop reads rp.port unguarded, so a server answering
Location: https://example.org:99999/next fell through to the broad handler as
"Failed to fetch URL: Port out of range 0-65535" rather than a deliberate
block. No request is dispatched either way; this just makes the two paths
report the same way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the redirect-port test compact
The formatter expands a signature carrying a spaced kwarg default, which put
the stub opener on eleven lines. **kw absorbs the timeout the fetch loop
passes and leaves the whole stub on four.
* Never let a malformed URL escape _fetch_url_raw as an exception
The URL is model-supplied, so every bad form should come back as one of the
documented (error, body, content_type) strings. Three gaps remained:
urlparse itself raises on an unmatched IPv6 bracket and on a netloc that
NFKC-decomposes into a delimiter (//exam(fullwidth-solidus)ple.com), and both
calls sat outside a guard. getaddrinfo raises UnicodeError, which is a
ValueError and not the OSError _validate_and_resolve_host catches, when IDNA
encoding rejects a hostname.
Over a 3158 URL corpus that injects tabs, newlines, C0 controls, delimiters and
NFKC confusables at every position, main raises 42 times and this raises none.
Also strip surrounding whitespace in _normalize_url_scheme. _web_search already
stripped, but normalization moved down to the fetch layer, so a direct
_fetch_page_text caller did not get it.
* Name the host in the status badge and tool card for bare URLs
status_for_tool and the web-search tool card both required an explicit scheme
before reading the hostname, so every URL this branch newly makes fetchable
showed the generic "Reading page..." and "Read page" instead of the host.
Under permission_mode=ask that means the approval card named no destination for
exactly the inputs the branch enables.
The backend reuses _normalize_url_scheme. The frontend cannot, since new URL()
throws on a bare host, so RE_BARE_HOST mirrors the same grammar: only a dotted
host with an optional in-range port gets the https prefix, leaving /login,
javascript: and userinfo forms to render generically as before.
Also mention bare hostnames in the url parameter description, since they are
part of the accepted interface now.
* Do not let a malformed URL in the status badge kill the tool turn
status_for_tool runs inside prepare_call, before the fetch and outside the
handler that wraps tool execution, so a ValueError from urlparse ends the whole
turn instead of letting _fetch_url_raw return its blocked message.
_normalize_url_scheme catches its own parse error and hands back the original
string, so the parse here still has to be guarded.
Reachable with https://[::1 or a host that NFKC-decomposes into a delimiter.
This predates the branch, main raises identically, but the badge is one of the
lines this branch touches and the rest of it already promises no malformed URL
escapes as an exception.
* Tighten the comments added by this branch
* Revert the web_search url description change
The premise of this branch is that models already emit bare hostnames
unprompted, which is why the fetch layer had to stop refusing them. Advertising
the bare form in the tool schema does not enable anything, it just steers models
toward it, and that is the form carrying every edge case: ambiguous with dotted
custom schemes, and unlike an explicit scheme it does not cover IPv6 literals,
IDN or trailing-dot FQDNs.
The fetch layer tolerates bare hosts. The schema should keep recommending a
full URL. This also drops the one change here with no regression test.
* Match the backend port rule in the tool card host
The card's bare-host pattern required at least one digit after the colon, but
the backend fetches an empty port (example.com: and example.com:/path go to the
default HTTPS port), so a successful fetch rendered as "Read page" with no
host.
Allowing an empty port alone would have swung it the other way: example.com:0
is refused by the backend but new URL() accepts it, so the card would have named
a host that is never fetched. That mismatch was there before this change too.
Mirror the backend rule instead, an empty port or one in 1-65535, checked
against every case in the normalizer's own matrix.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
1915ca98db