73af334d11* Studio: stream live tool output with SSE heartbeats and fix web page extraction Server-side python/terminal tools now stream incremental stdout to the chat UI while running (new tool_output SSE event), and every blocking tool execution emits heartbeat keepalives so reverse proxies (Cloudflare tunnels cap idle streams at ~100s) cannot drop the connection mid-turn. The tool loop routes also emit a stall keepalive during silent prompt prefill between tool iterations. The final role=tool message the model sees is byte-identical to before, so tool-call parsing, nudging, and healing are untouched. web_search page fetches now extract main content: GitHub repo root pages are rewritten to the README API (with HTML fallback), hidden/aria-hidden client error placeholders are dropped, conversion scopes to article/main, and known boilerplate fragments are stripped. Non-HTML responses are returned raw instead of being run through the HTML converter. The frontend renders live-scrolling tool output inside running python and terminal cards, and a chat stream that ends without a terminal signal now surfaces an explicit interrupted state with a Retry action instead of silently ending the turn. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix content-type sniffing, unlimited-timeout drain, and env parity in tool streaming Content-Type sniffing: get_content_type() defaults to text/plain when the header is absent, so the sniffing fallback never fired and header-less HTML came back as raw markup. Report an empty type for a missing header and sniff the body whenever the declared type is not HTML, so mislabeled text/plain HTML pages are converted like before the extraction change. Unlimited timeout drain: with tool_call_timeout disabled the old path used communicate(timeout=None) and waited for EOF, but the streaming drain capped the post-exit drain at a 5 second join, truncating output from a grandchild that holds stdout open. When timeout is None, drain until EOF or the cancel event fires; finite timeouts keep the bounded remaining-budget join. Env parity: drop the PYTHONUNBUFFERED=1 injection on the streaming path so the child invocation is byte-identical with and without streaming (the env var was model-visible via os.getenv). Live streaming granularity now depends on the child flushing; unflushed output arrives in ~8 KB chunks or at exit and the final result is unchanged, with SSE heartbeats covering the gaps. * Studio: stream tool-call arguments while the model writes them A model writing a large tool call (a full python game is minutes of generation) produced nothing on the stream: the structured path accumulated delta.tool_calls fragments silently after the provisional card, and the text path's DRAINING state consumed everything until stream end. The user saw a dead Running spinner while the model was in fact writing code, and the byte-silent SSE segment was also the window where proxies drop the connection. New tool_args SSE events stream the arguments as they generate. The structured path forwards each fragment once a provisional card exists (backlog first, so the card starts from the top of the call). The text path sniffs the drained call for an enabled tool name and streams the raw call text under the id the stream-end parser assigns its first call (call_0), so the final tool_start reconciles the same card; the sniff is gated on enabled names plus the provisional size floor, and prose or ordinary JSON answers never spawn a card. The safetensors loop streams the drained render_html call to its existing provisional card the same way. The chat adapter accumulates the raw stream per card and feeds a partial JSON parse (call envelopes and stringified arguments unwrapped) into the part's args, so the python and terminal cards render the code live and the render_html canvas builds while streaming; both cards now say Writing code / Writing command during this phase via useToolArgsStatus. Display only: the parser input, the executed call, and the conversation the model sees are byte-identical, covered by new loop-level tests for the structured path, the text path, and the no-tool JSON answer. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep full tool output visible past the model cap; heal /mnt/data habits Live testing surfaced two issues in the tool streaming UX. First, a long python stdout ended in '... (truncated' in the finished card: the model-visible result is capped by tools._truncate (_MAX_OUTPUT_CHARS, previously 8000 chars) to protect the context window, and the card rendered that capped text even though the live stream had already shown everything. The cap stays (raised to 16000, overridable via UNSLOTH_TOOL_RESULT_MAX_CHARS) but display and model concerns are now split: the adapter preserves the accumulated live stream on tool_end whenever it captured more than the final result, and the finished python/terminal cards prefer it. The live-stream ceiling rises from 16 KB to 400 KB (chunks batch per poll, so SSE stays cheap), and both the live pane and the finished card render only the last 2000 lines with a Show all control so a huge output cannot jank the DOM. The truncation notice now tells the model the user saw the full output and that written files persist in the working directory. The final result string remains byte-identical with and without streaming. Second, models trained on ChatGPT code-interpreter transcripts write to /mnt/data, which does not exist here (the sandbox CWD is a per-thread persistent dir). Three layers, all identical across streaming and non-streaming paths: the python/terminal tool descriptions gain one sentence saying to use relative paths in the persistent CWD; a failed execution whose output shows a missing-file error on a known code-interpreter prefix (/mnt/data, /mnt/outputs, /home/sandbox, /workspace) gets a model-visible retry hint appended after truncation so it always survives; and a sitecustomize shim on the sandbox PYTHONPATH remaps those prefixes onto the CWD in open()/os.makedirs() with a one-line stderr notice, covering the python tool and any Python launched from the terminal tool without touching the exec wrapper (so tracebacks keep their line numbers). Bash-level file operations cannot be redirected without root or mount namespaces, so they rely on the description and the hint. * Studio: fix hidden-element parsing, heartbeat gaps, and tool output id collisions Review follow-ups on the tool streaming work: - _html_to_md: treat any present hidden attribute value as hidden (it is an enumerated attribute whose invalid value default is the Hidden state, so hidden="false" is still hidden), and implement HTML5 optional end tags so an unclosed <p hidden> or <li hidden> ends at the next sibling start tag instead of swallowing every following sibling until the parent closes - tool_stream_exec: keep heartbeats flowing after the live-output cap; a tool that keeps printing past the cap kept the queue non-empty, so neither tool_output nor heartbeat events were emitted and the SSE stream went silent past proxy idle timeouts - routes/inference: forward tool heartbeats before the disable_parallel_tool_use drop window swallows events, so a dropped call that executes server-side cannot leave the Anthropic stream silent - llama_cpp: close the provisional text tool card with a tool_end when the drained call fails to parse (DRAINING false-positive path), so the card cannot spin forever while the text is delivered as content - tools: decode terminal output as utf-8 with errors=replace like the python tool; invalid bytes used to raise UnicodeDecodeError from communicate() on the non-streaming path and silently truncate the streaming reader, so the two paths diverged - sitecustomize: patch io.open alongside builtins.open; pathlib Path.open, read_text and write_text call io.open directly and bypassed the remap - frontend: scope the toolLiveOutput/toolFullOutput store keys by pane (modelType and pairId) and clear stale entries on tool_start; backend ids like call_0 repeat across turns and across concurrently streaming panes (compare mode), so a later turn or another pane could display the wrong preserved output, and run-end cleanup now clears only its own keys Each backend fix carries a regression test that fails on the previous code; the byte-identity tests between streaming and non-streaming stay green. * Studio: keep tool failure status visible and truncation/remap notices truthful Finished python/terminal cards preferred the fuller live stream by length alone, so a tool that printed a lot then timed out or exited non-zero showed the captured stdout but dropped the final result's status (timeout notice, Exit code N). preferFullToolOutput now shows the stream when the result is just its truncated prefix, and appends the result otherwise so the failure tail always survives and the copy button copies both. The result truncation notice claimed the user was shown the full output, but the same wrapper serves non-streaming chat/API and direct execute_tool() callers where nothing is streamed to anyone. The notice is now mode-neutral and stays byte-identical with and without an output_callback, keeping the streaming vs non-streaming invariant intact. The sandbox sitecustomize shim now remaps /tmp/outputs into the working directory only while it does not already exist, so a real /tmp/outputs the user's own code created is never shadowed; /tmp/outputs also joins the missing-path retry-hint list. * Studio: suppress hidden void elements and keep live output scroll pinned only when at bottom * Studio: drop capped tool output without concatenating; remap pathlib mkdir Past the live-output cap stream_tool_execution built item + _drain_pending() (the current chunk joined with every queued sibling) only to discard it in the capped branch, so a chatty tool (yes, a tight print loop) could enqueue far more than one poll interval of text and blow past the memory/CPU ceiling the cap exists to enforce. Drain and drop queued items without building a combined string, still counting each drain toward the heartbeat cadence so the SSE keepalive survives. Generated code often prepares code-interpreter paths with Path('/mnt/data').mkdir(parents=True, exist_ok=True); pathlib drives that through os.mkdir (not the patched os.makedirs) per component and, on FileExistsError, probes the unpatched os.stat via Path.is_dir(), so the setup raised before open() ever ran. Patch os.mkdir with the same remap and patch Path.mkdir so the whole parents/exist_ok dance lands on the mapped working directory and stays idempotent; real paths still pass through. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: generalize sandbox write remap and hint to any hallucinated absolute path Models invent absolute paths from seeing their CWD (a DeepSeek run did open('/home/ubuntu/Sandbox/flappy_bird.html', 'w') and died with FileNotFoundError). A prefix list cannot enumerate these, so the sitecustomize shim gains a write-mode fallback in open()/io.open(): when a write/create-mode open targets an absolute path outside the CWD whose parent directory does not exist, redirect it to the basename in the CWD and emit the same one-line stderr notice, echoing the original path. The prefix remaps still run first (they cover reads and preserve subpaths); read modes never hit the fallback so real system files fail or succeed truthfully; bytes paths pass through. The fallback is not applied to mkdir/makedirs/Path.mkdir, since creating an arbitrary absolute directory can legitimately succeed on the host, and that decision is documented in a comment. The model-visible retry hint now echoes the real failing path (parsed from the traceback tail) instead of the canned /mnt/data example, and fires for any absolute path outside the working directory, not just the enumerated prefixes, while a relative miss still gets no hint. The shim wrapper still adds one frame to tracebacks that surface open() errors; suppressing only our frame has no clean standard mechanism (a wrapper always adds a frame), so the frame is left as an accepted compromise. Tests: hallucinated absolute write remaps to the CWD basename across w/a/x/w+; reads of a missing absolute path pass through untouched; writes to an existing external dir pass through; prefix subpaths still preserved; end-to-end write fallback lands the file in the sandbox workdir identically with and without streaming; the hint echoes the actual path for convention and non-convention absolute paths alike. * Studio: kill exited process groups on drain; bound the over-cap output batch _drain_process_output killed the process only via _kill_process_tree, which short-circuits once the parent has exited, so a grandchild that inherited stdout and outlived the parent was never signaled: a finite-timeout run could return while it kept holding the pipe, and a timeout=None cancel left it behind. Capture the setsid process group before waiting and SIGKILL that group at both give-up points so the whole tree is torn down. The streaming wrapper's first over-cap batch joined the current chunk with the entire pending backlog before enforcing the live-output cap, so a chatty tool could allocate far past the cap on the crossing batch. Bound the drain to the remaining budget and drop the surplus in place, keeping the truncated output byte-identical to joining everything. * Studio: harden sandbox path healing and process/generator cleanup Sandbox sitecustomize shim: - Make the generalized write fallback collision-safe: never redirect an invented absolute path onto an already-present CWD file (refuse and let the original open raise FileNotFoundError, preserving the workspace file). - Only w/a/x create a file; r+/rb+ are read-update modes that require the target to exist, so a bare + no longer trips the write fallback. - Gate every convention-prefix remap (/mnt/data, /mnt/outputs, /home/sandbox, /workspace) on the prefix root being absent, so a real host mount is never shadowed; a miss under an existing real prefix passes through. - Patch os.open so Path.touch and other low-level creators heal convention paths too, matching the Path.mkdir patch. Local code execution (tools.py): - Capture the setsid process group right after Popen (before any watcher can poll/reap the leader) and thread it through the cancel watcher and drain. - Kill the captured group in the non-streaming python/terminal timeout branch so an exited leader no longer leaks a stdout-holding grandchild (matches the streaming drain path). - Guard os.getpgid/os.killpg by platform so streamed execution no longer raises on Windows; fall back to single-pid kill. - Judge missing-path hints against the executor's real workdir so a legitimate miss inside a project workspace outside the sandbox root is not mislabeled. Tool streaming routes (routes/inference.py): - Drain a pending next(gen) worker before closing the generator in the safetensors and Anthropic tool streams, so a disconnect no longer races gen.close() (generator already executing) or leaks the thread/generator. HTML to markdown: - Only drop boilerplate lines composed entirely of known furniture phrases so real prose that merely quotes one (for example "we use cookies to authenticate requests") is preserved. Adds hermetic tests for each change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep aside callouts, contain sandbox path remaps, and keepalive dropped Anthropic tool events _html_to_md: stop dropping <aside> unconditionally. Documentation pages render notes/warnings/examples as aside admonition callouts; those inside the selected article/main scope are real content. A furniture aside outside the scope is already excluded by the main-content pass. sitecustomize: contain the code-interpreter path remap under the sandbox CWD. A hallucinated habit path such as /mnt/data/../other_session/file no longer escapes the per-conversation workdir; parent-traversal components in the suffix are dropped and a '.'/'..' write-fallback basename is refused. routes/inference: emit a rate-limited comment keepalive when the Anthropic Messages stream drops tool_output/tool_args events. A chatty tool keeps the generator busy so the stall keepalive never fires and the tool wrapper emits heartbeats only while idle, which left the SSE stream silent past proxy idle caps; the OpenAI passthrough paths forward these events, this path now keeps the connection alive. * Studio: bound the tool-output chunk that first crosses the live cap _drain_queue joined the entire chunk that first crossed the live-output cap before dropping the rest, so a single multi-megabyte line (or any chunk dequeued once the budget was already met at max_chars <= 0) was materialized in full only to be truncated away, defeating the memory ceiling the cap enforces. Slice the crossing chunk to one character past the budget: that preserves the caller's overflow signal and its byte-identical truncation while dropping the arbitrarily large remainder in place. * Studio: scope missing-path hint to the failing line, keepalive dropped-call output, and preserve truncated tool streams over byte length - tools._missing_path_hint: the code-interpreter convention-prefix trigger scanned the whole output, so a convention prefix mentioned only in a traceback frame (a /workspace project root) or printed by the user's code would add a misleading 'use a relative path' hint even when the actual FileNotFoundError was a relative or in-workdir path. Scope the convention test to the failing-path error line(s), matching _extract_missing_abs_path. - _anthropic_tool_stream: the tool_output/tool_args rate-limited keepalive sat after the drop_until_tool_end skip, so under disable_parallel_tool_use a chatty second-or-later tool call was dropped whole with no keepalive, letting an idle proxy kill the SSE stream. Check the keepalive branch before the drop skip (like the heartbeat branch) so dropped-call output keeps the stream alive. - preferFullToolOutput / chat-adapter: a truncated result can be longer than the live stream by byte count once its footer, an 'Exit code N:' notice, or an __IMAGES__ base64 tail is appended, so the length-only gate discarded the full stream and the finished card fell back to the truncated text. Add a shared truncation-aware shouldPreserveFullOutput used by both the write and read sites: preserve the stream whenever the result carries the truncation footer. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: skip the habit-path hint for real project paths under a convention prefix * Kill captured process group on streamed wait-timeout The streamed drain path's proc.wait() timeout branch only called _kill_process_tree(proc). If the leader exits in the narrow window between the wait timing out and _kill_process_tree sampling its pgid, that helper short-circuits on the reaped leader and a stdout-holding grandchild in the same group survives. Also kill the captured pgid there, matching the non-streaming communicate() timeout path. Adds a hermetic regression test that models the reaped-leader race by stubbing _kill_process_tree. * Fix 3.10 pathlib write_text remap and honor cancel in finite drain On Python < 3.11 pathlib routes Path.open / read_text / write_text through a module-level accessor singleton whose open attribute captured the original io.open at import time (_NormalAccessor.open = io.open). Patching io.open in the sandbox shim therefore never reached that captured reference, so a Path('/mnt/data/x').write_text(...) raised FileNotFoundError on 3.10 while passing on 3.11+ (which dropped the accessor and calls io.open at call time). Repoint _NormalAccessor.open at the same io.open wrapper via a staticmethod, guarded so it is an idempotent no-op on 3.11+. Keep the test save/restore helpers symmetric so the accessor is restored too, and add a hermetic write_text/read_text remap test that covers every version. Also honor cancellation while draining inherited stdout after the leader exits. Once the leader is reaped the cancel watcher returns (its loop is while proc.poll() is None), so the finite-timeout drain did one blocking reader.join(timeout=remaining) that ignored cancel_event and kept draining a chatty grandchild for the whole budget after a disconnect/Stop. Poll cancel_event in 0.5s slices against a deadline like the timeout=None branch and kill the captured process group promptly on cancel. The normal path still reaches EOF on its own, so the streamed vs non-streamed result is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: port no-tool stream keepalive/drain and fix subprocess/queue/extraction asymmetries Streaming no-tool paths now match their tool twins: - _anthropic_plain_stream, safetensors/MLX no-tool stream, and standard GGUF no-tool stream run next(gen) in a worker with a timed SSE keepalive loop so a long prompt prefill cannot leave the stream idle past a proxy cap. - The Anthropic plain and safetensors/MLX no-tool teardowns now drain the pending next(gen) worker and close the generator on disconnect instead of leaking the suspended generator. Other asymmetries: - Non-streaming _python_exec/_bash_exec always drain via _drain_process_output (output_callback may be None) so a cancelled run reaps a stdout-holding grandchild that outlived the leader instead of blocking in communicate(). The joined bytes are identical to communicate(), so streamed vs non-streamed results stay byte-identical. - _build_bypass_env installs the sitecustomize path shim on PYTHONPATH (prepend, keeping the operator's entries) so /mnt/data remap works in bypass mode too. - GGUF forwards output_callback to execute_tool only when the callable accepts it (shared accepts_output_callback), matching safetensors and preserving legacy monkey-patched signatures. - tool_stream_exec bounds accepted live output at the producer boundary so a chatty tool cannot grow the queue without limit under consumer backpressure and cannot keep the drain spinning and starve heartbeats. - html_to_md implicit-close now searches past unclosed inline descendants so a hidden <p>/<li> is closed by a following block; main-content scoping gates on the largest single <article>/<main> so a swarm of tiny cards cannot pass the threshold in aggregate and displace the real main. - preferFullToolOutput re-attaches the "Exit code N:" prefix to the fuller stream instead of appending the still-prefixed result, so a failed truncated tool no longer duplicates its stdout in the finished card. Adds hermetic tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve short live output on timed-out tools; strip inline-CSS-hidden subtrees and score truncated main-content scopes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten chat tool streaming comments and docstrings * Keep HTML READMEs from the GitHub API and preserve interrupted tool output Convert a 200 HTML README body from the GitHub README API to Markdown instead of discarding it and falling back to the repo page chrome, and promote captured live stdout to full output when a tool never reaches tool_end (stream interrupted or cancelled) so the partial diagnostics stay on the finished card. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: anchor HTML sniff, keep repeated sandbox writes, reuse textual tool ids Anchor _looks_like_html to the leading doctype/tag so a Markdown README that opens with a fenced HTML example stays Markdown (no html_to_markdown corruption), while bare HTML fragments (<body>/<article>/<section>) are still detected and converted on a missing/wrong Content-Type. Let the sandbox write fallback re-serve a target it already healed for the same invented absolute path, so iterative overwrites of a generated artifact stop failing with FileNotFoundError while the anti-clobber guard still refuses unrelated same-basename files. Reconcile the first textual tool call carrying an explicit id onto the open provisional TEXT card instead of spawning a duplicate card under that id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: run implicit-close before skipping tags and keep leading README tables as Markdown A skipped block (<nav>/<footer>) is an HTML5 optional-end-tag closer of an open <p>, but handle_starttag returned before the implicit-close bookkeeping, so a never-closed <p hidden> kept its hidden mark and swallowed every following sibling. Run _close_implicit before the skip decision so the hidden mark is released and trailing content renders. Drop <table> (and its <thead>/<tbody>/<tr>/<td>/<th> children) from the _looks_like_html leading set: Markdown READMEs routinely open with a raw HTML <table> badge/layout row, and sniffing that as HTML collapsed the whole Markdown body through html_to_markdown, exactly like the already-excluded <div align>/<p align> layout headers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make bypass-permissions Popen double faithful to the unified drain path The non-streaming _python_exec/_bash_exec now share _drain_process_output, which reads proc.stdout in a reader thread and calls proc.wait(); the test double only implemented communicate(), so bypass-mode bash returned an AttributeError instead of the faked output. Give _FakeProc a readable stdout pipe (yields the fake line then EOF), wait()/poll()/pid, so the test exercises the real drain path on both the python and bash bypass branches. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: persist sandbox path heals across runs and suppress nested hidden lists * Studio: run tool Python child unbuffered (-u) so unflushed prints stream live A long-running snippet doing bare print() without flush=True never reached the live-output pane: CPython block-buffers stdout when writing to a pipe, so _drain_process_output's readline() saw nothing until the buffer filled or the process exited. Launch the child with the interpreter -u flag so stdout is unbuffered and each print streams as it is produced. -u is applied unconditionally on both the streaming and non-streaming path, so the child invocation stays byte-identical with and without streaming and the final joined result is unchanged (buffering/timing only). Unlike the earlier PYTHONUNBUFFERED=1 env injection that was removed, -u does not pollute the child's os.environ and is not visible via os.getenv. * Render only the selected main-content subtree in html_to_markdown The main-content heuristic sized each <article>/<main> candidate individually to pick the largest subtree, but then rendered every matching tag in the document. A page with one real article plus sibling related-post cards or comment threads passed the size gate on the real article yet still emitted the unrelated siblings. Size and render the same chosen subtree so only the selected main-content subtree reaches the output. * Studio: tighten chat-tool-streaming fix comments * Studio: store tool-output-scope separators as unicode escapes The pane-scope and tool-output-key separators were literal NUL (0x00) bytes, which made git treat the file as binary and hide its diff and blame. Write them as \u0000 escapes instead; the runtime key value is unchanged. * Studio: bound tool-stream teardown when the client disconnects stream_tool_execution ran its yield loop with no try/finally, so a gen.close() on client disconnect (GeneratorExit at a yield) skipped the worker join and never signalled cancellation. A tool that does not poll cancel_event mid-flight (web_search, MCP, search_knowledge_base) then kept request teardown blocked until the tool's own timeout. Thread the request cancel_event into the wrapper, set it only on the abnormal-exit path so a clean multi-tool turn is unaffected, and bound the worker join to a few seconds; the daemon worker cannot outlive the process. * Studio: sandbox path remap no longer masks missing reads The sandbox sitecustomize shim remapped code-interpreter prefixes (/mnt/data, /workspace, ...) onto the working directory for every open mode, including reads. A read of a path that truly did not exist was silently redirected onto a same-basename workdir file instead of raising on the path the model used, hiding real missing-input errors. Remap writes and creates as before, but remap a read only when the mapped workdir target already exists (re-reading a just-written artifact); otherwise keep the original absolute path so the failure stays truthful. * Studio: bound web fetch with one overall deadline and cancellation The web fetch applied timeouts per network operation, so a GitHub README API attempt plus its HTML fallback plus up to five redirect hops could run well past the tool timeout, and nothing aborted once the client had disconnected. Add a single wall-clock deadline shared across the API attempt, the fallback, every redirect hop and the body read, cap each hop's socket timeout at the time left on the budget, and poll cancel_event. SSRF host pinning, per-hop redirect revalidation, the five-hop cap and the size cap are unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep tool-stream teardown off the event loop on disconnect The bounded worker join added for disconnect safety still ran on an abnormal close, so a client disconnect could wait the full join timeout; and the safetensors and Anthropic tool streams closed their generator synchronously on the event loop, unlike the GGUF path. On abnormal exit the daemon worker is abandoned, so join with a zero timeout instead of waiting; offload the safetensors and Anthropic gen.close to a thread to match GGUF; and surface a heartbeat as soon as cancel_event is set while the worker is silent so the route regains control at once instead of after a heartbeat interval. * Studio: extend the web-fetch deadline to DNS, the body read, and search The overall fetch deadline did not cover host resolution or the response body read, and query-mode web_search ignored cancellation. Resolve hosts (initial and every redirect) on a budget-polled helper so a slow or pre-cancelled getaddrinfo aborts on time; read the capped body in chunks with the budget re-checked between them so a slow-drip server cannot stretch a single read past the deadline; and gate the blocking DDGS query on cancel_event on both sides. SSRF host pinning, per-hop redirect revalidation, the five-hop cap and the size cap are unchanged. * Studio: defer the sandbox remap notice and tighten os.open create flags The one-shot remap notice fired while computing the mapping, so a read that kept its original path emitted a false notice and spent the notice a later genuine remap needed. Only emit it once _remap_open commits to the redirect. Separately, os.open classified O_TRUNC / O_APPEND without O_CREAT as creating, but those cannot create a missing file, so a missing target now stays truthful (only O_CREAT maps to the creating mode). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: convert only genuine HTML README bodies, not Markdown with a leading block tag The GitHub README API returns the raw file, almost always Markdown. _looks_like_html classified a Markdown README opening with a block tag (<ul>, <ol>, <dl>, <pre>, <blockquote>) as HTML, so _fetch_page_text ran it through html_to_markdown and collapsed its headings, lists and fenced code into a single line. Sniff the README body with a stricter document-level check (doctype or a leading <html>/<head>/<body>) so only a real .html README is converted; the general page path is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Surface unclassified mid-stream Anthropic errors as SSE error events The local Anthropic tool-stream and plain-stream paths called _anthropic_stream_error_event(e) with force defaulting to False, so an unclassified mid-stream failure (llama-server crash, decode OOM, a dropped upstream socket) returned no event. The except block then fell through to emitter.finish(), emitting a normal message_delta and message_stop that masked a truncated turn as a clean finish. Pass force = True at both fall-through sites so an unclassified failure emits a 500 SSE error event and returns, matching the Anthropic passthrough path that already forces it. Add regression tests covering both stream paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give each tool run a unique part id so finished cards keep their own output Backend tool ids restart at call_0 every assistant response, and the transient toolLiveOutput/toolFullOutput store maps were keyed by pane scope plus that bare backend id. Two turns in the same pane therefore shared one key: the stale-clear at tool_start only guards the forward direction, so when a later call_0 finished and wrote its preserved full output, every earlier still-mounted finished card reading the same key re-rendered and displayed the newer tool's output instead of its own. Mint one per-run-unique part id per backend id (call_0:<uuid>) and route tool_start/output/args/end through a single resolver so all events for a call resolve the same id. The durable part carries the unique id, so the finished-card readers derive a collision-free key with no change, and the awaiting-confirmation path keeps its own synthesized id. Outbound replay stays paired (the assistant tool_call id and the role=tool result tool_call_id both come from the part id) and gains unique ids across turns, which strict providers require. * Studio: tighten PR comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
1196 lines
43 KiB
Python
1196 lines
43 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Main-content extraction and boilerplate stripping for the web fetch tool.
|
|
|
|
The HTML fixtures below snapshot the relevant fragments of a real GitHub repo
|
|
page (github.com/unslothai/unsloth, fetched 2026-07): the ``hidden``
|
|
client-side error placeholders ("Uh oh! There was an error while loading."),
|
|
the skip-link / nav / footer furniture, and the README rendered inside
|
|
``<article class="markdown-body">``. No network access is required.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
_BACKEND_DIR = str(Path(__file__).resolve().parent.parent)
|
|
if _BACKEND_DIR not in sys.path:
|
|
sys.path.insert(0, _BACKEND_DIR)
|
|
|
|
from core.inference._html_to_md import html_to_markdown
|
|
from core.inference.tools import (
|
|
_fetch_page_text,
|
|
_fetch_url_raw,
|
|
_github_repo_readme_api_url,
|
|
_looks_like_html,
|
|
)
|
|
|
|
|
|
# ── Fixtures: snapshot of GitHub repo page fragments ─────────────
|
|
|
|
# GitHub ships client-side error placeholders behind the `hidden` attribute (JS
|
|
# reveals them on a failed fetch); a text converter must not surface them.
|
|
_GITHUB_HIDDEN_ERROR_BLOCK = """
|
|
<div data-show-on-forbidden-error hidden>
|
|
<div class="Box">
|
|
<div class="blankslate-container">
|
|
<h3 class="blankslate-heading">Uh oh!</h3>
|
|
<p class="blankslate-description">
|
|
<p class="color-fg-muted my-2 mb-2 ws-normal">There was an error while loading.
|
|
<a class="Link--inTextBlock" href="" aria-label="Please reload this page">Please reload this page</a>.</p>
|
|
</p>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
"""
|
|
|
|
_GITHUB_PAGE = f"""<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head><title>unslothai/unsloth</title></head>
|
|
<body>
|
|
<a class="px-2 py-4" href="#start-of-content">Skip to content</a>
|
|
<header class="Header-old">
|
|
<div class="AppHeader-globalBar">
|
|
<a href="/login">Sign in</a>
|
|
<a href="/signup">Sign up</a>
|
|
</div>
|
|
</header>
|
|
<div class="js-notification-shelf"></div>
|
|
<div hidden>
|
|
You signed in with another tab or window. Reload to refresh your session.
|
|
You signed out in another tab or window. Reload to refresh your session.
|
|
You switched accounts on another tab or window. Reload to refresh your session.
|
|
Dismiss alert
|
|
</div>
|
|
<template>{{{{ message }}}}</template>
|
|
{_GITHUB_HIDDEN_ERROR_BLOCK}
|
|
<main id="js-repo-pjax-container">
|
|
{_GITHUB_HIDDEN_ERROR_BLOCK}
|
|
<div id="repository-container-header">
|
|
<a href="/unslothai">unslothai</a> / <a href="/unslothai/unsloth">unsloth</a>
|
|
<a href="/login?return_to=%2Funslothai%2Funsloth">Notifications</a>
|
|
You must be signed in to change notification settings
|
|
</div>
|
|
<div class="repository-content">
|
|
<table aria-labelledby="folders-and-files">
|
|
<tr><th>Name</th><th>Last commit message</th></tr>
|
|
<tr><td><a href="/unslothai/unsloth/tree/main/unsloth">unsloth</a></td><td></td></tr>
|
|
</table>
|
|
<article class="markdown-body entry-content container-lg" itemprop="text">
|
|
<h1>Unsloth Studio</h1>
|
|
<p>Unsloth Studio lets you run and train models locally. Fine-tune and
|
|
run LLMs on Windows, Linux and macOS with a single install command,
|
|
then export to GGUF, Ollama, vLLM or Hugging Face when you are done.</p>
|
|
<h2>Install</h2>
|
|
<pre>curl -fsSL https://unsloth.ai/install.sh | sh</pre>
|
|
<p>See the <a href="https://unsloth.ai/docs">documentation</a> for
|
|
quickstarts, notebooks, and fine-tuning guides for every major model
|
|
family including Llama, Gemma, Qwen and DeepSeek.</p>
|
|
</article>
|
|
</div>
|
|
<div class="Layout-sidebar">
|
|
<h2>Languages</h2>
|
|
<ul>
|
|
<li><a href="/unslothai/unsloth/search?l=javascript">JavaScript 89.3%</a></li>
|
|
<li><a href="/unslothai/unsloth/search?l=python">Python 9.7%</a></li>
|
|
</ul>
|
|
</div>
|
|
</main>
|
|
<footer>
|
|
<a href="https://docs.github.com">Docs</a>
|
|
<a href="https://github.com/contact">Contact</a>
|
|
</footer>
|
|
<div aria-live="polite" aria-hidden="true">You can't perform that action at this time.</div>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
|
|
# ── html_to_markdown: hidden elements ────────────────────────────
|
|
|
|
|
|
def test_hidden_attribute_subtree_is_dropped():
|
|
html = "<body><p>visible</p><div hidden><p>secret error text</p></div><p>after</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "visible" in out
|
|
assert "after" in out
|
|
assert "secret error text" not in out
|
|
|
|
|
|
def test_aria_hidden_true_subtree_is_dropped():
|
|
html = '<body><p>keep</p><span aria-hidden="true">decoration</span></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "decoration" not in out
|
|
|
|
|
|
def test_aria_hidden_false_subtree_is_kept():
|
|
html = '<body><span aria-hidden="false">still here</span></body>'
|
|
assert "still here" in html_to_markdown(html)
|
|
|
|
|
|
def test_inline_style_display_none_subtree_is_dropped():
|
|
# Error/loading blocks are often hidden with inline CSS rather than the
|
|
# ``hidden`` attribute; browsers do not render them, so they must not leak.
|
|
html = (
|
|
"<body><p>visible</p>"
|
|
'<div style="display:none">secret loading block</div>'
|
|
"<p>after</p></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "visible" in out
|
|
assert "after" in out
|
|
assert "secret loading block" not in out
|
|
|
|
|
|
def test_inline_style_visibility_hidden_subtree_is_dropped():
|
|
html = '<body><p>keep</p><span style="visibility:hidden">ghost</span></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "ghost" not in out
|
|
|
|
|
|
def test_inline_style_display_none_important_is_dropped():
|
|
# The !important flag must not defeat the display:none detection.
|
|
html = '<body><p>keep</p><div style="display:none !important">gone</div></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "gone" not in out
|
|
|
|
|
|
def test_inline_style_display_none_among_other_declarations():
|
|
html = (
|
|
"<body><p>keep</p>" '<div style="color: red; display : none ; margin:0">gone</div></body>'
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "gone" not in out
|
|
|
|
|
|
def test_inline_style_visible_display_is_kept():
|
|
# Over-strip guard: display:block / visibility:visible render, and a value or
|
|
# URL merely containing the substring "none" must not trigger the hidden path.
|
|
html = (
|
|
"<body>"
|
|
'<div style="display:block">block kept</div>'
|
|
'<div style="visibility:visible">visible kept</div>'
|
|
'<a style="background:url(none.png)">link kept</a>'
|
|
"</body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "block kept" in out
|
|
assert "visible kept" in out
|
|
assert "link kept" in out
|
|
|
|
|
|
def test_hidden_recovers_from_omitted_close_tags():
|
|
# <p hidden> is never closed; the parent </div> must still end the hidden region.
|
|
html = "<body><div><p hidden>gone</div><p>kept</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "gone" not in out
|
|
assert "kept" in out
|
|
|
|
|
|
def test_nested_hidden_regions():
|
|
html = "<body><div hidden><div hidden>inner</div>outer</div><p>ok</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "inner" not in out
|
|
assert "outer" not in out
|
|
assert "ok" in out
|
|
|
|
|
|
def test_hidden_false_is_still_hidden():
|
|
# ``hidden`` is enumerated: the spec maps invalid/empty values to the Hidden
|
|
# state, so hidden="false" is NOT rendered and must not reach the Markdown.
|
|
html = '<body><p>keep</p><div hidden="false">not rendered</div></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "not rendered" not in out
|
|
|
|
|
|
def test_hidden_paragraph_omitted_close_does_not_swallow_siblings():
|
|
# HTML5 optional end tags: a sibling <p> start tag implicitly closes an open
|
|
# <p hidden>, so the hidden region ends there instead of swallowing siblings.
|
|
html = (
|
|
"<body><div><p hidden>secret"
|
|
"<p>visible one</p><p>visible two</p></div><p>after</p></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "visible one" in out
|
|
assert "visible two" in out
|
|
assert "after" in out
|
|
|
|
|
|
def test_hidden_list_item_omitted_close_keeps_following_items():
|
|
# <li hidden> without </li> is implicitly closed by the next <li>.
|
|
html = "<body><ul><li hidden>secret<li>shown A</li><li>shown B</li></ul></body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "shown A" in out
|
|
assert "shown B" in out
|
|
|
|
|
|
def test_hr_implicitly_closes_hidden_paragraph():
|
|
# Void elements also imply closes: <hr> ends an open <p hidden>.
|
|
html = "<body><p hidden>secret<hr>kept text</body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "kept text" in out
|
|
|
|
|
|
def test_skipped_tag_implicitly_closes_hidden_paragraph():
|
|
# A skipped block (<nav>/<footer>) also closes an open <p>. The optional-close
|
|
# bookkeeping must run before the skip, or the never-closed <p hidden> keeps its
|
|
# hidden mark and swallows every following sibling.
|
|
for skipped in ("nav", "footer"):
|
|
html = f"<body><p hidden>secret<{skipped}>chrome</{skipped}>VISIBLE</body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "chrome" not in out
|
|
assert "VISIBLE" in out
|
|
|
|
|
|
def test_hidden_void_element_is_suppressed():
|
|
# A hidden void element (<hr>/<br>) never joins the open-element stack, so it
|
|
# must be suppressed inline rather than emitting its markup.
|
|
html = '<body><p>before</p><hr aria-hidden="true"><p>after</p></body>'
|
|
out = html_to_markdown(html)
|
|
assert "before" in out
|
|
assert "after" in out
|
|
assert "---" not in out
|
|
|
|
|
|
def test_hidden_void_br_emits_no_break():
|
|
html = "<body><p>one<br hidden>two</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "one" in out
|
|
assert "two" in out
|
|
# The hidden <br> must not inject a newline between the two runs.
|
|
assert "one\ntwo" not in out
|
|
|
|
|
|
def test_visible_void_hr_still_renders():
|
|
# Guard: the suppression must not affect non-hidden void elements.
|
|
html = "<body><p>a</p><hr><p>b</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "---" in out
|
|
|
|
|
|
# ── html_to_markdown: main-content scoping ───────────────────────
|
|
|
|
|
|
def test_github_page_main_content_keeps_readme_only():
|
|
out = html_to_markdown(_GITHUB_PAGE, main_content = True)
|
|
# README content survives.
|
|
assert "Unsloth Studio" in out
|
|
assert "install.sh" in out
|
|
assert "documentation" in out
|
|
# Client-side error placeholders and page furniture are gone.
|
|
assert "Uh oh!" not in out
|
|
assert "There was an error while loading" not in out
|
|
assert "Please reload this page" not in out
|
|
assert "You can't perform that action at this time" not in out
|
|
assert "Skip to content" not in out
|
|
assert "Sign in" not in out
|
|
assert "Reload to refresh your session" not in out
|
|
assert "JavaScript 89.3%" not in out
|
|
assert "Languages" not in out
|
|
assert "Last commit message" not in out
|
|
|
|
|
|
def test_main_scope_used_when_no_article():
|
|
html = """
|
|
<body>
|
|
<header><a href="/login">Sign in</a></header>
|
|
<main><h1>Doc title</h1><p>%s</p></main>
|
|
<footer>footer junk</footer>
|
|
</body>
|
|
""" % ("Body text. " * 40)
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Doc title" in out
|
|
assert "Body text." in out
|
|
assert "Sign in" not in out
|
|
assert "footer junk" not in out
|
|
|
|
|
|
def test_main_content_falls_back_to_full_document():
|
|
# No article/main and a tiny body: the unscoped conversion is returned.
|
|
html = "<body><h1>Tiny</h1><p>Just a short page.</p></body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Tiny" in out
|
|
assert "Just a short page." in out
|
|
|
|
|
|
def test_tiny_article_stub_does_not_hijack_scope():
|
|
# An <article> with negligible text must not swallow the real content.
|
|
body_text = "Real content paragraph. " * 30
|
|
html = f"<body><article>ad</article><main><p>{body_text}</p></main></body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Real content paragraph." in out
|
|
|
|
|
|
def test_sibling_articles_do_not_leak_after_main_selected():
|
|
# The size gate picks the largest single <article> and renders only that
|
|
# subtree: sibling articles (related-post cards, comment threads) must not leak
|
|
# in just because the real article cleared the threshold.
|
|
real = "Main article body content for selection. " * 20
|
|
card = "Unrelated related-post card teaser blurb. " * 3
|
|
cards = "".join(f"<article><p>{card}</p></article>" for _ in range(5))
|
|
html = f"<body><article><h1>Real</h1><p>{real}</p></article>{cards}</body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Main article body content" in out
|
|
assert "Unrelated related-post" not in out
|
|
|
|
|
|
def test_default_conversion_unscoped_and_unstripped():
|
|
# Without main_content the whole document converts (backwards compatible),
|
|
# boilerplate included; only hidden subtrees are dropped.
|
|
html = "<body><p>Skip to content</p><div hidden>gone</div><main><p>hello</p></main></body>"
|
|
out = html_to_markdown(html)
|
|
assert "Skip to content" in out
|
|
assert "hello" in out
|
|
assert "gone" not in out
|
|
|
|
|
|
def test_boilerplate_filter_preserves_phrase_inside_real_prose():
|
|
# The furniture filter once matched by substring, deleting a real sentence that
|
|
# merely CONTAINS a fragment ("we use cookies"). It must drop only lines COMPOSED
|
|
# of furniture, keeping real prose that quotes one.
|
|
body = (
|
|
"<article><h1>Authentication</h1>"
|
|
"<p>We use cookies to authenticate API requests and keep sessions safe.</p>"
|
|
"<p>%s</p></article>"
|
|
) % ("Additional documentation content to select the article. " * 8)
|
|
out = html_to_markdown(f"<body>{body}</body>", main_content = True)
|
|
assert "We use cookies to authenticate API requests" in out
|
|
|
|
|
|
def test_boilerplate_filter_still_drops_standalone_and_stacked_furniture():
|
|
# A line that is purely furniture is dropped, as is one stacking several
|
|
# furniture phrases (as GitHub renders them).
|
|
body = (
|
|
"<article>"
|
|
"<p>Skip to content</p>"
|
|
"<p>You signed in with another tab or window. Reload to refresh your session.</p>"
|
|
"<p>Real README body. %s</p>"
|
|
"</article>"
|
|
) % ("Genuine documentation text. " * 8)
|
|
out = html_to_markdown(f"<body>{body}</body>", main_content = True)
|
|
assert "Real README body." in out
|
|
assert "Skip to content" not in out
|
|
assert "Reload to refresh your session" not in out
|
|
|
|
|
|
def test_boilerplate_not_stripped_inside_code_fences():
|
|
html = (
|
|
"<body><article><p>%s</p>"
|
|
"<pre>assert 'There was an error while loading' in page</pre>"
|
|
"</article></body>" % ("Prose. " * 40)
|
|
)
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "There was an error while loading" in out
|
|
|
|
|
|
def test_aside_callout_inside_article_is_kept():
|
|
# Docs render notes/warnings as <aside> callouts. An aside inside the selected
|
|
# article/main scope is real content and must survive; dropping it unconditionally
|
|
# loses page text.
|
|
body = (
|
|
"<article><h1>Guide</h1>"
|
|
"<p>%s</p>"
|
|
"<aside class='admonition warning'><strong>Warning:</strong> "
|
|
"This operation is destructive and cannot be undone.</aside>"
|
|
"<p>Trailing paragraph.</p></article>"
|
|
) % ("Documentation body text to select the article scope. " * 6)
|
|
out = html_to_markdown(f"<body>{body}</body>", main_content = True)
|
|
assert "This operation is destructive and cannot be undone." in out
|
|
assert "Warning:" in out
|
|
# Also kept in the unscoped (backwards-compatible) conversion.
|
|
out_full = html_to_markdown(f"<body>{body}</body>")
|
|
assert "This operation is destructive and cannot be undone." in out_full
|
|
|
|
|
|
# ── GitHub README rewrite ────────────────────────────────────────
|
|
|
|
|
|
def test_github_repo_url_maps_to_readme_api():
|
|
assert (
|
|
_github_repo_readme_api_url("https://github.com/unslothai/unsloth")
|
|
== "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
)
|
|
assert (
|
|
_github_repo_readme_api_url("https://github.com/unslothai/unsloth/")
|
|
== "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
)
|
|
assert (
|
|
_github_repo_readme_api_url("http://www.github.com/owner/repo.git")
|
|
== "https://api.github.com/repos/owner/repo/readme"
|
|
)
|
|
|
|
|
|
def test_github_non_repo_urls_are_not_rewritten():
|
|
for url in (
|
|
"https://github.com/unslothai/unsloth/tree/main/studio",
|
|
"https://github.com/unslothai/unsloth/issues/123",
|
|
"https://github.com/topics/llm",
|
|
"https://github.com/orgs/unslothai/repositories",
|
|
"https://github.com/login/oauth",
|
|
"https://github.com/unslothai",
|
|
"https://example.com/owner/repo",
|
|
"https://raw.githubusercontent.com/owner/repo/main/README.md",
|
|
):
|
|
assert _github_repo_readme_api_url(url) is None, url
|
|
|
|
|
|
def test_fetch_page_text_prefers_github_readme(monkeypatch):
|
|
calls = []
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
calls.append((url, extra_headers))
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, "# Unsloth\n\nFine-tune LLMs faster.", "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "Fine-tune LLMs faster." in out
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
assert len(calls) == 1
|
|
assert calls[0][1]["Accept"] == "application/vnd.github.raw+json"
|
|
|
|
|
|
def test_fetch_page_text_keeps_html_readme_from_api(monkeypatch):
|
|
# A repo whose README is HTML returns HTML from the README API with a 200. That
|
|
# success is authoritative: convert to Markdown and keep it, never discard it in
|
|
# favour of the repo root page's UI chrome.
|
|
html_readme = (
|
|
"<!doctype html><html><body>"
|
|
"<h1>Project Title</h1>"
|
|
"<p>Install with the one-line script and read the docs.</p>"
|
|
"</body></html>"
|
|
)
|
|
calls = []
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
calls.append(url)
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, html_readme, "text/html"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
# The successful README is converted and returned; no fallback fetch fires.
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
assert "Project Title" in out
|
|
assert "Install with the one-line script" in out
|
|
assert "<html" not in out
|
|
assert len(calls) == 1
|
|
|
|
|
|
def test_fetch_page_text_falls_back_to_html_when_readme_api_fails(monkeypatch):
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
if url.startswith("https://api.github.com/"):
|
|
return "Failed to fetch URL: HTTP 403 rate limited", "", ""
|
|
return None, _GITHUB_PAGE, "text/html"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
# Fallback converts the HTML page with the main-content heuristic.
|
|
assert "Unsloth Studio" in out
|
|
assert "Uh oh!" not in out
|
|
assert "There was an error while loading" not in out
|
|
|
|
|
|
def test_fetch_page_text_non_html_returned_raw(monkeypatch):
|
|
raw = "line one\n indented code\nline three"
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, raw, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://raw.githubusercontent.com/o/r/main/file.txt")
|
|
# Whitespace preserved: the HTML renderer would have collapsed it.
|
|
assert " indented code" in out
|
|
|
|
|
|
def test_fetch_page_text_html_conversion(monkeypatch):
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, _GITHUB_PAGE, "text/html"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth/tree/main")
|
|
assert "Unsloth Studio" in out
|
|
assert "Uh oh!" not in out
|
|
|
|
|
|
def test_fetch_page_text_propagates_fetch_errors(monkeypatch):
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return "Failed to fetch URL: HTTP 404 Not Found", "", ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
assert _fetch_page_text("https://example.com/missing") == (
|
|
"Failed to fetch URL: HTTP 404 Not Found"
|
|
)
|
|
|
|
|
|
def test_looks_like_html():
|
|
assert _looks_like_html("<!DOCTYPE html><html></html>")
|
|
assert _looks_like_html("\n <HTML lang='en'>")
|
|
assert not _looks_like_html("# Markdown README\n\n<h1>embedded html later</h1>")
|
|
assert not _looks_like_html("plain text")
|
|
|
|
|
|
def test_looks_like_html_markdown_with_leading_fenced_example_stays_markdown():
|
|
# A Markdown README OPENING with a fenced HTML example must not be sniffed as
|
|
# HTML just because a doctype/tag appears in the first 256 chars; html_to_markdown
|
|
# would corrupt the fences and prose.
|
|
fenced = (
|
|
"```html\n<!DOCTYPE html>\n<html><body><div>hi</div></body></html>\n```\n\n# Real README\n"
|
|
)
|
|
assert not _looks_like_html(fenced)
|
|
# Prose that mentions a tag inline, and a centered-logo README that opens
|
|
# with <p align>/<div align>/<h1 align>, also stay Markdown.
|
|
assert not _looks_like_html("Use the <html> element to start a page.")
|
|
assert not _looks_like_html('<p align="center"><img src="logo.png"></p>\n\n# Project\n')
|
|
assert not _looks_like_html('<div align="center">\n\n# Project\n\n</div>\n')
|
|
assert not _looks_like_html('<h1 align="center">Project</h1>\n\nMarkdown body.\n')
|
|
# An autolink is not a tag opener.
|
|
assert not _looks_like_html("<https://example.com> is the homepage")
|
|
|
|
|
|
def test_looks_like_html_detects_bare_fragments():
|
|
# A body that is a bare HTML fragment (no <html>/doctype) must still be
|
|
# recognized so it is converted to Markdown.
|
|
assert _looks_like_html("<body><p>hello</p></body>")
|
|
assert _looks_like_html("\n<article><h1>Title</h1><p>Body</p></article>")
|
|
assert _looks_like_html("<section>content</section>")
|
|
|
|
|
|
def test_looks_like_html_leading_table_stays_markdown():
|
|
# Markdown READMEs routinely open with a raw HTML <table> badge row or logo
|
|
# layout, then continue in Markdown. Sniffing that as HTML would collapse the
|
|
# Markdown body, so a leading <table> (and its row/cell children) must stay
|
|
# Markdown, like the excluded <div align>/<p align> layout headers.
|
|
assert not _looks_like_html("<table><tr><td>cell</td></tr></table>")
|
|
assert not _looks_like_html(
|
|
'<table align="center"><tr><td><img src="logo.png"></td></tr></table>\n\n# Project\n'
|
|
)
|
|
assert not _looks_like_html("<tr><td>cell</td></tr>")
|
|
|
|
|
|
def test_fetch_page_text_keeps_markdown_readme_with_html_example(monkeypatch):
|
|
# A Markdown README opening with a fenced HTML snippet must be served verbatim,
|
|
# never run through html_to_markdown (which would drop the fences/tags).
|
|
md_readme = (
|
|
"```html\n"
|
|
"<!DOCTYPE html>\n"
|
|
"<html><body><h1>Demo</h1></body></html>\n"
|
|
"```\n\n"
|
|
"# My Project\n\nInstall and run.\n"
|
|
)
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, md_readme, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
# Markdown preserved verbatim: the fence and literal tags survive.
|
|
assert "```html" in out
|
|
assert "<!DOCTYPE html>" in out
|
|
assert "# My Project" in out
|
|
|
|
|
|
def test_fetch_page_text_keeps_markdown_readme_with_leading_table(monkeypatch):
|
|
# A README opening with a raw HTML <table> badge/layout row then continuing in
|
|
# Markdown must be served verbatim, never run through html_to_markdown (which
|
|
# would collapse the list/fence/heading body onto one line).
|
|
md_readme = (
|
|
'<table align="center">\n'
|
|
'<tr><td><img src="logo.png"></td><td>Badges</td></tr>\n'
|
|
"</table>\n\n"
|
|
"# My Project\n\n"
|
|
"- feature one\n"
|
|
"- feature two\n\n"
|
|
"```python\nprint('hi')\n```\n"
|
|
)
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, md_readme, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
# Markdown body verbatim: list, fence and heading survive on their own lines.
|
|
assert "- feature one\n- feature two" in out
|
|
assert "```python" in out
|
|
assert "# My Project" in out
|
|
|
|
|
|
def test_fetch_url_raw_missing_content_type_reported_empty(monkeypatch):
|
|
# Message.get_content_type() falls back to the RFC 2045 "text/plain" default
|
|
# when the header is absent; _fetch_url_raw must report "" instead so the HTML
|
|
# sniffing fallback can fire.
|
|
import email
|
|
import urllib.request
|
|
|
|
class _FakeResp:
|
|
headers = email.message_from_string("")
|
|
|
|
def __init__(self):
|
|
self._body = b"<html><body>hello</body></html>"
|
|
|
|
def read(self, n = -1):
|
|
# Hand back the body once, then EOF, so the chunked reader terminates.
|
|
body, self._body = self._body, b""
|
|
return body
|
|
|
|
class _FakeOpener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
return _FakeResp()
|
|
|
|
monkeypatch.setattr(
|
|
"core.inference.tools._validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _FakeOpener())
|
|
err, body, content_type = _fetch_url_raw("https://example.com/")
|
|
assert err is None
|
|
assert "hello" in body
|
|
assert content_type == ""
|
|
|
|
|
|
def test_fetch_page_text_missing_content_type_html_sniffed(monkeypatch):
|
|
# A header-less server returning an HTML body must still be converted.
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, _GITHUB_PAGE, ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/no-content-type")
|
|
assert "Unsloth Studio" in out
|
|
assert "<html" not in out
|
|
assert "Uh oh!" not in out
|
|
|
|
|
|
def test_fetch_page_text_missing_content_type_fragment_converted(monkeypatch):
|
|
# A header-less server returning a bare HTML fragment (no <html>/doctype) must
|
|
# still be sniffed as HTML and converted, not served as raw markup.
|
|
fragment = "<article><h1>Doc Title</h1><p>Readable fragment body.</p></article>"
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, fragment, ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/fragment")
|
|
assert "Doc Title" in out
|
|
assert "Readable fragment body." in out
|
|
assert "<article" not in out
|
|
|
|
|
|
def test_fetch_page_text_missing_content_type_plain_text_raw(monkeypatch):
|
|
# A header-less server returning plain text stays raw (whitespace kept).
|
|
raw = "line one\n indented code\nline three"
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, raw, ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/no-content-type.txt")
|
|
assert " indented code" in out
|
|
|
|
|
|
def test_fetch_page_text_mislabeled_text_plain_html_converted(monkeypatch):
|
|
# An explicit text/plain header on an HTML body is sniffed and converted, like
|
|
# the pre-extraction behavior of always converting HTML pages.
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, _GITHUB_PAGE, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/mislabeled")
|
|
assert "Unsloth Studio" in out
|
|
assert "<html" not in out
|
|
|
|
|
|
# ── implicit-close past unclosed inline descendants (finding 14) ──
|
|
|
|
|
|
def test_hidden_paragraph_with_inline_child_implicitly_closed_by_block():
|
|
# A browser closes an open <p> when a <div> arrives, even with an unclosed
|
|
# <span> on top of it. The hidden region must end there, not swallow the
|
|
# following visible blocks.
|
|
html = "<body><p hidden><span>secret<div>visible div</div><p>visible paragraph</body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "visible div" in out
|
|
assert "visible paragraph" in out
|
|
|
|
|
|
def test_hidden_list_item_with_inline_child_closed_by_next_item():
|
|
html = "<body><ul><li hidden><span>secret<li>visible item</ul><p>after</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "visible item" in out
|
|
assert "after" in out
|
|
|
|
|
|
# ── nested hidden list/table contents must stay suppressed ──
|
|
|
|
|
|
def test_nested_hidden_list_does_not_leak_child_items():
|
|
# The nested <ul> re-scopes the item, so the inner <li> is a DESCENDANT of the
|
|
# hidden outer <li>, not an optional-close sibling. Optional-end-tag recovery
|
|
# must not cross the intervening <ul>, or the outer li's hidden mark is popped
|
|
# and the nested text leaks.
|
|
html = (
|
|
"<body><ul>"
|
|
"<li hidden>parent<ul><li>secret child</li></ul></li>"
|
|
"<li>visible sibling</li>"
|
|
"</ul></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "parent" not in out
|
|
assert "secret child" not in out
|
|
assert "visible sibling" in out
|
|
|
|
|
|
def test_nested_hidden_list_with_omitted_closes_stays_suppressed():
|
|
# Same leak, doubly nested with omitted </li>/</ul>. Every hidden descendant
|
|
# stays gone; the following visible sibling (which implicitly closes the hidden
|
|
# outer <li>) still renders.
|
|
html = (
|
|
"<body><ul>"
|
|
"<li hidden>parent<ul><li>secret child<ul><li>deeper secret</ul></li></ul>"
|
|
"<li>visible sibling"
|
|
"</ul></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "parent" not in out
|
|
assert "secret child" not in out
|
|
assert "deeper secret" not in out
|
|
assert "visible sibling" in out
|
|
|
|
|
|
def test_nested_hidden_table_does_not_leak_inner_cells():
|
|
# A nested <table> re-scopes <tr>/<td>: an inner <td> must not be an
|
|
# optional-close sibling of a hidden outer <td> across the nested table.
|
|
html = (
|
|
"<body><table><tr>"
|
|
"<td hidden>outer<table><tr><td>secret cell</td></tr></table></td>"
|
|
"<td>visible cell</td>"
|
|
"</tr></table></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "secret cell" not in out
|
|
assert "visible cell" in out
|
|
|
|
|
|
# ── aggregate tiny <article> cards must not displace <main> (finding 15) ──
|
|
|
|
|
|
def test_many_tiny_articles_do_not_displace_substantial_main():
|
|
cards = "".join(
|
|
f"<article><h2>Teaser {i}</h2><p>Advertisement card blurb.</p></article>" for i in range(12)
|
|
)
|
|
main_body = "Authoritative main documentation content. " * 30
|
|
html = f"<body>{cards}<main><h1>Real page</h1><p>{main_body}</p></main></body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Authoritative main documentation content." in out
|
|
assert "Advertisement card blurb." not in out
|
|
|
|
|
|
def test_single_substantial_article_still_preferred_over_main():
|
|
# GitHub-README case: one substantial <article> inside <main> must still win
|
|
# over sibling <main> furniture.
|
|
article_body = "Real README documentation body text. " * 20
|
|
html = (
|
|
"<body><main>"
|
|
f"<article><h1>Guide</h1><p>{article_body}</p></article>"
|
|
"<div><h2>Languages</h2><p>JavaScript 89.3%</p></div>"
|
|
"</main></body>"
|
|
)
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Real README documentation body text." in out
|
|
assert "JavaScript 89.3%" not in out
|
|
|
|
|
|
# ── truncated (unclosed) main-content scopes must still be scored ──
|
|
|
|
|
|
def test_truncated_open_article_scope_is_scored_and_preferred():
|
|
# _fetch_url_raw caps large pages, so the download can end before the closing
|
|
# </article>. The scope is still the main content and must be preferred over the
|
|
# whole document (which re-leaks the page chrome).
|
|
chrome = "<nav>Skip to content</nav><div>Repository file tree and page chrome.</div>"
|
|
article_body = "Real README documentation body text. " * 20
|
|
# No closing </article> / </body> -- the fetch cap truncated the page.
|
|
html = f"<body>{chrome}<article><h1>Guide</h1><p>{article_body}</p>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Real README documentation body text." in out
|
|
assert "Repository file tree and page chrome." not in out
|
|
|
|
|
|
def test_truncated_open_main_scope_is_scored_and_preferred():
|
|
chrome = "<nav>Skip to content</nav><div>Repository file tree and page chrome.</div>"
|
|
main_body = "Authoritative main documentation content. " * 30
|
|
html = f"<body>{chrome}<main><h1>Doc</h1><p>{main_body}</p>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Authoritative main documentation content." in out
|
|
assert "Repository file tree and page chrome." not in out
|
|
|
|
|
|
# ── overall fetch deadline + cancellation (no per-hop timeout blowup) ──
|
|
|
|
|
|
def test_fetch_url_raw_overall_deadline_aborts_across_redirects(monkeypatch):
|
|
# Each hop advances a fake clock by 5s; an 8s overall budget is exhausted on the
|
|
# third hop even though every hop stays within its own socket timeout. Without
|
|
# the deadline this would redirect until the 5-hop cap, so the "timed out" error
|
|
# proves the overall budget aborted it, not the hop cap.
|
|
import urllib.request
|
|
from urllib.error import HTTPError
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
clock = {"t": 1000.0}
|
|
monkeypatch.setattr(tools_mod.time, "monotonic", lambda: clock["t"])
|
|
|
|
hops = {"n": 0}
|
|
|
|
class _RedirectingOpener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
clock["t"] += 5.0
|
|
hops["n"] += 1
|
|
raise HTTPError(
|
|
req.full_url,
|
|
302,
|
|
"Found",
|
|
{"Location": "https://example.com/next"},
|
|
None,
|
|
)
|
|
|
|
monkeypatch.setattr(
|
|
tools_mod,
|
|
"_validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _RedirectingOpener())
|
|
|
|
err, body, content_type = tools_mod._fetch_url_raw(
|
|
"https://example.com/start",
|
|
timeout = 30,
|
|
deadline = clock["t"] + 8.0,
|
|
)
|
|
assert err == "Failed to fetch URL: timed out."
|
|
assert body == ""
|
|
assert hops["n"] < 5
|
|
|
|
|
|
def test_fetch_url_raw_cancel_event_aborts_before_network(monkeypatch):
|
|
# A set cancel_event (client disconnected) stops the fetch before it opens any
|
|
# socket, so a dropped stream cannot leave a tool blocking on the wire.
|
|
import threading
|
|
import urllib.request
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
ev = threading.Event()
|
|
ev.set()
|
|
opened = {"n": 0}
|
|
|
|
class _Opener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
opened["n"] += 1
|
|
raise AssertionError("network must not be touched after cancel")
|
|
|
|
monkeypatch.setattr(
|
|
tools_mod,
|
|
"_validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _Opener())
|
|
|
|
err, body, content_type = tools_mod._fetch_url_raw(
|
|
"https://example.com/",
|
|
cancel_event = ev,
|
|
)
|
|
assert err == "Failed to fetch URL: cancelled."
|
|
assert opened["n"] == 0
|
|
|
|
|
|
def test_fetch_page_text_shares_one_deadline_across_readme_and_fallback(monkeypatch):
|
|
# The README API attempt and its HTML fallback must draw from ONE budget: a
|
|
# failed API call cannot hand the fallback a fresh full timeout.
|
|
seen_deadlines = []
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
seen_deadlines.append(deadline)
|
|
# Fail the README API so the HTML fallback also runs.
|
|
return "Failed to fetch URL: HTTP 429 rate limited", "", ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth", timeout = 30)
|
|
assert out == "Failed to fetch URL: HTTP 429 rate limited"
|
|
# Both attempts ran and shared the same, single deadline value.
|
|
assert len(seen_deadlines) == 2
|
|
assert seen_deadlines[0] is not None
|
|
assert seen_deadlines[0] == seen_deadlines[1]
|
|
|
|
|
|
# -- overall deadline reaches the body read, the resolver, and the query path --
|
|
|
|
|
|
def test_fetch_url_raw_deadline_aborts_slow_body(monkeypatch):
|
|
# A server dribbling the body must not stretch the read past the overall
|
|
# deadline: the body is read in chunks with the budget re-checked between them,
|
|
# so a single slow resp.read cannot outlast the fetch budget.
|
|
import email
|
|
import urllib.request
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
clock = {"t": 1000.0}
|
|
monkeypatch.setattr(tools_mod.time, "monotonic", lambda: clock["t"])
|
|
|
|
class _DrippingResp:
|
|
headers = email.message_from_string("")
|
|
|
|
def read(self, n = -1):
|
|
# One chunk, then jump the clock past the deadline so the next
|
|
# between-chunk budget check aborts instead of reading forever.
|
|
clock["t"] += 10.0
|
|
return b"x" * 16
|
|
|
|
def close(self):
|
|
pass
|
|
|
|
class _Opener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
return _DrippingResp()
|
|
|
|
monkeypatch.setattr(
|
|
tools_mod,
|
|
"_validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _Opener())
|
|
|
|
err, body, content_type = tools_mod._fetch_url_raw(
|
|
"https://example.com/",
|
|
timeout = 30,
|
|
deadline = clock["t"] + 5.0,
|
|
)
|
|
assert err == "Failed to fetch URL: timed out."
|
|
assert body == ""
|
|
|
|
|
|
def test_resolve_with_budget_aborts_on_slow_resolver(monkeypatch):
|
|
# getaddrinfo has no deadline of its own; a resolver slower than the budget must
|
|
# abort on time instead of blocking the whole fetch.
|
|
import threading
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
clock = {"t": 1000.0}
|
|
monkeypatch.setattr(tools_mod.time, "monotonic", lambda: clock["t"])
|
|
|
|
release = threading.Event()
|
|
|
|
def slow_resolve(host, port):
|
|
release.wait(5.0) # block until released; the budget should abort first
|
|
return True, "", "203.0.113.7"
|
|
|
|
monkeypatch.setattr(tools_mod, "_validate_and_resolve_host", slow_resolve)
|
|
|
|
def advance_past_deadline():
|
|
import time as _t
|
|
_t.sleep(0.1)
|
|
clock["t"] += 100.0
|
|
|
|
t = threading.Thread(target = advance_past_deadline, daemon = True)
|
|
t.start()
|
|
try:
|
|
ok, reason, ip = tools_mod._resolve_with_budget(
|
|
"example.com",
|
|
443,
|
|
1005.0,
|
|
None,
|
|
)
|
|
finally:
|
|
release.set()
|
|
assert ok is False
|
|
assert reason == "Failed to fetch URL: timed out."
|
|
|
|
|
|
def test_web_search_query_cancelled_skips_search(monkeypatch):
|
|
# A pre-set cancel_event (client disconnected) skips the blocking DDGS query,
|
|
# matching the direct-URL path's cancellation.
|
|
import sys
|
|
import threading
|
|
import types
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
ev = threading.Event()
|
|
ev.set()
|
|
called = {"n": 0}
|
|
|
|
class _DDGS:
|
|
def __init__(self, *a, **k):
|
|
called["n"] += 1
|
|
|
|
def text(self, *a, **k):
|
|
called["n"] += 1
|
|
return []
|
|
|
|
fake_mod = types.ModuleType("ddgs")
|
|
fake_mod.DDGS = _DDGS
|
|
monkeypatch.setitem(sys.modules, "ddgs", fake_mod)
|
|
|
|
out = tools_mod._web_search("some query", cancel_event = ev)
|
|
assert out == "Search cancelled."
|
|
assert called["n"] == 0
|
|
|
|
|
|
def test_fetch_page_text_markdown_readme_with_leading_block_tag_stays_markdown(monkeypatch):
|
|
# A raw-Markdown README that OPENS with an HTML block tag (<blockquote>, <ul>,
|
|
# <pre>, ...) must not be run through html_to_markdown, which would collapse its
|
|
# headings/list/fence. Only a real HTML document (doctype / <html>) is converted.
|
|
md_readme = (
|
|
"<blockquote>Note: pre-release.</blockquote>\n\n"
|
|
"# My Project\n\n"
|
|
"Install:\n\n"
|
|
"- step one\n"
|
|
"- step two\n\n"
|
|
"```bash\npip install myproject\n```\n"
|
|
)
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, md_readme, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
# Markdown structure survives verbatim (heading, list, fenced code).
|
|
assert "# My Project" in out
|
|
assert "- step one" in out
|
|
assert "```bash" in out
|
|
|
|
|
|
def test_looks_like_html_document_only_matches_real_documents():
|
|
from core.inference.tools import _looks_like_html_document
|
|
|
|
assert _looks_like_html_document("<!doctype html><html><body>x</body></html>")
|
|
assert _looks_like_html_document("\n <HTML lang='en'>")
|
|
assert _looks_like_html_document("<body><h1>x</h1></body>")
|
|
# Block tags a Markdown README can open with are NOT full documents.
|
|
for frag in (
|
|
"<blockquote>q</blockquote>",
|
|
"<ul><li>x</li></ul>",
|
|
"<pre>x</pre>",
|
|
"<dl><dt>x</dt></dl>",
|
|
):
|
|
assert not _looks_like_html_document(frag), frag
|