502730bbba* Studio: add durable Deep Research workflows * Studio: preserve research integration after upstream updates * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep research worker compatible with Python 3.11 * Studio: address Deep Research lifecycle review * Studio: preserve durable research recovery * Studio: preserve research stream and context * Studio: harden research sources and limits * Studio: align research with shared chats * Studio: guard durable research actions * Studio: protect durable research turns * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: deepen durable research decisions * Studio: protect research prompts and queries * Studio: slim research stream deltas * Studio: preserve research evidence and citations * Studio: harden Deep Research (CI, prompt injection, query PII, config, citations) - Fix backend CI: add research_runs_router to the synthetic routes stub in test_desktop_auth so studio.backend.main imports under the health-check test. - Escape prompt-delimiter tags in the decision and synthesis prompts so gathered web/document content cannot close an <untrusted_...> wrapper and inject instructions into the local planner/decision/synthesis model. - Extend the public-query sanitizer to redact Luhn-valid payment cards, phone numbers, non-global IPs, and labeled private identifiers before a query can reach web search. - Reject nested credential keys in inferenceRequest and ragScope, not just top-level keys, when persisting a durable run config. - Treat maxSources as one budget shared across web and document sources (collection and resume paths) instead of per type, which allowed up to 2x the configured cap. - Preserve document citations whose filename contains a closing bracket by tokenizing valid citations before stripping invalid ones. - Persist Deep Research off when switching to an external model and when enabling Web Fetch so a refresh cannot rehydrate a mutually-exclusive state. - Add regression tests for the query, prompt, citation, and config hardening. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the research claims table migration atomic The owner-scoped to global claims migration ran its RENAME, CREATE, INSERT and DROP in autocommit, so an interruption after CREATE left the new table empty, orphaned the rows in the legacy table, and never re-triggered. Wrap the rebuild in an explicit transaction so a crash rolls back cleanly and the migration re-runs on the next boot. * Studio: block message edits and regeneration during an active research run After a reload a durable research run is followed by the research store rather than an assistant-ui run, so thread.isRunning is false while research is still active. Message edit, refresh and the edit composer previously gated only on isRunning, which let a normal generation start alongside the running research run. Gate them on the active thread's research state as well. * Studio: keep the plan review mounted through approval Keying PlanReview on planRevision remounted it mid-approve when updateResearchPlan bumped the revision, resetting the local pending flag and re-enabling Start research while the approve was still in flight, which allowed a duplicate approve. Key on runId only. * Studio: drop the redundant deep-research persistence change setCheckpoint already persists Deep Research off for external models at the top of the function, so the added saveBool was a duplicate, and clearing Deep Research from setWebFetchToolsEnabled guarded a state that is not reachable (Deep Research is local-model only while the Web Fetch pill is external-provider only). Revert both to the pre-hardening version. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden Deep Research citations, query privacy, and message protection Address review findings in the Deep Research backend: - Escape an unbalanced ")" in citation destinations so a source URL cannot close the markdown link early and inject a second link, keeping balanced parentheses literal. - Match raw-URL citations on whole tokens so a URL sharing another URL's prefix is no longer partially rewritten. - Redact non-global IPv6 addresses in public search queries, matching the existing IPv4 handling. - Detect credential key names after normalizing case and separators so nested openaiApiKey, accessToken, and clientSecret values cannot be persisted. - Reject client edits to server-managed research prompts and reports at the storage layer; only the internal writers pass allow_research_update. - Scope research searches to the first allowed domains instead of dropping site scoping for large allow lists. - Persist the same fetch evidence bound used during live synthesis so a resumed run is not shortened. - Scope run completion so it only replaces this run's message parts. Add regression tests for the above. * Studio: fix Deep Research SSE framing, source counts, and favicon privacy - Normalize the whole SSE buffer so a CRLF split across transport chunks still frames events. - Count web and document sources together in the activity header so a RAG-only run is not shown as zero sources. - Cap the plan editor at the run's configured maxSteps instead of a hard-coded 30. - Add an allowRemoteIcons opt-out to the sources components and disable third-party favicon requests for research sources so visited domains are not leaked. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address final Deep Research review findings * Studio: fit Deep Research synthesis evidence to loaded context, add opt-in web grounding Size the synthesis evidence budget to the loaded model context so the prompt is not silently truncated on small contexts. When the evidence overflowed the window the report degenerated (it echoed the evidence tail instead of writing); the budget now reserves tokens for the prompt scaffolding and converts the remainder to chars, keeping the full cap when the context is unknown. Add opt-in web grounding for auto-read: read the top search results, ingest them into an ephemeral RAG scope, hybrid-retrieve the passages most relevant to the question with the existing knowledge-base retriever, and fold those chunks into the step evidence. The scope is per call and deleted afterwards, so a user's knowledge base is never touched. Off by default; enable with UNSLOTH_RESEARCH_AUTO_SCRAPE=1. Gated per run by budgets["maxAutoScrape"], so runs created without it keep legacy snippet-only behavior, and grounding is skipped when the loaded context is too small for the prompt. Add tests for the adaptive evidence budget, scraped-text cleaning, the ephemeral web-RAG retrieval and scope cleanup, and the auto-read evidence path. * Studio: read Deep Research synthesis context from the inference orchestrator Make the adaptive synthesis-evidence budget actually engage in the normal Studio architecture. _loaded_context_length read core.inference.inference, the low-level backend that lives in the model subprocess and stays unpopulated in the main web process where the research supervisor runs, so it returned None and the budget silently fell back to the 32000 character cap (leaving the report exposed to the truncation this was meant to fix). Read the inference orchestrator instead, and the llama.cpp backend for GGUF, mirroring routes.inference._monitor_context_length so the budget sizes to the context the API layer serves. Verified on a running server: at a 12288 token load the probe now reports 12288 and the budget adapts to 24576 characters instead of the 32000 fallback. Also: - Reserve context for the generated report as well as the prompt scaffolding (raise the reserve to 4096 tokens) so evidence does not crowd out the output on a small window. - Honor a numeric UNSLOTH_RESEARCH_AUTO_SCRAPE by passing the per-run maxAutoScrape as the page cap to the scraper, instead of always reading the maximum. - Guard the web-RAG connection acquisition so a get_connection failure returns the documented empty result rather than propagating. - Add a synthesis-context test that patches the real backend accessor (not the probe itself) so the production wiring is exercised, plus a scrape page-cap test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden Deep Research query redaction and research autosave - research_runs: extend the opaque-token allowlist so unlabeled Hugging Face (hf_) and GitLab (glpat-) tokens are redacted before a query can reach web search, without over-redacting public model or version ids. - runtime-provider: for a server-managed research message, echo the backend-stored metadata verbatim on autosave. Merging the client metadata re-added client-only fields the server never persisted, so the server-side guard saw a diff and rejected every streamed or snapshot update with 409. * Studio: keep composer tool pills always accessible after merge The merge left the composer line marked always-expanded (data-expanded "true") while the inner pill row was still gated behind composerExpanded, so the Search and Code toggles disappeared once the permission mode was "off" with no other toggle set. Render the primary tool pills unconditionally, matching the always-expanded layout, and drop the now unused composerExpanded and permissionMode locals. Fixes the Chat UI Playwright check that asserts the Search and Code pills stay visible. * Studio: update Deep Research composer contract to always-expanded layout The always-expanded composer no longer routes effectiveDeepResearchEnabled through a composerExpanded expression, so the frontend contract now checks that it gates the Deep Research composer button render instead. * Studio: do not bind a research run to a populated assistant reply create_run adopted any assistant message under the user turn whose researchRunId was unset, including a prior answer reused by a retry. On completion _update_assistant drops the untagged text and source parts, so that answer was silently overwritten. Only bind to an empty placeholder or this run's own message, and reject a reply that already carries content. * Studio: harden Deep Research synthesis budget, prompt shielding, and message protection - research_runs: split the synthesis evidence budget evenly across notes so a small context still keeps a slice of every research step instead of dropping the later steps after the earliest ones fill the budget. - research_runs: shield the research question and approved plan before placing them in the decision and synthesis prompts, so a closing delimiter in either cannot escape its block and inject sibling sections. - research_runs: redact bearer authorization tokens from public search queries. - studio_db: include attachments in the research-message change check and guard direct attachment deletion, so server-managed research prompts and responses cannot be mutated through the attachment paths. - chat_history: map the protected-message conflict on attachment deletion to 409. * Studio: strip invalid document citations that contain brackets The invalid-citation regex stopped at the first closing bracket, so a citation whose filename contained brackets left its tail (".pdf, p. 9]") in the report. Match a balanced bracketed span so the whole invalid citation is removed; valid citations stay protected by the earlier tokenization pass. * Studio: free the RAG search slot when a lookup times out or is cancelled The bounded knowledge-base search held the sole admission slot in a detached worker until the search returned, so a lookup that outlived its timeout (a stalled embedding or blocked vector call) kept the slot forever and starved every later lookup, disabling knowledge-base retrieval globally. Release the slot from the caller when it stops waiting, exactly once, so a detached worker finishes without re-holding it. * Studio: remove Websites label from research composer * Studio: fix Deep Research review findings (RAG slot bound, orphaned workers, hardening) - Bound the shared RAG search slot to one running worker. The search that is doing the embedding/index/GPU work now owns the admission slot until it finishes, instead of freeing it on caller timeout while the detached worker keeps running, which let a second search enter and stack concurrent work behind the capacity-of-one semaphore. - Cancel active research runs before deleting their thread, project, or all history. Deleting cascade-drops the run row, but the worker only notices at its next lease check, so it could keep doing model/web/RAG work for a run that no longer exists; signalling cancel first shortens that window. - Shield the planner prompt's conversation and question with _shield_untrusted, matching the decision and synthesis prompts, so untrusted text cannot forge planner delimiters. - Do not let a research key-revocation failure replace a successful non-streaming completion; log it like the streaming path does. - Include created_at in the protected research-message guard so a client cannot reorder server-managed prompt/response messages while leaving the body intact. - Reject non-scalar ragScope values; a nested container evades the sensitive-key scan when its inner keys are unlisted and would reach retrieval code that expects a scalar scope id. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: remove research composer globe icon * Studio: use Hugeicons telescope in research composer * Studio: use Telescope02 icon in research composer * Studio: standardize Deep Research telescope icons * Studio: move Deep Research below web and code tools * Studio: merge grounded page excerpts with search snippets instead of replacing When auto-scrape grounding retrieved page-body chunks, it replaced the raw search-result text for that step. If the retrieved chunk was a distractor or dropped the key fact, the answer-bearing search snippet was lost and grounded runs regressed below snippet-only accuracy on factual questions (e.g. returning Apache 2.0 instead of the Qwen License, 403 instead of 404, or a single mirror diameter instead of the sum). Keep the search snippets and append the grounded excerpts as supplementary evidence via a small _merge_scraped_evidence helper. Grounding stays opt-in and off by default, so legacy runs are unchanged. Adds regression tests. * Studio: fix stale website access assertion in Deep Research contract test The dialog heading was renamed to a DialogTitle, so the contract test still asserted a <span>Websites</span> that no longer exists and failed on every branch built on this one. Assert the current heading instead. * Add AGPL-3.0 SPDX header to the two new test files for PR #7219 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix citation loss, effort clamping and nested inferenceRequest for PR #7219 Three review findings, each with a regression test that fails without the fix. Citation dropped for a bare URL in prose parentheses. _RAW_URL swallows the closing paren and the old trim set only stripped ".,;:!?", so the catalog lookup missed and the validator deleted the whole citation, leaving an unbalanced "(" in the report. New _trim_url_tail follows GFM extended autolink path validation: one right-to-left pass that interleaves punctuation and unmatched-")" trimming. Both rules must run in the same loop, else "https://x/y.)" keeps a stray dot. Balanced parens inside a URL (Wikipedia-style) still survive. Output verified against cmark-gfm on nine cases, including "https://x/foo)bar)" which must keep ")bar". Research runs forwarded reasoningEffort unclamped. The local chat path clamps to the loaded model's advertised levels; the research branch did not, and the backend only validates enum membership, so llama.cpp dropped a level the model lacks and the whole durable run silently fell back to the template default. Now uses the same helper and the same levels as normal chat. Note this makes "max" on a gpt-oss low|medium|high model resolve to "low" rather than falling through to the template default, matching normal chat exactly; the divergence between the two paths was the bug. Nested inferenceRequest values were persisted. Every allowed field is a scalar and the numeric/bool/enum ones reject a container while coercing, but "model" is stringified with str(), which never raises, so {"auth": "sk-..."} slipped past the sensitive-key scan ("auth" is not on the list) into the durable run config as the model id. Mirrors the ragScope guard already in this PR. Verified: 542 passed across the research/web/sandbox/chat-history backend suites, frontend contract 10 passed, tsc --noEmit clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix report-stalling regex, uncataloged KB evidence and bracketed titles for PR #7219 Catastrophic backtracking in _DOCUMENT_CITATION. The alternation (?:[^\[\]]+|\[[^\[\]]*\])* backtracks exponentially on an unterminated "[Document:" with no later bare "]", which is ordinary malformed model output and exactly what this sanitizer exists to handle. Runtime quadrupled every two characters; one realistic 76-char line did not finish in 90s. It runs synchronously inside async _research (the line below it uses asyncio.to_thread), so a single bad report pins the event loop and stalls all of Studio, not just the run. Replaced with the language-equivalent unrolled form, verified identical on well-formed inputs including bracketed filenames, and linear: a 20,000-char tail now takes 0.4ms. Not using possessive quantifiers or atomic groups, which need Python 3.11 while this package declares >=3.9. Uncataloged knowledge base evidence reached synthesis. When maxSources is already full, every returned chunk hits the continue, so accepted_rag_sources stays empty, the "if accepted_rag_sources" rebuild no-ops and rag_result keeps the raw KB text. That text has no document_source_catalog entry, so the validator strips any citation to it and synthesis is left building claims on private KB chunks it cannot attribute. Cleared, gated on rag_sources so a text-only KB reply is still passed through. The resume branch built rag_evidence from all restored sources with the same hole, so it now mirrors the live loop. Bracketed source titles destroyed their own citation. The catalog gave the model the raw title while the citation writer stripped brackets. Search titles routinely carry one ("[PDF] Annual Report"), and the prompt tells the model to copy the title verbatim, producing a label the validator cannot match. Both sides now share _citation_title. Verified: 756 passed across the research/web/sandbox/chat-history/rag backend suites. Each fix has a regression test that fails without it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep a durable run alive when no model is loaded for PR #7219 A durable run is claimable within the supervisor's poll interval of startup (main.py starts it in the lifespan, and claim_next takes any 'running' run whose lease expired), Studio has no startup model auto-load, and the browser is not connected yet. So restarting Studio mid-run reliably lands the next model call on the local endpoint's HTTP 400 "No model loaded". That 400 is not retryable: _completion retries only >= 500, and _stream_completion, which serves both planning and synthesis, has no retry at all. The run is marked failed, and the only recovery is retry, which sets report_text NULL and deletes every research_plan_step, research_source and research_document_source. Up to an hour of scraping and synthesis is lost on a plain restart, on the feature whose whole point is surviving one. Treat only that refusal as transient: wait up to the run's own modelTimeoutSeconds for a model to come back, then re-send. Any other 400 still fails immediately, so no behaviour changes on the happy path. The wait polls _check_active, so cancellation and lease loss are still honoured, and the model probe fails open, so a probe error can only send a request, never withhold one. Each wait is bounded by the run timeout and the number of waits per call is capped, so a model that keeps disappearing cannot re-send forever. Deliberately not pinning or restoring the model, which the review comment also suggested. Auto-switch is opt-in, default off, and GGUF-only, so restoring would silently evict the model the user just loaded from a background worker, and comparing the configured name to the loaded id is fragile across variant suffixes and advertised aliases, so it would break working runs. Verified: 853 passed across the research/web/sandbox/chat-history/rag/inference backend suites. Eight of the nine new tests fail without the fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make website-policy search reach the whole allowlist and refill past blocks for PR #7219 Two review findings on the website access policy. Domains past the site: filter cap were undiscoverable. The policy accepts up to 100 allowed domains and the prompt tells the model all of them are searchable, but scope_search_query always scoped to allowed[:8], so a source in the ninth or later domain could never be found, and an undiscovered URL cannot be fetched either. The cap itself is right, search engines stop honouring long OR chains, so the window now rotates by a hash of the query instead of being a fixed head. Every allowed domain is reachable across a multi-step run, the same query is always scoped the same way, and lists at or under the cap are unchanged. A page of blocked results returned nothing. The policy filters after the search while DDGS was asked for exactly max_results candidates, so if those happened to be disallowed the tool reported no results even when valid ones ranked just below, wasting a research step. Ask for a deeper pool when a policy is set and stop at max_results allowed entries. No policy means no over-fetch, so ordinary searches are unchanged. Verified: 2324 passed across the research/web/sandbox/chat-history/rag/tool backend suites. The 8 test_studio_api.py failures are pre-existing and need live OpenAI/Anthropic credentials; they fail identically with these changes stashed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only overfetch search results when the website policy restricts for PR #7219 Follow-up to8be0b3699. Every run stores normalize_website_policy(...), which returns {"allowedDomains": [], "blockedDomains": []} and is truthy even when nothing is restricted, so the default unrestricted path asked DDGS for four times as many results on every step. That is pure added latency and timeout risk, since the filter passes everything and only max_results entries are returned either way. Test the domain lists rather than the dict. * Budget the whole research prompt against the loaded context for PR #7219 Only the synthesis evidence was budgeted, so the budget could not prevent the overflow it existed to prevent. Measured at head with a realistic prompt (40-source catalog, 12-step plan): the untrimmable scaffolding is about 7,900 chars and the conversation context adds up to 12,000 more. On a 4096-token context, which is the GGUF auto-fit floor and the transformers default, the synthesis request came to about 1.7x the window. Worse, _synthesis_evidence_budget computed usable_tokens = 0 at or below the 4,096-token reserve and then returned the 1,500-char floor anyway, so it added evidence to a prompt that already did not fit. The decision prompt had no context awareness at all: a fixed evidence[-60000:], roughly ten times a small window, on every step rather than once at the end. Overflow is not cosmetic here. It either silently truncates and degenerates the report, as the comment above these constants already warned, or fails the run, and a failed run is only recoverable via retry, which deletes every plan step, source and document source and nulls the report. Both paths now share _prompt_char_budget plus _trimmable_budget: each trimmable section is measured against what the rest of the prompt leaves, and can reach 0 instead of a floor, because a shorter report beats a destroyed run. Evidence is budgeted before the chat history, since the evidence is the report. Unknown context still keeps the full cap. At 4096 tokens the synthesis prompt now fits (0.6x). Below that it is still over, since a 40-source catalog alone exceeds the window; that needs a smaller maxSources, and the context box does accept values down to 128. test_synthesis_evidence_budget_tracks_loaded_context asserted the old floor at 2048 tokens, which is the bug, so it now asserts 0 and that the rest of the prompt counts against the same budget. Verified: 2325 passed across the research/web/sandbox/chat-history/rag/tool suites. The test_mcp_stdio_sessions failure is pre-existing and fails identically with these changes stashed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope replayed research history to its own attempt for PR #7219 A retry deletes the previous attempt's research_plan_steps, research_sources and research_document_sources rows but keeps its events, and the SSE route attaches one live run snapshot to every event it emits, replayed history included. The step.completed payload carries only position, title, action, input and sourceCount, so that snapshot is the sole source of the excerpt and evidence. On any refresh after a retry, a replayed attempt-0 step was therefore matched against attempt-1's step row by position alone, and start_position resets to 0 after the delete, so the positions line up exactly. The preserved attempt-0 activity then showed attempt-1's excerpt and evidence, or lost them entirely when attempt 1 had not yet reached that position, under a banner that says previous activity is preserved. The run.started resumed branch read the same cross-attempt snapshot and spliced those activities out. Both are gated on the event's attempt matching the snapshot's retryCount, which is the same attempt scoping get_reasoning_text already applies server-side. The excerpt and evidence fall back to what the activity already holds, so a mismatch is non-destructive rather than blanking it. Verified: frontend contract 11 passed, tsc -b exit 0, and the new test fails without the store change. * Retry pre-stream failures in the research stream for PR #7219 _stream_completion serves planning, every decision step and synthesis, and it had no transport retry: a connection error or a 5xx raised before any response byte failed the durable run, and retry then deletes every gathered source, document source and plan step. _completion already treats the identical failures on the identical endpoint as retryable, so the two paths disagreed. This is partly a hole my own689b06535opened. After the no-model 400 the body is read, the connection returns to the pool, and _wait_for_local_model then sleeps for up to modelTimeoutSeconds before re-sending on the same client. Uvicorn's keep-alive is 5s, so that pooled connection is essentially always server-closed by then, and losing the has_expired race raises RemoteProtocolError, killing the run the wait existed to save. Also reachable via a read timeout waiting for headers under prompt-eval load. Retrying is safe only because nothing has been consumed at that point, and that is structural rather than a convention: with stream=True httpx returns on the response headers without calling aread(), and raise_for_status() reads no body, both verified against the installed 0.28.1. The handler is scoped to the inner try that ends at break, and _iter_stream_lines sits outside the loop with no path back to send, so a re-send cannot duplicate report text. Bounded and mirrors _completion: same >= 500 predicate, same 3 attempts, same 2**attempt backoff, lease and cancellation re-checked before re-sending. The transport counter and the model-wait counter are independent, so they cannot multiply. The response is closed before every re-send, as manual stream mode requires. Note HTTPStatusError is not a TransportError in httpx, so both are caught explicitly. Verified: 2330 passed. Five of the new tests fail without the fix; the three that pass either way are the invariants that must not change (fail fast on a real 400, never retry once the report has streamed, existing model-wait path). * Bound the planning prompt to the loaded context for PR #7219 Completesdc16598a4, which budgeted the decision and synthesis prompts but left planning unbounded. The question reaches the planner verbatim (a pasted document arrives here as-is) and the history is capped only at the fixed 12,000 chars, so on a small context planning could overflow before any plan was persisted, failing the run without doing any research at all. Same helpers as the other two paths. The question is budgeted before the history, since the question is the request. A test now asserts all three prompt paths hold their own context budget, so a fourth path cannot be added later without one. Verified: 2331 passed; the new test fails without the change. * Keep prompt inputs non-empty and fit the source catalog for PR #7219 Two follow-ups to the prompt budgeting, the first a regression I introduced indc16598a4. The output reserve was a flat 4096 tokens, so on any context at or below that, including the documented 4096-token GGUF floor, the whole prompt budget came out as 0. Every trimmable section then sliced to nothing: planning_question became the empty string, so the planner never saw the request at all, and synthesis dropped all its evidence. Removing the old floor outright went too far; an empty prompt is worse than the overflow it was avoiding. The reserve is now capped at half the window, and the question and the evidence each keep a floor, since one carries the request and the other carries the answer. A truncated completion is recoverable, a confidently empty report is not. The source catalog was the one section still inserted whole. It holds up to maxSources entries with snippets persisted at up to 4000 chars each, so on a smaller context it alone could exceed the budget while the code responded only by zeroing the evidence and history. It is now fitted first, dropping whole entries from the tail rather than slicing mid-entry, because a half-truncated URL is worse than an absent one: the validator would strip it and the claim would be left uncited. Verified: 2333 passed. All three new tests fail without the change; the question now keeps 1072 chars at a 2048-token context and 4144 at 4096, where both were previously 0. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten Deep Research comments for PR #7219 Post-convergence comment pass over the 40 source files in the PR diff, limited to lines the PR itself adds so untouched upstream code in the same files is left alone. 15 files, 110 insertions, 141 deletions. The reduction is deliberately small. Almost every comment here records why something non-obvious is done, a measured result, a spec rule, or the exact bug it prevents, and those are worth more than the lines they cost, so nearly every edit is a same-meaning compression rather than a deletion. Kept in full: the GFM autolink citation for the URL trim, the catastrophic-backtracking note on _DOCUMENT_CITATION, the prompt-budget notes recording that a reserve at or above the context leaves nothing, the two measured site: filter findings, and the remount note on the activity panel key. Verified comment-only three ways: comment_tools.py reports 15/15 code-unchanged, and an independent ast.dump comparison with docstrings stripped shows zero of the 12 Python files differing. 421 backend tests and the 11 frontend contract tests pass, and the phrase the contract test asserts on is still present on one line. * Harden Deep Research model streams * Fit Deep Research decision prompts * Preserve Deep Research follow-up context * Redact composite credentials from research queries * Scale Deep Research UI typography * Address Deep Research refinement review * Harden Deep Research refinement edge cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>
1271 lines
45 KiB
Python
1271 lines
45 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Main-content extraction and boilerplate stripping for the web fetch tool.
|
|
|
|
The HTML fixtures below snapshot the relevant fragments of a real GitHub repo
|
|
page (github.com/unslothai/unsloth, fetched 2026-07): the ``hidden``
|
|
client-side error placeholders ("Uh oh! There was an error while loading."),
|
|
the skip-link / nav / footer furniture, and the README rendered inside
|
|
``<article class="markdown-body">``. No network access is required.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
_BACKEND_DIR = str(Path(__file__).resolve().parent.parent)
|
|
if _BACKEND_DIR not in sys.path:
|
|
sys.path.insert(0, _BACKEND_DIR)
|
|
|
|
from core.inference._html_to_md import html_to_markdown
|
|
from core.inference.tools import (
|
|
_fetch_page_text,
|
|
_fetch_url_raw,
|
|
_github_repo_readme_api_url,
|
|
_looks_like_html,
|
|
)
|
|
|
|
|
|
# ── Fixtures: snapshot of GitHub repo page fragments ─────────────
|
|
|
|
# GitHub ships client-side error placeholders behind the `hidden` attribute (JS
|
|
# reveals them on a failed fetch); a text converter must not surface them.
|
|
_GITHUB_HIDDEN_ERROR_BLOCK = """
|
|
<div data-show-on-forbidden-error hidden>
|
|
<div class="Box">
|
|
<div class="blankslate-container">
|
|
<h3 class="blankslate-heading">Uh oh!</h3>
|
|
<p class="blankslate-description">
|
|
<p class="color-fg-muted my-2 mb-2 ws-normal">There was an error while loading.
|
|
<a class="Link--inTextBlock" href="" aria-label="Please reload this page">Please reload this page</a>.</p>
|
|
</p>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
"""
|
|
|
|
_GITHUB_PAGE = f"""<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head><title>unslothai/unsloth</title></head>
|
|
<body>
|
|
<a class="px-2 py-4" href="#start-of-content">Skip to content</a>
|
|
<header class="Header-old">
|
|
<div class="AppHeader-globalBar">
|
|
<a href="/login">Sign in</a>
|
|
<a href="/signup">Sign up</a>
|
|
</div>
|
|
</header>
|
|
<div class="js-notification-shelf"></div>
|
|
<div hidden>
|
|
You signed in with another tab or window. Reload to refresh your session.
|
|
You signed out in another tab or window. Reload to refresh your session.
|
|
You switched accounts on another tab or window. Reload to refresh your session.
|
|
Dismiss alert
|
|
</div>
|
|
<template>{{{{ message }}}}</template>
|
|
{_GITHUB_HIDDEN_ERROR_BLOCK}
|
|
<main id="js-repo-pjax-container">
|
|
{_GITHUB_HIDDEN_ERROR_BLOCK}
|
|
<div id="repository-container-header">
|
|
<a href="/unslothai">unslothai</a> / <a href="/unslothai/unsloth">unsloth</a>
|
|
<a href="/login?return_to=%2Funslothai%2Funsloth">Notifications</a>
|
|
You must be signed in to change notification settings
|
|
</div>
|
|
<div class="repository-content">
|
|
<table aria-labelledby="folders-and-files">
|
|
<tr><th>Name</th><th>Last commit message</th></tr>
|
|
<tr><td><a href="/unslothai/unsloth/tree/main/unsloth">unsloth</a></td><td></td></tr>
|
|
</table>
|
|
<article class="markdown-body entry-content container-lg" itemprop="text">
|
|
<h1>Unsloth Studio</h1>
|
|
<p>Unsloth Studio lets you run and train models locally. Fine-tune and
|
|
run LLMs on Windows, Linux and macOS with a single install command,
|
|
then export to GGUF, Ollama, vLLM or Hugging Face when you are done.</p>
|
|
<h2>Install</h2>
|
|
<pre>curl -fsSL https://unsloth.ai/install.sh | sh</pre>
|
|
<p>See the <a href="https://unsloth.ai/docs">documentation</a> for
|
|
quickstarts, notebooks, and fine-tuning guides for every major model
|
|
family including Llama, Gemma, Qwen and DeepSeek.</p>
|
|
</article>
|
|
</div>
|
|
<div class="Layout-sidebar">
|
|
<h2>Languages</h2>
|
|
<ul>
|
|
<li><a href="/unslothai/unsloth/search?l=javascript">JavaScript 89.3%</a></li>
|
|
<li><a href="/unslothai/unsloth/search?l=python">Python 9.7%</a></li>
|
|
</ul>
|
|
</div>
|
|
</main>
|
|
<footer>
|
|
<a href="https://docs.github.com">Docs</a>
|
|
<a href="https://github.com/contact">Contact</a>
|
|
</footer>
|
|
<div aria-live="polite" aria-hidden="true">You can't perform that action at this time.</div>
|
|
</body>
|
|
</html>
|
|
"""
|
|
|
|
|
|
# ── html_to_markdown: hidden elements ────────────────────────────
|
|
|
|
|
|
def test_hidden_attribute_subtree_is_dropped():
|
|
html = "<body><p>visible</p><div hidden><p>secret error text</p></div><p>after</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "visible" in out
|
|
assert "after" in out
|
|
assert "secret error text" not in out
|
|
|
|
|
|
def test_aria_hidden_true_subtree_is_dropped():
|
|
html = '<body><p>keep</p><span aria-hidden="true">decoration</span></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "decoration" not in out
|
|
|
|
|
|
def test_aria_hidden_false_subtree_is_kept():
|
|
html = '<body><span aria-hidden="false">still here</span></body>'
|
|
assert "still here" in html_to_markdown(html)
|
|
|
|
|
|
def test_inline_style_display_none_subtree_is_dropped():
|
|
# Error/loading blocks are often hidden with inline CSS rather than the
|
|
# ``hidden`` attribute; browsers do not render them, so they must not leak.
|
|
html = (
|
|
"<body><p>visible</p>"
|
|
'<div style="display:none">secret loading block</div>'
|
|
"<p>after</p></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "visible" in out
|
|
assert "after" in out
|
|
assert "secret loading block" not in out
|
|
|
|
|
|
def test_inline_style_visibility_hidden_subtree_is_dropped():
|
|
html = '<body><p>keep</p><span style="visibility:hidden">ghost</span></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "ghost" not in out
|
|
|
|
|
|
def test_inline_style_display_none_important_is_dropped():
|
|
# The !important flag must not defeat the display:none detection.
|
|
html = '<body><p>keep</p><div style="display:none !important">gone</div></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "gone" not in out
|
|
|
|
|
|
def test_inline_style_display_none_among_other_declarations():
|
|
html = (
|
|
"<body><p>keep</p>" '<div style="color: red; display : none ; margin:0">gone</div></body>'
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "gone" not in out
|
|
|
|
|
|
def test_inline_style_visible_display_is_kept():
|
|
# Over-strip guard: display:block / visibility:visible render, and a value or
|
|
# URL merely containing the substring "none" must not trigger the hidden path.
|
|
html = (
|
|
"<body>"
|
|
'<div style="display:block">block kept</div>'
|
|
'<div style="visibility:visible">visible kept</div>'
|
|
'<a style="background:url(none.png)">link kept</a>'
|
|
"</body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "block kept" in out
|
|
assert "visible kept" in out
|
|
assert "link kept" in out
|
|
|
|
|
|
def test_hidden_recovers_from_omitted_close_tags():
|
|
# <p hidden> is never closed; the parent </div> must still end the hidden region.
|
|
html = "<body><div><p hidden>gone</div><p>kept</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "gone" not in out
|
|
assert "kept" in out
|
|
|
|
|
|
def test_nested_hidden_regions():
|
|
html = "<body><div hidden><div hidden>inner</div>outer</div><p>ok</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "inner" not in out
|
|
assert "outer" not in out
|
|
assert "ok" in out
|
|
|
|
|
|
def test_hidden_false_is_still_hidden():
|
|
# ``hidden`` is enumerated: the spec maps invalid/empty values to the Hidden
|
|
# state, so hidden="false" is NOT rendered and must not reach the Markdown.
|
|
html = '<body><p>keep</p><div hidden="false">not rendered</div></body>'
|
|
out = html_to_markdown(html)
|
|
assert "keep" in out
|
|
assert "not rendered" not in out
|
|
|
|
|
|
def test_hidden_paragraph_omitted_close_does_not_swallow_siblings():
|
|
# HTML5 optional end tags: a sibling <p> start tag implicitly closes an open
|
|
# <p hidden>, so the hidden region ends there instead of swallowing siblings.
|
|
html = (
|
|
"<body><div><p hidden>secret"
|
|
"<p>visible one</p><p>visible two</p></div><p>after</p></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "visible one" in out
|
|
assert "visible two" in out
|
|
assert "after" in out
|
|
|
|
|
|
def test_hidden_list_item_omitted_close_keeps_following_items():
|
|
# <li hidden> without </li> is implicitly closed by the next <li>.
|
|
html = "<body><ul><li hidden>secret<li>shown A</li><li>shown B</li></ul></body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "shown A" in out
|
|
assert "shown B" in out
|
|
|
|
|
|
def test_hr_implicitly_closes_hidden_paragraph():
|
|
# Void elements also imply closes: <hr> ends an open <p hidden>.
|
|
html = "<body><p hidden>secret<hr>kept text</body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "kept text" in out
|
|
|
|
|
|
def test_skipped_tag_implicitly_closes_hidden_paragraph():
|
|
# A skipped block (<nav>/<footer>) also closes an open <p>. The optional-close
|
|
# bookkeeping must run before the skip, or the never-closed <p hidden> keeps its
|
|
# hidden mark and swallows every following sibling.
|
|
for skipped in ("nav", "footer"):
|
|
html = f"<body><p hidden>secret<{skipped}>chrome</{skipped}>VISIBLE</body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "chrome" not in out
|
|
assert "VISIBLE" in out
|
|
|
|
|
|
def test_hidden_void_element_is_suppressed():
|
|
# A hidden void element (<hr>/<br>) never joins the open-element stack, so it
|
|
# must be suppressed inline rather than emitting its markup.
|
|
html = '<body><p>before</p><hr aria-hidden="true"><p>after</p></body>'
|
|
out = html_to_markdown(html)
|
|
assert "before" in out
|
|
assert "after" in out
|
|
assert "---" not in out
|
|
|
|
|
|
def test_hidden_void_br_emits_no_break():
|
|
html = "<body><p>one<br hidden>two</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "one" in out
|
|
assert "two" in out
|
|
# The hidden <br> must not inject a newline between the two runs.
|
|
assert "one\ntwo" not in out
|
|
|
|
|
|
def test_visible_void_hr_still_renders():
|
|
# Guard: the suppression must not affect non-hidden void elements.
|
|
html = "<body><p>a</p><hr><p>b</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "---" in out
|
|
|
|
|
|
# ── html_to_markdown: main-content scoping ───────────────────────
|
|
|
|
|
|
def test_github_page_main_content_keeps_readme_only():
|
|
out = html_to_markdown(_GITHUB_PAGE, main_content = True)
|
|
# README content survives.
|
|
assert "Unsloth Studio" in out
|
|
assert "install.sh" in out
|
|
assert "documentation" in out
|
|
# Client-side error placeholders and page furniture are gone.
|
|
assert "Uh oh!" not in out
|
|
assert "There was an error while loading" not in out
|
|
assert "Please reload this page" not in out
|
|
assert "You can't perform that action at this time" not in out
|
|
assert "Skip to content" not in out
|
|
assert "Sign in" not in out
|
|
assert "Reload to refresh your session" not in out
|
|
assert "JavaScript 89.3%" not in out
|
|
assert "Languages" not in out
|
|
assert "Last commit message" not in out
|
|
|
|
|
|
def test_main_scope_used_when_no_article():
|
|
html = """
|
|
<body>
|
|
<header><a href="/login">Sign in</a></header>
|
|
<main><h1>Doc title</h1><p>%s</p></main>
|
|
<footer>footer junk</footer>
|
|
</body>
|
|
""" % ("Body text. " * 40)
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Doc title" in out
|
|
assert "Body text." in out
|
|
assert "Sign in" not in out
|
|
assert "footer junk" not in out
|
|
|
|
|
|
def test_main_content_falls_back_to_full_document():
|
|
# No article/main and a tiny body: the unscoped conversion is returned.
|
|
html = "<body><h1>Tiny</h1><p>Just a short page.</p></body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Tiny" in out
|
|
assert "Just a short page." in out
|
|
|
|
|
|
def test_tiny_article_stub_does_not_hijack_scope():
|
|
# An <article> with negligible text must not swallow the real content.
|
|
body_text = "Real content paragraph. " * 30
|
|
html = f"<body><article>ad</article><main><p>{body_text}</p></main></body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Real content paragraph." in out
|
|
|
|
|
|
def test_sibling_articles_do_not_leak_after_main_selected():
|
|
# The size gate picks the largest single <article> and renders only that
|
|
# subtree: sibling articles (related-post cards, comment threads) must not leak
|
|
# in just because the real article cleared the threshold.
|
|
real = "Main article body content for selection. " * 20
|
|
card = "Unrelated related-post card teaser blurb. " * 3
|
|
cards = "".join(f"<article><p>{card}</p></article>" for _ in range(5))
|
|
html = f"<body><article><h1>Real</h1><p>{real}</p></article>{cards}</body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Main article body content" in out
|
|
assert "Unrelated related-post" not in out
|
|
|
|
|
|
def test_default_conversion_unscoped_and_unstripped():
|
|
# Without main_content the whole document converts (backwards compatible),
|
|
# boilerplate included; only hidden subtrees are dropped.
|
|
html = "<body><p>Skip to content</p><div hidden>gone</div><main><p>hello</p></main></body>"
|
|
out = html_to_markdown(html)
|
|
assert "Skip to content" in out
|
|
assert "hello" in out
|
|
assert "gone" not in out
|
|
|
|
|
|
def test_boilerplate_filter_preserves_phrase_inside_real_prose():
|
|
# The furniture filter once matched by substring, deleting a real sentence that
|
|
# merely CONTAINS a fragment ("we use cookies"). It must drop only lines COMPOSED
|
|
# of furniture, keeping real prose that quotes one.
|
|
body = (
|
|
"<article><h1>Authentication</h1>"
|
|
"<p>We use cookies to authenticate API requests and keep sessions safe.</p>"
|
|
"<p>%s</p></article>"
|
|
) % ("Additional documentation content to select the article. " * 8)
|
|
out = html_to_markdown(f"<body>{body}</body>", main_content = True)
|
|
assert "We use cookies to authenticate API requests" in out
|
|
|
|
|
|
def test_boilerplate_filter_still_drops_standalone_and_stacked_furniture():
|
|
# A line that is purely furniture is dropped, as is one stacking several
|
|
# furniture phrases (as GitHub renders them).
|
|
body = (
|
|
"<article>"
|
|
"<p>Skip to content</p>"
|
|
"<p>You signed in with another tab or window. Reload to refresh your session.</p>"
|
|
"<p>Real README body. %s</p>"
|
|
"</article>"
|
|
) % ("Genuine documentation text. " * 8)
|
|
out = html_to_markdown(f"<body>{body}</body>", main_content = True)
|
|
assert "Real README body." in out
|
|
assert "Skip to content" not in out
|
|
assert "Reload to refresh your session" not in out
|
|
|
|
|
|
def test_boilerplate_not_stripped_inside_code_fences():
|
|
html = (
|
|
"<body><article><p>%s</p>"
|
|
"<pre>assert 'There was an error while loading' in page</pre>"
|
|
"</article></body>" % ("Prose. " * 40)
|
|
)
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "There was an error while loading" in out
|
|
|
|
|
|
def test_aside_callout_inside_article_is_kept():
|
|
# Docs render notes/warnings as <aside> callouts. An aside inside the selected
|
|
# article/main scope is real content and must survive; dropping it unconditionally
|
|
# loses page text.
|
|
body = (
|
|
"<article><h1>Guide</h1>"
|
|
"<p>%s</p>"
|
|
"<aside class='admonition warning'><strong>Warning:</strong> "
|
|
"This operation is destructive and cannot be undone.</aside>"
|
|
"<p>Trailing paragraph.</p></article>"
|
|
) % ("Documentation body text to select the article scope. " * 6)
|
|
out = html_to_markdown(f"<body>{body}</body>", main_content = True)
|
|
assert "This operation is destructive and cannot be undone." in out
|
|
assert "Warning:" in out
|
|
# Also kept in the unscoped (backwards-compatible) conversion.
|
|
out_full = html_to_markdown(f"<body>{body}</body>")
|
|
assert "This operation is destructive and cannot be undone." in out_full
|
|
|
|
|
|
# ── GitHub README rewrite ────────────────────────────────────────
|
|
|
|
|
|
def test_github_repo_url_maps_to_readme_api():
|
|
assert (
|
|
_github_repo_readme_api_url("https://github.com/unslothai/unsloth")
|
|
== "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
)
|
|
assert (
|
|
_github_repo_readme_api_url("https://github.com/unslothai/unsloth/")
|
|
== "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
)
|
|
assert (
|
|
_github_repo_readme_api_url("http://www.github.com/owner/repo.git")
|
|
== "https://api.github.com/repos/owner/repo/readme"
|
|
)
|
|
|
|
|
|
def test_github_non_repo_urls_are_not_rewritten():
|
|
for url in (
|
|
"https://github.com/unslothai/unsloth/tree/main/studio",
|
|
"https://github.com/unslothai/unsloth/issues/123",
|
|
"https://github.com/topics/llm",
|
|
"https://github.com/orgs/unslothai/repositories",
|
|
"https://github.com/login/oauth",
|
|
"https://github.com/unslothai",
|
|
"https://example.com/owner/repo",
|
|
"https://raw.githubusercontent.com/owner/repo/main/README.md",
|
|
):
|
|
assert _github_repo_readme_api_url(url) is None, url
|
|
|
|
|
|
def test_fetch_page_text_prefers_github_readme(monkeypatch):
|
|
calls = []
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
calls.append((url, extra_headers))
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, "# Unsloth\n\nFine-tune LLMs faster.", "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "Fine-tune LLMs faster." in out
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
assert len(calls) == 1
|
|
assert calls[0][1]["Accept"] == "application/vnd.github.raw+json"
|
|
|
|
|
|
def test_fetch_page_text_keeps_html_readme_from_api(monkeypatch):
|
|
# A repo whose README is HTML returns HTML from the README API with a 200. That
|
|
# success is authoritative: convert to Markdown and keep it, never discard it in
|
|
# favour of the repo root page's UI chrome.
|
|
html_readme = (
|
|
"<!doctype html><html><body>"
|
|
"<h1>Project Title</h1>"
|
|
"<p>Install with the one-line script and read the docs.</p>"
|
|
"</body></html>"
|
|
)
|
|
calls = []
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
calls.append(url)
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, html_readme, "text/html"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
# The successful README is converted and returned; no fallback fetch fires.
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
assert "Project Title" in out
|
|
assert "Install with the one-line script" in out
|
|
assert "<html" not in out
|
|
assert len(calls) == 1
|
|
|
|
|
|
def test_fetch_page_text_falls_back_to_html_when_readme_api_fails(monkeypatch):
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
if url.startswith("https://api.github.com/"):
|
|
return "Failed to fetch URL: HTTP 403 rate limited", "", ""
|
|
return None, _GITHUB_PAGE, "text/html"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
# Fallback converts the HTML page with the main-content heuristic.
|
|
assert "Unsloth Studio" in out
|
|
assert "Uh oh!" not in out
|
|
assert "There was an error while loading" not in out
|
|
|
|
|
|
def test_fetch_page_text_non_html_returned_raw(monkeypatch):
|
|
raw = "line one\n indented code\nline three"
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, raw, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://raw.githubusercontent.com/o/r/main/file.txt")
|
|
# Whitespace preserved: the HTML renderer would have collapsed it.
|
|
assert " indented code" in out
|
|
|
|
|
|
def test_fetch_page_text_html_conversion(monkeypatch):
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, _GITHUB_PAGE, "text/html"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth/tree/main")
|
|
assert "Unsloth Studio" in out
|
|
assert "Uh oh!" not in out
|
|
|
|
|
|
def test_fetch_page_text_propagates_fetch_errors(monkeypatch):
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return "Failed to fetch URL: HTTP 404 Not Found", "", ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
assert _fetch_page_text("https://example.com/missing") == (
|
|
"Failed to fetch URL: HTTP 404 Not Found"
|
|
)
|
|
|
|
|
|
def test_looks_like_html():
|
|
assert _looks_like_html("<!DOCTYPE html><html></html>")
|
|
assert _looks_like_html("\n <HTML lang='en'>")
|
|
assert not _looks_like_html("# Markdown README\n\n<h1>embedded html later</h1>")
|
|
assert not _looks_like_html("plain text")
|
|
|
|
|
|
def test_looks_like_html_markdown_with_leading_fenced_example_stays_markdown():
|
|
# A Markdown README OPENING with a fenced HTML example must not be sniffed as
|
|
# HTML just because a doctype/tag appears in the first 256 chars; html_to_markdown
|
|
# would corrupt the fences and prose.
|
|
fenced = (
|
|
"```html\n<!DOCTYPE html>\n<html><body><div>hi</div></body></html>\n```\n\n# Real README\n"
|
|
)
|
|
assert not _looks_like_html(fenced)
|
|
# Prose that mentions a tag inline, and a centered-logo README that opens
|
|
# with <p align>/<div align>/<h1 align>, also stay Markdown.
|
|
assert not _looks_like_html("Use the <html> element to start a page.")
|
|
assert not _looks_like_html('<p align="center"><img src="logo.png"></p>\n\n# Project\n')
|
|
assert not _looks_like_html('<div align="center">\n\n# Project\n\n</div>\n')
|
|
assert not _looks_like_html('<h1 align="center">Project</h1>\n\nMarkdown body.\n')
|
|
# An autolink is not a tag opener.
|
|
assert not _looks_like_html("<https://example.com> is the homepage")
|
|
|
|
|
|
def test_looks_like_html_detects_bare_fragments():
|
|
# A body that is a bare HTML fragment (no <html>/doctype) must still be
|
|
# recognized so it is converted to Markdown.
|
|
assert _looks_like_html("<body><p>hello</p></body>")
|
|
assert _looks_like_html("\n<article><h1>Title</h1><p>Body</p></article>")
|
|
assert _looks_like_html("<section>content</section>")
|
|
|
|
|
|
def test_looks_like_html_leading_table_stays_markdown():
|
|
# Markdown READMEs routinely open with a raw HTML <table> badge row or logo
|
|
# layout, then continue in Markdown. Sniffing that as HTML would collapse the
|
|
# Markdown body, so a leading <table> (and its row/cell children) must stay
|
|
# Markdown, like the excluded <div align>/<p align> layout headers.
|
|
assert not _looks_like_html("<table><tr><td>cell</td></tr></table>")
|
|
assert not _looks_like_html(
|
|
'<table align="center"><tr><td><img src="logo.png"></td></tr></table>\n\n# Project\n'
|
|
)
|
|
assert not _looks_like_html("<tr><td>cell</td></tr>")
|
|
|
|
|
|
def test_fetch_page_text_keeps_markdown_readme_with_html_example(monkeypatch):
|
|
# A Markdown README opening with a fenced HTML snippet must be served verbatim,
|
|
# never run through html_to_markdown (which would drop the fences/tags).
|
|
md_readme = (
|
|
"```html\n"
|
|
"<!DOCTYPE html>\n"
|
|
"<html><body><h1>Demo</h1></body></html>\n"
|
|
"```\n\n"
|
|
"# My Project\n\nInstall and run.\n"
|
|
)
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, md_readme, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
# Markdown preserved verbatim: the fence and literal tags survive.
|
|
assert "```html" in out
|
|
assert "<!DOCTYPE html>" in out
|
|
assert "# My Project" in out
|
|
|
|
|
|
def test_fetch_page_text_keeps_markdown_readme_with_leading_table(monkeypatch):
|
|
# A README opening with a raw HTML <table> badge/layout row then continuing in
|
|
# Markdown must be served verbatim, never run through html_to_markdown (which
|
|
# would collapse the list/fence/heading body onto one line).
|
|
md_readme = (
|
|
'<table align="center">\n'
|
|
'<tr><td><img src="logo.png"></td><td>Badges</td></tr>\n'
|
|
"</table>\n\n"
|
|
"# My Project\n\n"
|
|
"- feature one\n"
|
|
"- feature two\n\n"
|
|
"```python\nprint('hi')\n```\n"
|
|
)
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, md_readme, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
# Markdown body verbatim: list, fence and heading survive on their own lines.
|
|
assert "- feature one\n- feature two" in out
|
|
assert "```python" in out
|
|
assert "# My Project" in out
|
|
|
|
|
|
def test_fetch_url_raw_missing_content_type_reported_empty(monkeypatch):
|
|
# Message.get_content_type() falls back to the RFC 2045 "text/plain" default
|
|
# when the header is absent; _fetch_url_raw must report "" instead so the HTML
|
|
# sniffing fallback can fire.
|
|
import email
|
|
import urllib.request
|
|
|
|
class _FakeResp:
|
|
headers = email.message_from_string("")
|
|
|
|
def __init__(self):
|
|
self._body = b"<html><body>hello</body></html>"
|
|
|
|
def read(self, n = -1):
|
|
# Hand back the body once, then EOF, so the chunked reader terminates.
|
|
body, self._body = self._body, b""
|
|
return body
|
|
|
|
class _FakeOpener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
return _FakeResp()
|
|
|
|
monkeypatch.setattr(
|
|
"core.inference.tools._validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _FakeOpener())
|
|
err, body, content_type = _fetch_url_raw("https://example.com/")
|
|
assert err is None
|
|
assert "hello" in body
|
|
assert content_type == ""
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"disable_dns_pinning,expected_url",
|
|
[
|
|
(False, "https://203.0.113.7:8443/page?q=1"),
|
|
(True, "https://example.com:8443/page?q=1"),
|
|
],
|
|
)
|
|
def test_fetch_url_raw_dns_pinning_proxy_opt_out(monkeypatch, disable_dns_pinning, expected_url):
|
|
import email
|
|
import urllib.request
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
class _FakeResp:
|
|
headers = email.message_from_string("Content-Type: text/plain\n")
|
|
|
|
def __init__(self):
|
|
self._body = b"ok"
|
|
|
|
def read(self, n = -1):
|
|
body, self._body = self._body, b""
|
|
return body
|
|
|
|
requested = []
|
|
|
|
class _FakeOpener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
requested.append(req)
|
|
return _FakeResp()
|
|
|
|
resolved = []
|
|
|
|
def resolve(host, port):
|
|
resolved.append((host, port))
|
|
return True, "", "203.0.113.7"
|
|
|
|
monkeypatch.setenv("UNSLOTH_STUDIO_DISABLE_DNS_PINNING", "1" if disable_dns_pinning else "0")
|
|
monkeypatch.setattr(tools_mod, "_validate_and_resolve_host", resolve)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _FakeOpener())
|
|
|
|
# No embedded credentials: the web access policy rejects those outright
|
|
# (see test_fetch_url_raw_rejects_embedded_credentials).
|
|
err, body, _content_type = tools_mod._fetch_url_raw("https://example.com:8443/page?q=1")
|
|
|
|
assert err is None
|
|
assert body == "ok"
|
|
assert resolved == [("example.com", 8443)]
|
|
assert [req.full_url for req in requested] == [expected_url]
|
|
assert requested[0].get_header("Host") == "example.com:8443"
|
|
|
|
|
|
def test_fetch_url_raw_rejects_embedded_credentials(monkeypatch):
|
|
# Credentials in the URL are blocked rather than stripped, so they can never
|
|
# leak to a redirect target or into logs.
|
|
import core.inference.tools as tools_mod
|
|
|
|
def resolve(host, port):
|
|
raise AssertionError("must be rejected before DNS resolution")
|
|
|
|
monkeypatch.setattr(tools_mod, "_validate_and_resolve_host", resolve)
|
|
|
|
err, body, _content_type = tools_mod._fetch_url_raw(
|
|
"https://user:secret@example.com:8443/page?q=1"
|
|
)
|
|
|
|
assert err is not None and "credentials" in err
|
|
assert body == ""
|
|
|
|
|
|
def test_fetch_page_text_missing_content_type_html_sniffed(monkeypatch):
|
|
# A header-less server returning an HTML body must still be converted.
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, _GITHUB_PAGE, ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/no-content-type")
|
|
assert "Unsloth Studio" in out
|
|
assert "<html" not in out
|
|
assert "Uh oh!" not in out
|
|
|
|
|
|
def test_fetch_page_text_missing_content_type_fragment_converted(monkeypatch):
|
|
# A header-less server returning a bare HTML fragment (no <html>/doctype) must
|
|
# still be sniffed as HTML and converted, not served as raw markup.
|
|
fragment = "<article><h1>Doc Title</h1><p>Readable fragment body.</p></article>"
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, fragment, ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/fragment")
|
|
assert "Doc Title" in out
|
|
assert "Readable fragment body." in out
|
|
assert "<article" not in out
|
|
|
|
|
|
def test_fetch_page_text_missing_content_type_plain_text_raw(monkeypatch):
|
|
# A header-less server returning plain text stays raw (whitespace kept).
|
|
raw = "line one\n indented code\nline three"
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, raw, ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/no-content-type.txt")
|
|
assert " indented code" in out
|
|
|
|
|
|
def test_fetch_page_text_mislabeled_text_plain_html_converted(monkeypatch):
|
|
# An explicit text/plain header on an HTML body is sniffed and converted, like
|
|
# the pre-extraction behavior of always converting HTML pages.
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
return None, _GITHUB_PAGE, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://example.com/mislabeled")
|
|
assert "Unsloth Studio" in out
|
|
assert "<html" not in out
|
|
|
|
|
|
# ── implicit-close past unclosed inline descendants (finding 14) ──
|
|
|
|
|
|
def test_hidden_paragraph_with_inline_child_implicitly_closed_by_block():
|
|
# A browser closes an open <p> when a <div> arrives, even with an unclosed
|
|
# <span> on top of it. The hidden region must end there, not swallow the
|
|
# following visible blocks.
|
|
html = "<body><p hidden><span>secret<div>visible div</div><p>visible paragraph</body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "visible div" in out
|
|
assert "visible paragraph" in out
|
|
|
|
|
|
def test_hidden_list_item_with_inline_child_closed_by_next_item():
|
|
html = "<body><ul><li hidden><span>secret<li>visible item</ul><p>after</p></body>"
|
|
out = html_to_markdown(html)
|
|
assert "secret" not in out
|
|
assert "visible item" in out
|
|
assert "after" in out
|
|
|
|
|
|
# ── nested hidden list/table contents must stay suppressed ──
|
|
|
|
|
|
def test_nested_hidden_list_does_not_leak_child_items():
|
|
# The nested <ul> re-scopes the item, so the inner <li> is a DESCENDANT of the
|
|
# hidden outer <li>, not an optional-close sibling. Optional-end-tag recovery
|
|
# must not cross the intervening <ul>, or the outer li's hidden mark is popped
|
|
# and the nested text leaks.
|
|
html = (
|
|
"<body><ul>"
|
|
"<li hidden>parent<ul><li>secret child</li></ul></li>"
|
|
"<li>visible sibling</li>"
|
|
"</ul></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "parent" not in out
|
|
assert "secret child" not in out
|
|
assert "visible sibling" in out
|
|
|
|
|
|
def test_nested_hidden_list_with_omitted_closes_stays_suppressed():
|
|
# Same leak, doubly nested with omitted </li>/</ul>. Every hidden descendant
|
|
# stays gone; the following visible sibling (which implicitly closes the hidden
|
|
# outer <li>) still renders.
|
|
html = (
|
|
"<body><ul>"
|
|
"<li hidden>parent<ul><li>secret child<ul><li>deeper secret</ul></li></ul>"
|
|
"<li>visible sibling"
|
|
"</ul></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "parent" not in out
|
|
assert "secret child" not in out
|
|
assert "deeper secret" not in out
|
|
assert "visible sibling" in out
|
|
|
|
|
|
def test_nested_hidden_table_does_not_leak_inner_cells():
|
|
# A nested <table> re-scopes <tr>/<td>: an inner <td> must not be an
|
|
# optional-close sibling of a hidden outer <td> across the nested table.
|
|
html = (
|
|
"<body><table><tr>"
|
|
"<td hidden>outer<table><tr><td>secret cell</td></tr></table></td>"
|
|
"<td>visible cell</td>"
|
|
"</tr></table></body>"
|
|
)
|
|
out = html_to_markdown(html)
|
|
assert "secret cell" not in out
|
|
assert "visible cell" in out
|
|
|
|
|
|
# ── aggregate tiny <article> cards must not displace <main> (finding 15) ──
|
|
|
|
|
|
def test_many_tiny_articles_do_not_displace_substantial_main():
|
|
cards = "".join(
|
|
f"<article><h2>Teaser {i}</h2><p>Advertisement card blurb.</p></article>" for i in range(12)
|
|
)
|
|
main_body = "Authoritative main documentation content. " * 30
|
|
html = f"<body>{cards}<main><h1>Real page</h1><p>{main_body}</p></main></body>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Authoritative main documentation content." in out
|
|
assert "Advertisement card blurb." not in out
|
|
|
|
|
|
def test_single_substantial_article_still_preferred_over_main():
|
|
# GitHub-README case: one substantial <article> inside <main> must still win
|
|
# over sibling <main> furniture.
|
|
article_body = "Real README documentation body text. " * 20
|
|
html = (
|
|
"<body><main>"
|
|
f"<article><h1>Guide</h1><p>{article_body}</p></article>"
|
|
"<div><h2>Languages</h2><p>JavaScript 89.3%</p></div>"
|
|
"</main></body>"
|
|
)
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Real README documentation body text." in out
|
|
assert "JavaScript 89.3%" not in out
|
|
|
|
|
|
# ── truncated (unclosed) main-content scopes must still be scored ──
|
|
|
|
|
|
def test_truncated_open_article_scope_is_scored_and_preferred():
|
|
# _fetch_url_raw caps large pages, so the download can end before the closing
|
|
# </article>. The scope is still the main content and must be preferred over the
|
|
# whole document (which re-leaks the page chrome).
|
|
chrome = "<nav>Skip to content</nav><div>Repository file tree and page chrome.</div>"
|
|
article_body = "Real README documentation body text. " * 20
|
|
# No closing </article> / </body> -- the fetch cap truncated the page.
|
|
html = f"<body>{chrome}<article><h1>Guide</h1><p>{article_body}</p>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Real README documentation body text." in out
|
|
assert "Repository file tree and page chrome." not in out
|
|
|
|
|
|
def test_truncated_open_main_scope_is_scored_and_preferred():
|
|
chrome = "<nav>Skip to content</nav><div>Repository file tree and page chrome.</div>"
|
|
main_body = "Authoritative main documentation content. " * 30
|
|
html = f"<body>{chrome}<main><h1>Doc</h1><p>{main_body}</p>"
|
|
out = html_to_markdown(html, main_content = True)
|
|
assert "Authoritative main documentation content." in out
|
|
assert "Repository file tree and page chrome." not in out
|
|
|
|
|
|
# ── overall fetch deadline + cancellation (no per-hop timeout blowup) ──
|
|
|
|
|
|
def test_fetch_url_raw_overall_deadline_aborts_across_redirects(monkeypatch):
|
|
# Each hop advances a fake clock by 5s; an 8s overall budget is exhausted on the
|
|
# third hop even though every hop stays within its own socket timeout. Without
|
|
# the deadline this would redirect until the 5-hop cap, so the "timed out" error
|
|
# proves the overall budget aborted it, not the hop cap.
|
|
import urllib.request
|
|
from urllib.error import HTTPError
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
clock = {"t": 1000.0}
|
|
monkeypatch.setattr(tools_mod.time, "monotonic", lambda: clock["t"])
|
|
|
|
hops = {"n": 0}
|
|
|
|
class _RedirectingOpener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
clock["t"] += 5.0
|
|
hops["n"] += 1
|
|
raise HTTPError(
|
|
req.full_url,
|
|
302,
|
|
"Found",
|
|
{"Location": "https://example.com/next"},
|
|
None,
|
|
)
|
|
|
|
monkeypatch.setattr(
|
|
tools_mod,
|
|
"_validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _RedirectingOpener())
|
|
|
|
err, body, content_type = tools_mod._fetch_url_raw(
|
|
"https://example.com/start",
|
|
timeout = 30,
|
|
deadline = clock["t"] + 8.0,
|
|
)
|
|
assert err == "Failed to fetch URL: timed out."
|
|
assert body == ""
|
|
assert hops["n"] < 5
|
|
|
|
|
|
def test_fetch_url_raw_cancel_event_aborts_before_network(monkeypatch):
|
|
# A set cancel_event (client disconnected) stops the fetch before it opens any
|
|
# socket, so a dropped stream cannot leave a tool blocking on the wire.
|
|
import threading
|
|
import urllib.request
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
ev = threading.Event()
|
|
ev.set()
|
|
opened = {"n": 0}
|
|
|
|
class _Opener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
opened["n"] += 1
|
|
raise AssertionError("network must not be touched after cancel")
|
|
|
|
monkeypatch.setattr(
|
|
tools_mod,
|
|
"_validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _Opener())
|
|
|
|
err, body, content_type = tools_mod._fetch_url_raw(
|
|
"https://example.com/",
|
|
cancel_event = ev,
|
|
)
|
|
assert err == "Failed to fetch URL: cancelled."
|
|
assert opened["n"] == 0
|
|
|
|
|
|
def test_fetch_page_text_shares_one_deadline_across_readme_and_fallback(monkeypatch):
|
|
# The README API attempt and its HTML fallback must draw from ONE budget: a
|
|
# failed API call cannot hand the fallback a fresh full timeout.
|
|
seen_deadlines = []
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
seen_deadlines.append(deadline)
|
|
# Fail the README API so the HTML fallback also runs.
|
|
return "Failed to fetch URL: HTTP 429 rate limited", "", ""
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth", timeout = 30)
|
|
assert out == "Failed to fetch URL: HTTP 429 rate limited"
|
|
# Both attempts ran and shared the same, single deadline value.
|
|
assert len(seen_deadlines) == 2
|
|
assert seen_deadlines[0] is not None
|
|
assert seen_deadlines[0] == seen_deadlines[1]
|
|
|
|
|
|
# -- overall deadline reaches the body read, the resolver, and the query path --
|
|
|
|
|
|
def test_fetch_url_raw_deadline_aborts_slow_body(monkeypatch):
|
|
# A server dribbling the body must not stretch the read past the overall
|
|
# deadline: the body is read in chunks with the budget re-checked between them,
|
|
# so a single slow resp.read cannot outlast the fetch budget.
|
|
import email
|
|
import urllib.request
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
clock = {"t": 1000.0}
|
|
monkeypatch.setattr(tools_mod.time, "monotonic", lambda: clock["t"])
|
|
|
|
class _DrippingResp:
|
|
headers = email.message_from_string("")
|
|
|
|
def read(self, n = -1):
|
|
# One chunk, then jump the clock past the deadline so the next
|
|
# between-chunk budget check aborts instead of reading forever.
|
|
clock["t"] += 10.0
|
|
return b"x" * 16
|
|
|
|
def close(self):
|
|
pass
|
|
|
|
class _Opener:
|
|
def open(
|
|
self,
|
|
req,
|
|
timeout = None,
|
|
):
|
|
return _DrippingResp()
|
|
|
|
monkeypatch.setattr(
|
|
tools_mod,
|
|
"_validate_and_resolve_host",
|
|
lambda host, port: (True, "", "203.0.113.7"),
|
|
)
|
|
monkeypatch.setattr(urllib.request, "build_opener", lambda *handlers: _Opener())
|
|
|
|
err, body, content_type = tools_mod._fetch_url_raw(
|
|
"https://example.com/",
|
|
timeout = 30,
|
|
deadline = clock["t"] + 5.0,
|
|
)
|
|
assert err == "Failed to fetch URL: timed out."
|
|
assert body == ""
|
|
|
|
|
|
def test_resolve_with_budget_aborts_on_slow_resolver(monkeypatch):
|
|
# getaddrinfo has no deadline of its own; a resolver slower than the budget must
|
|
# abort on time instead of blocking the whole fetch.
|
|
import threading
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
clock = {"t": 1000.0}
|
|
monkeypatch.setattr(tools_mod.time, "monotonic", lambda: clock["t"])
|
|
|
|
release = threading.Event()
|
|
|
|
def slow_resolve(host, port):
|
|
release.wait(5.0) # block until released; the budget should abort first
|
|
return True, "", "203.0.113.7"
|
|
|
|
monkeypatch.setattr(tools_mod, "_validate_and_resolve_host", slow_resolve)
|
|
|
|
def advance_past_deadline():
|
|
import time as _t
|
|
_t.sleep(0.1)
|
|
clock["t"] += 100.0
|
|
|
|
t = threading.Thread(target = advance_past_deadline, daemon = True)
|
|
t.start()
|
|
try:
|
|
ok, reason, ip = tools_mod._resolve_with_budget(
|
|
"example.com",
|
|
443,
|
|
1005.0,
|
|
None,
|
|
)
|
|
finally:
|
|
release.set()
|
|
assert ok is False
|
|
assert reason == "Failed to fetch URL: timed out."
|
|
|
|
|
|
def test_web_search_query_cancelled_skips_search(monkeypatch):
|
|
# A pre-set cancel_event (client disconnected) skips the blocking DDGS query,
|
|
# matching the direct-URL path's cancellation.
|
|
import sys
|
|
import threading
|
|
import types
|
|
|
|
import core.inference.tools as tools_mod
|
|
|
|
ev = threading.Event()
|
|
ev.set()
|
|
called = {"n": 0}
|
|
|
|
class _DDGS:
|
|
def __init__(self, *a, **k):
|
|
called["n"] += 1
|
|
|
|
def text(self, *a, **k):
|
|
called["n"] += 1
|
|
return []
|
|
|
|
fake_mod = types.ModuleType("ddgs")
|
|
fake_mod.DDGS = _DDGS
|
|
monkeypatch.setitem(sys.modules, "ddgs", fake_mod)
|
|
|
|
out = tools_mod._web_search("some query", cancel_event = ev)
|
|
assert out == "Search cancelled."
|
|
assert called["n"] == 0
|
|
|
|
|
|
def test_fetch_page_text_markdown_readme_with_leading_block_tag_stays_markdown(monkeypatch):
|
|
# A raw-Markdown README that OPENS with an HTML block tag (<blockquote>, <ul>,
|
|
# <pre>, ...) must not be run through html_to_markdown, which would collapse its
|
|
# headings/list/fence. Only a real HTML document (doctype / <html>) is converted.
|
|
md_readme = (
|
|
"<blockquote>Note: pre-release.</blockquote>\n\n"
|
|
"# My Project\n\n"
|
|
"Install:\n\n"
|
|
"- step one\n"
|
|
"- step two\n\n"
|
|
"```bash\npip install myproject\n```\n"
|
|
)
|
|
|
|
def fake_fetch(
|
|
url,
|
|
timeout = 30,
|
|
extra_headers = None,
|
|
deadline = None,
|
|
cancel_event = None,
|
|
):
|
|
assert url == "https://api.github.com/repos/unslothai/unsloth/readme"
|
|
return None, md_readme, "text/plain"
|
|
|
|
monkeypatch.setattr("core.inference.tools._fetch_url_raw", fake_fetch)
|
|
out = _fetch_page_text("https://github.com/unslothai/unsloth")
|
|
assert "README of https://github.com/unslothai/unsloth" in out
|
|
# Markdown structure survives verbatim (heading, list, fenced code).
|
|
assert "# My Project" in out
|
|
assert "- step one" in out
|
|
assert "```bash" in out
|
|
|
|
|
|
def test_looks_like_html_document_only_matches_real_documents():
|
|
from core.inference.tools import _looks_like_html_document
|
|
|
|
assert _looks_like_html_document("<!doctype html><html><body>x</body></html>")
|
|
assert _looks_like_html_document("\n <HTML lang='en'>")
|
|
assert _looks_like_html_document("<body><h1>x</h1></body>")
|
|
# Block tags a Markdown README can open with are NOT full documents.
|
|
for frag in (
|
|
"<blockquote>q</blockquote>",
|
|
"<ul><li>x</li></ul>",
|
|
"<pre>x</pre>",
|
|
"<dl><dt>x</dt></dl>",
|
|
):
|
|
assert not _looks_like_html_document(frag), frag
|