- _has_unclosed_code_fence() ignores a fence run when the trailing
text on the same line starts with a space (typical English prose
like "Use \`\`\` to start a markdown fence."). Real fence openers
either end the line right after the delimiters or carry an info
string with no leading space (\`\`\`python, \`\`\`bash-session).
- _DIRECT_NUMBERED_PLAN_FRAMING accepts "take these steps",
"follow these steps", and "perform these actions" as first-person
intent verbs. Plans like "I'll take these steps:\n1. Open URL\n
2. Read" still re-prompt instead of being read as final answers.
- Re-prompt path calls _strip_tool_markup(final=True) on content_accum
before measuring intent / artifact / length. An orphan
``<tool_call>...</tool_call>`` block containing a code fence no
longer hides the intent-only visible answer from the artifact check.
- _DIRECT_NUMBERED_PLAN_FRAMING splits into two branches:
* First-person intent ("I'll", "Let me", "I will", etc.) accepts a
broader work-verb set (open, read, search, check, review, inspect,
examine, etc.). Direct first-person announcements are strong
plan-like signals.
* Bare "First, ..." / "Step N: ..." keeps the narrow verb set so
algorithmic answers ("First, use binary search:") stay valid.
Catches stalls like "I will check the docs:\n1. Gather..." and
"Let me read the uploaded file:\n1. Identify the columns..." that
previously slipped past the freshness-gated lookup verbs.
When a real complete artifact is already in the response, prose
mentions of bare <html> / <svg> tags in explanatory text are common
(for example "Use the <html> tag for the root"). The unbalanced-
open/close count would falsely classify the response as mid-stream
and wipe the valid answer. The artifact-counting cross-check now
only runs when NO real artifact has been emitted yet; once a real
artifact exists, mid-stream second markup is rare enough that the
count-based detector is not worth the false-positive cost.
This also unblocks complete <html> answers that nest <svg> children
or contain JS string literals like "<svg width=10>", since those
unmatched markup tokens were being flagged as unclosed.
- _has_answer_artifact() now strips closed code fences before checking
for unclosed markup, and strips closed markup before checking for
unclosed code fences. A Python / JS snippet containing literal
"<html>" / "<svg>" strings no longer trips the unclosed-markup
cross-check, and complete HTML containing a JS string with literal
backticks no longer trips the unclosed-fence cross-check.
- _looks_like_real_artifact() iterates every artifact match. An empty
<html></html> / <svg></svg> skeleton followed by a real complete
page no longer hides the real artifact.
- _is_empty_markup_skeleton() strips an optional <!doctype ...> prefix
before testing the empty-skeleton pattern, so
"<!doctype html><html></html>" plan-only mentions also re-prompt.
- _BARE_INTENT_NUMBERED_PLAN catches the tight "I'll:\n1. Open ..." /
"Let me:\n1. Parse ..." shape where bare first-person intent +
colon + newline is immediately followed by numbered action items.
No work verb is required between the intent and the list.
- _has_unclosed_markup_block() now compares open / close tag counts.
A response with one closed <html> followed by a second still-open
<html> (multi-page mid-stream) or <svg></svg><svg> is unbalanced,
so the artifact path returns False and the re-prompt fires. The
helper now runs BEFORE _HAS_ANSWER_ARTIFACT so an earlier complete
artifact cannot mask a later open block.
- _looks_like_real_artifact() rejects empty <html></html> /
<svg></svg> skeletons. Plan-only mentions ("First, I'll create an
<html></html> skeleton, then add CSS.") no longer suppress the
re-prompt.
- _NUMBERED_ACTION_ITEM + _STRONG_INTENT_BEFORE_LIST catches plans
where the work verbs sit in the list ITEMS rather than before the
list (e.g. "First, I'll:\n1. Load the CSV.\n2. Compute total").
The verb whitelist is intentionally narrow (load, parse, calculate,
compute, analyze, run, execute, fetch, download, query, inspect,
extract) so ordinary algorithm answers ("First, use binary search:
1. Search the left half") stay valid. The intent gate excludes
bare "First" / "Step N:" for the same reason - direct first-person
pronoun is required.
- _has_unclosed_markup_block() short-circuits the numbered-list fallback
when the response contains an open <html> or <svg> with no matching
close. A partial markup body that happens to contain two numbered
lines no longer reads as a final answer.
- _TOOL_ACTION_VERBS adds freshness-gated "compare" and "review" so
plans phrased as "Compare the latest release sources" or "Review
the current documentation" still re-prompt.
- Re-prompt call site defers the visible-artifact regex scan until
the cheap gates (tools enabled, _reprompt_count, length window,
intent regex) have all passed. Long final answers that can never
re-prompt no longer pay the artifact-scan cost.
- _has_unclosed_code_fence() now scans every line with re.search and a
shared FENCE_RUN regex, so an inline opening fence such as
"First, let me write it. \`\`\`python" is tracked alongside the
column-0 openers. A numbered list emitted INSIDE an inline-open
fence no longer reads as a final answer.
- _DIRECT_NUMBERED_PLAN_FRAMING adds "first" and "step N(:?)" to its
intent prefixes and "look up" to its verb whitelist. Plans like
"First, analyze the uploaded CSV:\n1. Load rows\n2. Compute total"
or "I'll look that up:\n1. Search the docs" now re-prompt instead
of being mis-classified as final answers. The verb whitelist still
excludes bare search/find/check/verify so "First, use binary
search:\n1. Search the left half" stays an answer.
- _DIRECT_NUMBERED_PLAN_FRAMING matches first-person intent ("I'll",
"Let me", etc.) plus a narrow follow-up verb ("do this", "do these",
"create", "build", "set up", "calculate", "parse", "run", etc.)
followed by a numbered list. This catches stalls like "First, I'll
do this:\n1. Search for X." or "Let me do this:\n1. Parse the
JSON.\n2. Calculate the average." where the model announces actions
but never invokes a tool. The verb whitelist stays narrow so
"Let me explain" / "Let me show" / "Let me draft a poem" answers
are NOT misclassified.
- _has_answer_artifact() now checks for an unclosed code fence BEFORE
consulting _HAS_ANSWER_ARTIFACT. A response with one complete fence
followed by a second, still-open fence (mid-stream multi-file
answers) no longer suppresses the re-prompt; the unclosed second
fence wins.
- _HAS_ANSWER_ARTIFACT closing fence now accepts strictly more delimiters
than the opener (CommonMark rule). The opener stays anchored on both
sides so a 4-open / 3-close payload still does not match, but a
legitimate 3-open / 4-close (and 3-tilde / 4-tilde) answer is now
recognised as a completed artifact.
- _EXPLICIT_PLAN_HEADER triggers the plan classification by itself when
the response contains \"Here's my plan\" / \"Here's my approach\" /
\"Here's the plan\". Numbered stalls like \"Here's my plan:\n1. Analyze\n
2. Draft\" re-prompt again without needing a freshness-gated verb.
Plain \"Plan:\" / \"My weekly plan:\" stay valid answers because they
lack the possessive first-person header.
- _TOOL_ACTION_VERBS adds \"use python (tool) to ...\", \"use the python
tool\", \"invoke the python tool\", and \"use the search tool\" so
numbered plans that route through these phrasings still re-prompt.
- _TOOL_ACTION_VERBS gates the lookup verbs (search / look up /
browse / google / fetch / research / investigate / find / check /
verify) on a freshness or web/internet/online target. Plain answer
prose like \"binary search: 1. Search the left half\" or \"1. Find
the bug\" stays a valid answer, while \"1. Search the web for X\"
/ \"1. Google the current chart\" / \"1. Research the latest docs\"
still re-prompts. Strong unambiguous patterns (web search, query
the web, call a tool, run python) remain bare.
- _HAS_ANSWER_ARTIFACT anchors the fence opener and closer with
(?<!\\`) / (?!\\`) lookarounds so a 4-backtick opener cannot
backtrack to a 3-backtick fence and treat the surplus delimiter as
info-string text. Same rule for tildes.
- _has_answer_artifact now consults a small _has_unclosed_code_fence
helper before the numbered-list fallback. A numbered list embedded
INSIDE an open fence no longer masquerades as a final answer.
- Existing plan-framing tests updated to use freshness-gated lookup
phrasing so they continue to assert the intended invariants.
- _HAS_ANSWER_ARTIFACT now matches fences with three OR MORE backticks
/ tildes using a named-group backreference (CommonMark rule). Models
routinely emit \`\`\`\` / \`\`\`\`\` when the body itself contains a triple
fence. The previous regex only matched exactly three.
- _TOOL_ACTION_VERBS adds \"query / consult the web / internet / online
sources\" so numbered plan stalls phrased with these synonyms still
re-prompt instead of being read as final answers.
- _PLAN_LIST_FRAMING widens the intent-to-action scan from 80 chars to
the full short candidate (caller already gates at _REPROMPT_MAX_CHARS
= 2000). Realistic plans where item 1 is preamble and item 2 is the
explicit tool action no longer slip through.
- Re-prompt call site separates VISIBLE-content artifact check from
hidden reasoning. When content_accum is empty AND has_content_tokens
is False, reasoning_accum is the user-visible text and counts for
the artifact check. Otherwise reasoning stays hidden and an artifact
inside it must not suppress the re-prompt.
- Re-prompt path now treats a closed artifact in hidden reasoning as
no artifact for the user; only visible content_accum counts. Stops
hidden chain-of-thought from suppressing the tool-forcing nudge
when content_accum is empty.
- Closed backtick / tilde fences must end the line cleanly. Trailing
prose after the closing fence (```not actually closed) no longer
reads as a complete artifact.
- _TOOL_ACTION_VERBS admits find / check / verify only when paired
with a freshness signal (current / latest / today / up-to-date /
live / online / web). Numbered plan stalls like \"1. Find the
current Billboard chart\" re-prompt again, while \"1. Find the
bug\" / \"2. Check the answer\" stay valid answer text.
Reviewer round 9 (5 of 10 reviewers) flagged that the new bare
``Plan:`` / ``Approach:`` / ``Here is the plan`` intent branches
reintroduced the original "wipe a complete answer" failure for
realistic final answers whose topic happens to contain a tool-action
word. Triggers for prompts like "Create a lesson plan for teaching
search skills" when the model answers:
Plan:
1. Search skills: students learn query keywords.
2. Source evaluation: compare domains.
3. Reflection: write what worked.
``_INTENT_SIGNAL`` matched the new ``Plan:`` lookahead because
``search`` appears within 120 chars, then ``_PLAN_LIST_FRAMING``
disqualified the numbered list, and the synthetic STOP turn wiped a
valid answer.
Revert the additions in ``_INTENT_SIGNAL``:
* Drop ``Plan:`` / ``Approach:`` (newline + action-verb lookahead).
* Drop ``Here is the plan`` / ``Here are my steps`` (action-verb
lookahead).
Plan stalls phrased with explicit first-person intent ("I'll search...",
"First, I'll fetch...", "Let me look up...") are still caught by the
existing intent patterns and ``_PLAN_LIST_FRAMING``.
Also narrow the plan-list action-verb whitelist to tool-specific verbs
(``search`` / ``look up`` / ``fetch`` / ``browse`` / ``web search`` /
``call (a) tool`` / ``run python`` / ``execute python``). Broad verbs
like ``use`` / ``compare`` / ``check`` / ``find`` / ``think`` /
``respond`` / ``answer`` / ``analyse`` / ``explore`` / ``outline`` /
``reason`` are removed because real answer lists use them ("1. Use
BFS", "1. Compare versions").
Finally, fix the test module's ``loggers`` / ``structlog`` stub
injection to only fire when the real module is missing AND to set
``__path__ = []`` on the stub. Previously the bare ``ModuleType`` could
poison ``sys.modules`` for any later test that imports a real
submodule (``from loggers.handlers import ...``).
Net behavioural change vs the previous commit: stricter on what
counts as a plan stall, never wipes a final answer titled
``Plan:`` / ``My plan:`` / ``Here is the plan you asked for``.
Reviewer round 8 surfaced a real false positive in the previous commit:
a final answer naturally titled "Plan:" / "My plan:" / "Approach:" with
numbered content items now slipped through _INTENT_SIGNAL and got
wiped by the synthetic STOP turn. Examples:
Plan:
1. Warm-up: Students review fractions.
2. Group practice.
3. Assessment.
My plan:
1. Breakfast: oatmeal and fruit.
2. Lunch: rice bowl.
3. Dinner: lentil soup.
Here is the plan you asked for. It is two pages long.
Add a lookahead requiring one of the conservative re-prompt action
verbs (search / fetch / verify / look up / call / compare / think /
respond / etc.) to appear within 120 chars after the "Plan:" /
"Approach:" / "Here is the plan" / "Here are my steps" marker. Plan
stalls whose items are tool actions ("Plan:\n1. search the docs\n2.
summarise the result") still match and re-prompt; prose plans whose
items are content do not.
Also mirror "first" in _PLAN_LIST_FRAMING so numbered action plans
that start with "First" stay disqualified even after the helper enters
the numbered-list branch.
Factor the action-verb set out as _REPROMPT_ACTION_VERBS so both
regexes share one source of truth.
Six new regression samples: three lesson / meal / weather plans that
must NOT wipe, three action-plan headers that must re-prompt, three
prose "Here is the plan" answers that must not wipe.
After narrowing the colon marker to lines starting with a generic
determiner ("My plan:" / "The approach:" / ...), inline product or
pricing answers like "Your current Plan: Pro includes local chats",
"The plan: Basic is free, Pro is $10/month", or
"My plan: use dynamic programming" still slipped into the re-prompt
path and could wipe a valid answer.
Add a lookahead requiring a newline (with optional trailing horizontal
whitespace) after the colon, so only header-style framings like
"Plan:\n1. search\n2. summarise" or "My approach:\n1. fetch" count.
Inline "Plan: <text>" is now treated as ordinary prose.
Add eight regression samples (lesson plan, meal plan, marketing plan,
pricing plan, recommended approach, migration plan, dynamic-programming
plan, currently active plan) all of which previously re-prompted under
the unanchored matcher and now correctly do not.
Two more gaps surfaced by another reviewer sweep on the previous commit:
1. _PLAN_LIST_FRAMING was missing several intent forms that
_INTENT_SIGNAL accepts, so numbered tool-action plans phrased with
"Allow me", "I'm going to", "I'm gonna", "I am gonna", or "I shall"
were silently classified as completed answers and skipped the
tool-call re-prompt. Mirror the full intent set from _INTENT_SIGNAL
so the two regexes stay in lock-step.
2. Bare \b(?:plan|approach): in _INTENT_SIGNAL / _PLAN_LIST_FRAMING
matched any in-text occurrence of "plan:" / "approach:", including
"lesson plan:" / "meal plan:" / "migration plan:". A direct answer
like "Here is a lesson plan:\n1. Warm-up\n2. Group practice" would
trip _INTENT_SIGNAL and risk wiping the response. Anchor the colon
marker to start of line and only allow generic determiners (my, the,
our, a, this, that) between the line start and the keyword.
3. Add "Here is the plan" / "Here are my steps" to both _INTENT_SIGNAL
and _PLAN_LIST_FRAMING so non-apostrophe phrasings of the same
framing pattern are caught.
Added regression tests covering every intent form against a numbered
action plan, and a line-anchor test that distinguishes generic plan
framings ("My plan:", "The approach:") from content noun phrases
("lesson plan:", "meal plan:").
The follow-up commit used ``i['’]?ll`` (apostrophe optional) in
``_PLAN_LIST_FRAMING``. With the apostrophe optional the alternative
also matches the word "ill" (sick), so a response like
"She is ill. Here is the list:\n1. ...\n2. ..." plus an unrelated
action verb within 80 chars was misclassified as a plan and re-prompted.
Make the apostrophe required (``i['’]ll``) to mirror the original
_INTENT_SIGNAL definition. Add a regression test that pins the
distinction: "ill" as adjective does not trigger plan framing, but
"I'll" / "I will" do.
Three follow-up gaps surfaced by another reviewer sweep on the previous commit:
1. Tilde-fenced code (~~~lang ... ~~~) was not detected. CommonMark allows
it and several models emit it when the body itself contains backticks.
Add a tilde alternative to _HAS_ANSWER_ARTIFACT mirroring the backtick
form (any info string, optional indent on close, length-bounded body).
2. Bare "Plan:" / "Approach:" lines did not match _INTENT_SIGNAL, so a
"Plan:\n1. search\n2. summarise" stall slipped past the entry gate
entirely. Add the colon form to the step / plan framing alternative.
3. The plan-framing verb whitelist missed common contemplative verbs
(think / respond / answer / analy[sz]e / explore / outline / gather /
query / reason) so plan stalls phrased without explicit "Here's my
plan" framing were misclassified as completed answers. Keep the
whitelist conservative: write / create / make / build / read / list /
try are intentionally out because real answer lists use them
("1. Write a poem", "1. Read War and Peace").
Added regression tests for each fix plus an extra ReDoS budget test for
the doctype/<html alternation worst case (about 7 ms today; assert < 50
ms so a future quantifier change that drops the inner {0,4000} bound
fails loudly).
Addresses three follow-ups flagged on the first cut of this PR by static
reviewers and parallel reviewer runs:
1. Numbered plan-only stalls were treated as completed answers. A
response like `Here's my plan:\n1. Search the web\n2. Summarise`
matched both `_INTENT_SIGNAL` and the numbered-list branch of
`_HAS_ANSWER_ARTIFACT`, so the tool-forcing re-prompt was skipped.
That contradicted the PR's stated invariant that plan-only stalls
still re-prompt. The list now has to be paired with no plan framing
(no `Here's my plan` / `plan:` / `approach:`, no intent phrase
followed by a tool-action verb) to count as an artifact.
2. Closed code fences with non-alpha info strings (`python3`, `c++`,
`c#`, `objective-c`, `ts-node`, `bash-session`, `python linenums="1"`)
were not recognised by the `[a-zA-Z]*` info-string class. Complete
answers in those languages still re-prompted and could be wiped.
The info-string class is now `[^\r\n]{0,200}` and the closing fence
may be indented.
3. Bare `<!doctype` or `<html` text was treated as an artifact. A
plan-only response that mentions `<html>` in prose now no longer
bypasses the re-prompt; the HTML branch requires a closing
`</html>` (doctype prefix optional).
All `[\s\S]{...}?` runs are length-bounded so ReDoS-style adversarial
input stays linear. ReDoS guard tests cover CRLF spam and repeated
`<html ` openings without close.
Two robustness fixes for the `_HAS_ANSWER_ARTIFACT` regex from the
parent commit, both caught by a thorough simulation suite covering
Linux/Mac/Windows line-ending portability and adversarial inputs.
1. **CRLF line endings.** The original `\n` literals missed Windows-
authored or CRLF-converted content (model echoing a pasted prompt,
etc.). Replaced with `\r?\n` everywhere a newline is required, so
closed code fences, numbered lists, and end-to-end re-prompt
decisions all work on `\r\n` as well as `\n`.
2. **Catastrophic backtracking on whitespace spam.** The numbered-list
alternative `(?:^|\r?\n)\s*\d+\.\s+\S.*?\r?\n\s*\d+\.` was
O(n^2) on long whitespace runs: `\s*` greedy + `\d+` failing +
`\s` matching `\r\n` led to repeated backtracking through the
newline characters. Measured at ~630ms for 10KB of `\r\n` repeats.
Fix: restrict the post-newline indent to `[ \t]*` (spaces / tabs
only). After `\r?\n` we are at column 0 and only spaces / tabs
are a sensible leading indent for a list item; greedy whitespace
was never needed. New worst case on the same input: <1ms (1000x
speedup).
Added 5 in-tree tests:
- test_artifact_regex_handles_crlf_code_fence
- test_artifact_regex_handles_crlf_numbered_list
- test_artifact_regex_handles_mixed_lf_crlf
- test_no_backtrack_on_crlf_spam (asserts <50ms on 10KB \r\n)
- test_no_reprompt_on_crlf_complete_python_game
All 18 reprompt-guard tests pass. All 253 llama_cpp-related tests pass.
Out-of-tree simulation suite (84 tests) passes on both Python 3.12 and
Python 3.13 inside isolated uv venvs.
The plan-without-action re-prompt at
`studio/backend/core/inference/llama_cpp.py` fires when the model
emits intent-only language ("first I'll ...", "let me ...") without
calling a tool. Previously the heuristic only checked an intent regex
and a 2000-char length cap. The same intent words occur in long
explanations that accompany REAL code or markup, so a complete reply
like "First, let me set up pygame. ```python ... ```" still tripped
the re-prompt, and the synthetic follow-up ("STOP. Do NOT write code
or explain.") wiped the user-visible answer.
Reproduced at scale in a 900-run sweep across 15 Qwen3.5/3.6 GGUF
configs: prompts that emit code or markup (Create a Python game,
Create a Flappy Bird game, weather dashboard HTML, sloth SVG)
landed empty `final_text` for the majority of seeds even on the
strongest configs.
Fix adds a `_HAS_ANSWER_ARTIFACT` regex covering:
- closed code fences (```...```)
- HTML pages (<!doctype, <html)
- complete SVG (<svg...</svg>)
- 2+ item numbered lists
and a `and not _HAS_ANSWER_ARTIFACT.search(_stripped)` guard on the
re-prompt condition. Plan-only stalls still re-prompt; complete
responses no longer do.
13 new unit tests in `test_llama_cpp_reprompt_guard.py` pin both
directions (artifact present -> no re-prompt; plan-only -> still
re-prompts).
