Commit graph

1 commit

Author SHA1 Message Date
Nilay
22493242a3
Studio: Don't re-prompt finished answers in the tool loop (#7505)
* don't re-prompt finished answers in the tool loop

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* keep a separate post-tool reprompt budget and tighten the intent regexes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reset the repeat guard after a tool runs and suppress 'I should call ...' forced stalls

* Cover 'must' in forced-retry suppression, keep appended answers, and count RAG autoinject as a prior tool run

* Anchor obligation suppression to sentence starts and wire the repeat guard into the safetensors loop

* Keep deletions out of restatement and nudge pronoun-free first-step plans

* Tighten repeat similarity, anchor subjectless plans, and restore first-step plan forms

* Keep first-person plan framing and punctuation-bearing terms out of repeat detection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Keep leading term punctuation, accept colon-delimited first steps, and drop invoke/query from suppression

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Tighten comments on the plan-without-action re-prompt guards

* Compare plans by token sequence, suppress subjectless modals, and accept dash-delimited first steps

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* studio: narrow the first-step plan match and make repeat detection content-based

Restrict the bare "First, <word>" intent alternative to a pronoun, an explicit
plan, or an investigative verb, so ordinal prose ("First place went to Alice")
and user-facing advice ("First, install the package") no longer count as a plan
without action.

Keep punctuation-only tokens in the repeat comparison, so "the value is 5" and
"the value is < 5" stay distinct, and compare content-word sequences instead of
a similarity ratio: any ratio is length-dependent, so one corrected token in a
54-token plan still scored 0.98 and cost the model its remaining nudge.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* studio: tighten comments in the plan-without-action re-prompt path

* studio: keep a forced retry that pivots from a plan to an answer

The obligation-plan branch discarded the whole turn, so a retry such as
"I should call web_search, but the answer is Tokyo." reached the user as
nothing at all. Suppress the plan only when nothing follows it: a pivot
after the match keeps the output, and _FINAL_ANSWER_SIGNAL now recognises
"the answer is" and "to summarise" alongside "answer:".

Leaking a plan sentence is cosmetic, dropping an answer is not, so the
doubtful case now resolves towards shipping the turn.

* studio: keep articles in repeat comparison and exclude missing-answer phrasing

Articles are not filler: dropping them made "search for The Who" and
"search for Who" compare equal, so a corrected target ended the nudge.

_FINAL_ANSWER_SIGNAL matched "the answer is not in the provided context",
which announces a missing answer, so the plan behind it shipped as the final
response instead of being suppressed. Negated forms are now excluded.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* studio: tighten the pivot and final-answer signals, drop filler-insensitive repeats

The purpose clause in "call web_search to summarize the results" matched the
final-answer signal, so the plan shipped instead of being suppressed; that
alternative is gone. A pivot word now has to carry text of its own, since
"I should call web_search, though." answers nothing.

Repeat detection no longer ignores filler words. No word is reliably filler:
dropping them to absorb rewording also absorbed the target ("OK Go" became
"Go"). A missed repeat costs one nudge out of the cap; a false one strands the
plan unexecuted.

* studio: exempt offers of help, and add a measured accuracy floor

Offering to help hands control back exactly like the existing "let me know"
exemption. On a corpus of real model turns, "I'll do my best to help" and
"allow me to assist" close a clarification request and never precede a tool
call, but they were read as intent and re-prompted. "help you" keeps its plan
reading when an action verb follows it.

The new test scores the classifier against 300 turns captured from three local
GGUF models, each one a finished answer: the turn called no tool, and three
regenerations behind the production nudge produced no tool call either. Over
those turns, wasted nudges go from 36 (12.0%) on main to 5 (1.7%), and retries
whose text would be discarded from 60 (20.2%) to 1 (0.3%).

Until now these patterns were tuned on hand-written example sentences, which
cannot show how often the classifier is right on real output.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-07-29 02:38:17 -07:00