Studio: soften tool-use nudge for small models and add synthesise directive

The existing _TOOL_ACTION_NUDGE tells the model "For any factual question,
call web_search" and "Never describe what you plan to do -- just call the
tool immediately". On small GGUF models (<9B) this causes two failure modes
we can measure:

1. First turn: the model calls web_search on questions it could answer from
   training data, often queuing several parallel searches in one assistant
   message. Measured on Qwen3.5-4B UD-Q4_K_XL, n=30: 28/30 tool_call, 2/30
   answer. On Qwen3.5-4B Q4_K_M: 30/30 tool_call, 0/30 answer.

2. Subsequent turns: even with tool results already in context, the
   "prefer tools" directive keeps dominating and the model searches again
   instead of synthesising. Measured on UD-Q4_K_XL with one tool_result
   present: 23/30 tool_call.

Two changes:

_TOOL_ACTION_NUDGE_SMALL: a softer nudge used for models under 9B. Asks
for tool use only when current information or a calculation is actually
needed, and explicitly discourages queuing multiple parallel tool calls.
Larger models keep the original aggressive nudge -- they weren't the ones
over-triggering.

_TOOL_SYNTHESISE_NUDGE: appended whenever the conversation already has
a tool result, and injected into the system message inside the internal
tool-call loop once the first tool result has been added. Phrasing
matters here -- framing this as a concrete action ("write the final
answer to the user's original question using what you have") works.
Framing it as an opt-out clause ("do not call more tools unless...") is
actually worse than no nudge, measured 33% vs 57% synthesis rate on
UD-Q4_K_XL.

Measured end-to-end on the same "How do you fine-tune an audio model
with Unsloth?" query, n=30 per cell:

Qwen3.5-4B UD-Q4_K_XL
                            OLD              NEW
  first turn tool_call      28/30 (93%)      2/30  (7%)
  first turn answer         2/30             28/30
  second turn tool_call     23/30 (77%)      6/30  (20%)
  second turn answer        7/30             24/30

Qwen3.5-4B Q4_K_M (LM Studio)
                            OLD              NEW
  first turn tool_call      30/30 (100%)     0/30  (0%)
  first turn answer         0/30             30/30
  second turn tool_call     2/30             0/30
  second turn answer        28/30            30/30

Test scripts under tests/test_ab_nudge.py and tests/test_stronger_synth.py.
This commit is contained in:
Daniel Han 2026-04-16 11:03:17 +00:00
commit 62827a9f3f
2 changed files with 80 additions and 2 deletions

View file

@ -2473,6 +2473,39 @@ class LlamaCppBackend:
_accumulated_predicted_ms = 0.0
_accumulated_predicted_n = 0
# Flipped once a tool result has been appended to the conversation
# and the system message updated with a synthesise-now directive.
# Small models otherwise keep honouring the initial "prefer tools"
# nudge and loop on search forever, even when the result they need
# is already in context.
_synthesise_nudge_applied = False
# Phrased as a concrete action rather than an opt-out clause --
# the "do not call more tools" wording actively harmed small-model
# synthesis rate in our benchmarks.
_SYNTHESISE_NUDGE = (
" Tool results have been gathered. Now write the final answer to the"
" user's original question using what you have. Tool calls are no"
" longer needed for this turn."
)
def _apply_synthesise_nudge() -> None:
nonlocal _synthesise_nudge_applied
if _synthesise_nudge_applied:
return
for _msg in conversation:
if _msg.get("role") == "system":
_content = _msg.get("content") or ""
if _SYNTHESISE_NUDGE.strip() not in _content:
_msg["content"] = _content.rstrip() + _SYNTHESISE_NUDGE
_synthesise_nudge_applied = True
return
# No system message yet: insert one
conversation.insert(
0,
{"role": "system", "content": _SYNTHESISE_NUDGE.lstrip()},
)
_synthesise_nudge_applied = True
def _strip_tool_markup(text: str, *, final: bool = False) -> str:
if not auto_heal_tool_calls:
return text
@ -3135,6 +3168,10 @@ class LlamaCppBackend:
tool_msg["tool_call_id"] = tool_call_id
conversation.append(tool_msg)
# First tool result of the loop: tell the model to
# synthesise an answer rather than continue searching.
_apply_synthesise_nudge()
# Clear tool status badge before next generation iteration
yield {"type": "status", "text": ""}
# Continue the loop to let model respond with context