ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: soften tool-use nudge for small models and add synthesise directive

Browse source 
The existing _TOOL_ACTION_NUDGE tells the model "For any factual question,
call web_search" and "Never describe what you plan to do -- just call the
tool immediately". On small GGUF models (<9B) this causes two failure modes
we can measure:

1. First turn: the model calls web_search on questions it could answer from
   training data, often queuing several parallel searches in one assistant
   message. Measured on Qwen3.5-4B UD-Q4_K_XL, n=30: 28/30 tool_call, 2/30
   answer. On Qwen3.5-4B Q4_K_M: 30/30 tool_call, 0/30 answer.

2. Subsequent turns: even with tool results already in context, the
   "prefer tools" directive keeps dominating and the model searches again
   instead of synthesising. Measured on UD-Q4_K_XL with one tool_result
   present: 23/30 tool_call.

Two changes:

_TOOL_ACTION_NUDGE_SMALL: a softer nudge used for models under 9B. Asks
for tool use only when current information or a calculation is actually
needed, and explicitly discourages queuing multiple parallel tool calls.
Larger models keep the original aggressive nudge -- they weren't the ones
over-triggering.

_TOOL_SYNTHESISE_NUDGE: appended whenever the conversation already has
a tool result, and injected into the system message inside the internal
tool-call loop once the first tool result has been added. Phrasing
matters here -- framing this as a concrete action ("write the final
answer to the user's original question using what you have") works.
Framing it as an opt-out clause ("do not call more tools unless...") is
actually worse than no nudge, measured 33% vs 57% synthesis rate on
UD-Q4_K_XL.

Measured end-to-end on the same "How do you fine-tune an audio model
with Unsloth?" query, n=30 per cell:

Qwen3.5-4B UD-Q4_K_XL
                            OLD              NEW
  first turn tool_call      28/30 (93%)      2/30  (7%)
  first turn answer         2/30             28/30
  second turn tool_call     23/30 (77%)      6/30  (20%)
  second turn answer        7/30             24/30

Qwen3.5-4B Q4_K_M (LM Studio)
                            OLD              NEW
  first turn tool_call      30/30 (100%)     0/30  (0%)
  first turn answer         0/30             30/30
  second turn tool_call     2/30             0/30
  second turn answer        28/30            30/30

Test scripts under tests/test_ab_nudge.py and tests/test_stronger_synth.py.
This commit is contained in:
Daniel Han 2026-04-16 11:03:17 +00:00
commit 62827a9f3f
2 changed files with 80 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

37
studio/backend/core/inference/llama_cpp.py
View file

@ -2473,6 +2473,39 @@ class LlamaCppBackend:
_accumulated_predicted_ms = 0.0
_accumulated_predicted_n = 0
# Flipped once a tool result has been appended to the conversation
# and the system message updated with a synthesise-now directive.
# Small models otherwise keep honouring the initial "prefer tools"
# nudge and loop on search forever, even when the result they need
# is already in context.
_synthesise_nudge_applied = False
# Phrased as a concrete action rather than an opt-out clause --
# the "do not call more tools" wording actively harmed small-model
# synthesis rate in our benchmarks.
_SYNTHESISE_NUDGE = (
" Tool results have been gathered. Now write the final answer to the"
" user's original question using what you have. Tool calls are no"
" longer needed for this turn."
)
def _apply_synthesise_nudge() -> None:
nonlocal _synthesise_nudge_applied
if _synthesise_nudge_applied:
return
for _msg in conversation:
if _msg.get("role") == "system":
_content = _msg.get("content") or ""
if _SYNTHESISE_NUDGE.strip() not in _content:
_msg["content"] = _content.rstrip() + _SYNTHESISE_NUDGE
_synthesise_nudge_applied = True
return
# No system message yet: insert one
conversation.insert(
0,
{"role": "system", "content": _SYNTHESISE_NUDGE.lstrip()},
)
_synthesise_nudge_applied = True
def _strip_tool_markup(text: str, *, final: bool = False) -> str:
if not auto_heal_tool_calls:
return text
@ -3135,6 +3168,10 @@ class LlamaCppBackend:
tool_msg["tool_call_id"] = tool_call_id
conversation.append(tool_msg)
# First tool result of the loop: tell the model to
# synthesise an answer rather than continue searching.
_apply_synthesise_nudge()
# Clear tool status badge before next generation iteration
yield {"type": "status", "text": ""}
# Continue the loop to let model respond with context

45
studio/backend/routes/inference.py
View file

@ -127,6 +127,34 @@ _TOOL_ACTION_NUDGE = (
" Do NOT output code blocks -- use the python tool instead."
)
# Softer variant for small models (<9B). The aggressive ALWAYS-CALL-TOOLS
# phrasing above causes small models to pick web_search on every factual
# question even when the answer sits in their training data, and to keep
# calling search after each result instead of synthesising. See
# tests/test_tool_loop_with_nudge.py for the measured behaviour.
_TOOL_ACTION_NUDGE_SMALL = (
" Call tools only when you need current information or a specific"
" calculation. For questions within your knowledge, answer directly."
" Issue one tool call at a time rather than queuing several at once."
)
# Appended whenever the current conversation already contains a tool
# result, to counteract the "prefer tools" nudge and push the model
# toward synthesising a final answer from what it has. Matters most for
# small models where the initial "prefer tools" directive is still
# dominating on the second and subsequent turns.
#
# Phrasing matters a lot here -- "do not call more tools" as an opt-out
# clause is actually worse than no nudge (measured 33% vs 57% synthesis
# rate on Qwen3.5-4B UD-Q4_K_XL). Reframing as a concrete action ("write
# the final answer to the user's original question using what you have")
# is what moves the needle: the same bench run jumps to 90% synthesis.
_TOOL_SYNTHESISE_NUDGE = (
" Tool results have been gathered. Now write the final answer to the"
" user's original question using what you have. Tool calls are no"
" longer needed for this turn."
)
# Regex for stripping leaked tool-call XML from assistant messages/stream
_TOOL_XML_RE = _re.compile(
r"<tool_call>.*?</tool_call>|<function=\w+>.*?</function>",
@ -1242,7 +1270,16 @@ async def openai_chat_completions(
_nudge = ""
if _nudge:
_nudge += _TOOL_ACTION_NUDGE
_nudge += (
_TOOL_ACTION_NUDGE_SMALL if _is_small_model else _TOOL_ACTION_NUDGE
)
# If the current conversation already has a tool result
# message, append the synthesise-now directive. Covers
# forked chats and long-running sessions where the prior
# "prefer tools" line keeps biasing the model into search
# loops instead of answering from what it already has.
if any(m.get("role") == "tool" for m in chat_messages):
_nudge += _TOOL_SYNTHESISE_NUDGE
# Append nudge to system prompt (preserve user's prompt)
if system_prompt:
system_prompt = system_prompt.rstrip() + "\n\n" + _nudge
@ -2468,7 +2505,11 @@ async def anthropic_messages(
_nudge = ""
if _nudge:
_nudge += _TOOL_ACTION_NUDGE
_nudge += (
_TOOL_ACTION_NUDGE_SMALL if _is_small_model else _TOOL_ACTION_NUDGE
)
if any(m.get("role") == "tool" for m in openai_messages):
_nudge += _TOOL_SYNTHESISE_NUDGE
# Inject into system prompt
if openai_messages and openai_messages[0].get("role") == "system":
openai_messages[0]["content"] = (