62827a9f3f
The existing _TOOL_ACTION_NUDGE tells the model "For any factual question,
call web_search" and "Never describe what you plan to do -- just call the
tool immediately". On small GGUF models (<9B) this causes two failure modes
we can measure:
1. First turn: the model calls web_search on questions it could answer from
training data, often queuing several parallel searches in one assistant
message. Measured on Qwen3.5-4B UD-Q4_K_XL, n=30: 28/30 tool_call, 2/30
answer. On Qwen3.5-4B Q4_K_M: 30/30 tool_call, 0/30 answer.
2. Subsequent turns: even with tool results already in context, the
"prefer tools" directive keeps dominating and the model searches again
instead of synthesising. Measured on UD-Q4_K_XL with one tool_result
present: 23/30 tool_call.
Two changes:
_TOOL_ACTION_NUDGE_SMALL: a softer nudge used for models under 9B. Asks
for tool use only when current information or a calculation is actually
needed, and explicitly discourages queuing multiple parallel tool calls.
Larger models keep the original aggressive nudge -- they weren't the ones
over-triggering.
_TOOL_SYNTHESISE_NUDGE: appended whenever the conversation already has
a tool result, and injected into the system message inside the internal
tool-call loop once the first tool result has been added. Phrasing
matters here -- framing this as a concrete action ("write the final
answer to the user's original question using what you have") works.
Framing it as an opt-out clause ("do not call more tools unless...") is
actually worse than no nudge, measured 33% vs 57% synthesis rate on
UD-Q4_K_XL.
Measured end-to-end on the same "How do you fine-tune an audio model
with Unsloth?" query, n=30 per cell:
Qwen3.5-4B UD-Q4_K_XL
OLD NEW
first turn tool_call 28/30 (93%) 2/30 (7%)
first turn answer 2/30 28/30
second turn tool_call 23/30 (77%) 6/30 (20%)
second turn answer 7/30 24/30
Qwen3.5-4B Q4_K_M (LM Studio)
OLD NEW
first turn tool_call 30/30 (100%) 0/30 (0%)
first turn answer 0/30 30/30
second turn tool_call 2/30 0/30
second turn answer 28/30 30/30
Test scripts under tests/test_ab_nudge.py and tests/test_stronger_synth.py.
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _html_to_md.py | ||
| anthropic_compat.py | ||
| audio_codecs.py | ||
| defaults.py | ||
| inference.py | ||
| llama_cpp.py | ||
| orchestrator.py | ||
| tools.py | ||
| worker.py | ||