fix(studio): improve tool-calling re-prompt for small models (#4783)
Small GGUF models (<9B) frequently generate full code or lengthy
explanations instead of calling tools, bypassing the existing
plan-without-action re-prompt mechanism. Three issues:
1. _REPROMPT_MAX_CHARS=500 was too low -- models that output full
HTML/code responses (often 1000+ chars) never triggered the
re-prompt at all, since it only fires on short responses.
2. _MAX_REPROMPTS=1 gave the model only one chance to comply.
Small models often need 2-3 nudges before switching from
text generation to tool calling.
3. The re-prompt text ("Please use the available tools...") was
too polite for small models to follow reliably.
4. Tool-calling detection missed chat templates using Jinja
whitespace-trimming syntax ({%- if tools -%}) since only
({%- if tools %}) and ({% if tools %}) were checked.
Changes:
- Raise _REPROMPT_MAX_CHARS from 500 to 2000 so longer responses
(code blocks, multi-paragraph plans) still trigger re-prompts
- Raise _MAX_REPROMPTS from 1 to 3 for more retry budget
- Use direct, imperative re-prompt language that small models
follow more reliably ("STOP. You MUST call a tool NOW.")
- Strengthen the system prompt tool nudge to explicitly forbid
outputting code blocks (redirect to the python tool instead)
- Add Jinja whitespace-trimmed variants to the tool_markers
list so all template styles are detected correctly
This commit is contained in:
parent
f9c4b08726
commit
4f9986ecb9
2 changed files with 8 additions and 5 deletions
|
|
@ -45,8 +45,8 @@ _INTENT_SIGNAL = re.compile(
|
|||
r"\b(?:now i|next i)\b"
|
||||
r")"
|
||||
)
|
||||
_MAX_REPROMPTS = 1
|
||||
_REPROMPT_MAX_CHARS = 500
|
||||
_MAX_REPROMPTS = 3
|
||||
_REPROMPT_MAX_CHARS = 2000
|
||||
|
||||
# ── Pre-compiled patterns for GGUF shard detection ───────────
|
||||
_SHARD_FULL_RE = re.compile(r"^(.*)-(\d{5})-of-(\d{5})\.gguf$")
|
||||
|
|
@ -2659,8 +2659,9 @@ class LlamaCppBackend:
|
|||
{
|
||||
"role": "user",
|
||||
"content": (
|
||||
"Please use the available tools to complete "
|
||||
"the task instead of describing what to do."
|
||||
"STOP. Do NOT write code or explain. "
|
||||
"You MUST call a tool NOW. "
|
||||
"Call web_search or python immediately."
|
||||
),
|
||||
}
|
||||
)
|
||||
|
|
|
|||
|
|
@ -96,8 +96,10 @@ router = APIRouter()
|
|||
|
||||
# Appended to tool-use nudge to discourage plan-without-action
|
||||
_TOOL_ACTION_NUDGE = (
|
||||
" Always call tools directly."
|
||||
" IMPORTANT: Always call tools directly -- never write code yourself."
|
||||
" Never describe what you plan to do -- just call the tool immediately."
|
||||
" For any code request, call the python tool. For any factual question, call web_search."
|
||||
" Do NOT output code blocks -- use the python tool instead."
|
||||
)
|
||||
|
||||
# Regex for stripping leaked tool-call XML from assistant messages/stream
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue