ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: don't re-prompt after model already produced a complete answer

Browse source 
The plan-without-action re-prompt at
`studio/backend/core/inference/llama_cpp.py` fires when the model
emits intent-only language ("first I'll ...", "let me ...") without
calling a tool. Previously the heuristic only checked an intent regex
and a 2000-char length cap. The same intent words occur in long
explanations that accompany REAL code or markup, so a complete reply
like "First, let me set up pygame. ```python ... ```" still tripped
the re-prompt, and the synthetic follow-up ("STOP. Do NOT write code
or explain.") wiped the user-visible answer.

Reproduced at scale in a 900-run sweep across 15 Qwen3.5/3.6 GGUF
configs: prompts that emit code or markup (Create a Python game,
Create a Flappy Bird game, weather dashboard HTML, sloth SVG)
landed empty `final_text` for the majority of seeds even on the
strongest configs.

Fix adds a `_HAS_ANSWER_ARTIFACT` regex covering:
  - closed code fences (```...```)
  - HTML pages (<!doctype, <html)
  - complete SVG (<svg...</svg>)
  - 2+ item numbered lists

and a `and not _HAS_ANSWER_ARTIFACT.search(_stripped)` guard on the
re-prompt condition. Plan-only stalls still re-prompt; complete
responses no longer do.

13 new unit tests in `test_llama_cpp_reprompt_guard.py` pin both
directions (artifact present -> no re-prompt; plan-only -> still
re-prompts).
This commit is contained in:
Daniel Han 2026-05-22 15:41:11 +00:00 • committed by danielhanchen
commit 078ae64cdf
2 changed files with 226 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

18
studio/backend/core/inference/llama_cpp.py
View file

@ -71,6 +71,23 @@ _INTENT_SIGNAL = re.compile(
)
_MAX_REPROMPTS = 3
# Substantive answer artifacts. Re-prompt fires when the model emits
# intent-only language ("first I'll ...", "let me ...") without a tool
# call, but the same intent words appear in long explanations that
# accompany REAL code or markup. Without this guard, a complete reply
# like "First, let me set up pygame. ```python ... ```" trips the
# re-prompt and the next user-visible message wipes the code. We
# require ALL of (intent signal, length < _REPROMPT_MAX_CHARS, no
# answer artifact) to fire.
_HAS_ANSWER_ARTIFACT = re.compile(
r"```[a-zA-Z]*\n[\s\S]+?\n```" # closed code fence
r"|<!doctype\b" # HTML page
r"|<html\b"
r"|<svg\b[\s\S]*?</svg>" # complete SVG
r"|(?:^|\n)\s*\d+\.\s+\S.*?\n\s*\d+\.", # 2+ numbered list items
re.IGNORECASE,
)
# Without max_tokens, llama-server defaults to n_predict = n_ctx (up to
# 262144 for Qwen3.5), producing many-minute zombie decodes when cancel
# fails. t_max_predict_ms is a wall-clock backstop applied unconditionally,
@ -4818,6 +4835,7 @@ class LlamaCppBackend:
and _reprompt_count < _MAX_REPROMPTS
and 0 < len(_stripped) < _REPROMPT_MAX_CHARS
and _INTENT_SIGNAL.search(_stripped)
and not _HAS_ANSWER_ARTIFACT.search(_stripped)
):
_reprompt_count += 1
logger.info(