ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Address remaining reviewer findings: late tool_call IDs and XML speculation

Browse source 
1. Late-arriving tool_calls.id: when a provider sends the real ID on a
   later delta chunk (after the initial one with index and function
   name), the accumulator now updates the ID instead of keeping the
   synthetic "call_{idx}" placeholder. (P2, 2/10 reviewers)

2. XML speculation respects auto_heal_tool_calls: when auto_heal is
   explicitly disabled, _TOOL_XML_SIGNALS is empty so the BUFFERING
   state never speculatively holds content for XML prefix detection.
   Content starting with literal "<tool_call>" or "<function=" text
   flows straight through without delay. (P2, 1/10 reviewers)

Skipped: finish_reason="tool_calls" without delta.tool_calls fallback
(P1, 1/10 reviewers). llama-server always sends delta.tool_calls
fragments in streaming mode. A non-streaming fallback for this edge
case would add complexity for a scenario that does not occur in
practice with the supported backend.
This commit is contained in:
Daniel Han 2026-03-27 06:46:23 +00:00
commit aa9bea12b9
1 changed files with 12 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

14
studio/backend/core/inference/llama_cpp.py
View file

@ -1725,8 +1725,12 @@ class LlamaCppBackend:
text = pat.sub("", text)
return text.strip() if final else text
# XML prefixes that signal a tool call in content
_TOOL_XML_SIGNALS = ("<tool_call>", "<function=")
# XML prefixes that signal a tool call in content.
# Empty when auto_heal is disabled so the buffer never
# speculatively holds content for XML detection.
_TOOL_XML_SIGNALS = (
("<tool_call>", "<function=") if auto_heal_tool_calls else ()
)
_MAX_BUFFER_CHARS = 32
for iteration in range(max_tool_iterations):
@ -1871,6 +1875,12 @@ class LlamaCppBackend:
"arguments": "",
},
}
elif tc_d.get("id"):
# Update ID if real one
# arrives on a later delta
tool_calls_acc[idx]["id"] = (
tc_d["id"]
)
func = tc_d.get("function", {})
if func.get("name"):
tool_calls_acc[idx]["function"][