ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Drop anchored orphan tools to preserve template validity for PR #5710

Browse source 
The final invariant sweep skipped anchored (multimodal) tool messages
whose tcid had no matching prior assistant tool_calls. The anchor
survived but the chat template did not -- a tool_call_id pointing at
nothing returns 400 from llama-server / OpenAI on the next call.

Pair-validity is the hard invariant (the module exists to enforce it
upstream of llama-server) and the multimodal-anchor rule is the soft
quality preference. Drop the orphan rather than violate the template:
lose the image content for that one turn, keep the conversation alive.
This commit is contained in:
Daniel Han 2026-05-26 01:03:10 +00:00
commit 3c248dca26
2 changed files with 36 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

11
studio/backend/core/inference/context_compaction.py
View file

@ -317,20 +317,25 @@ class SlidingWindowCompact(CompactStrategy):
# Final invariant sweep, two passes so the asst-strip decision
# sees the post-orphan-drop tool set:
# pass 1: drop tools whose tcid has no matching assistant
# ``tool_calls`` earlier in the kept output (anchored tools
# stay; same leak-rather-than-violate-anchor rule);
# ``tool_calls`` earlier in the kept output;
# pass 2: recompute responded_ids from the surviving tools and
# mark assts whose tool_calls aren't all responded for strip.
# Computing responded_ids before pass 1 would let a stale tcid
# of a just-dropped orphan tool satisfy `ids <= responded_ids`,
# leaving the asst with dangling tool_calls (OpenAI 400).
# Anchored (multimodal) orphan tools are still dropped here:
# keeping them violates the chat-template pair invariant (a hard
# invariant that 400s upstream) to honor the multimodal-anchor
# quality preference, which the docstring describes as a soft
# quality rule. The pair-validity wins; the image content is
# lost rather than the entire turn.
seen_ids: set[str] = set()
for i, m in enumerate(messages):
if i in dropped:
continue
if m.get("role") == "assistant":
seen_ids |= _assistant_tool_call_ids(m)
elif m.get("role") == "tool" and i not in anchor_idx:
elif m.get("role") == "tool":
tcid = m.get("tool_call_id")
if isinstance(tcid, str) and tcid and tcid not in seen_ids:
dropped.add(i)

28
studio/backend/tests/test_context_compaction_edge.py
View file

@ -907,3 +907,31 @@ def test_orphan_tool_before_asst_does_not_leave_dangling_tool_calls():
out = SlidingWindowCompact(keep_recent = 5).compact(msgs, budget_tokens = 1000)
asst_ids, tool_ids = _surviving_tool_ids(out)
assert asst_ids == tool_ids, (asst_ids, tool_ids)
def test_anchored_multimodal_orphan_tool_dropped():
"""Anchored multimodal tool message references a tool_call_id no
surviving assistant declares. Keeping the anchor would leave a
dangling `tool_call_id` in the output and 400 upstream. Pair
validity is the hard invariant; the multimodal anchor is the soft
quality preference, so we drop the orphan rather than violate the
template. Earlier behavior preserved the anchor and produced an
invalid chat template.
"""
msgs = [
_msg("system", "sys"),
_msg("user", "first task"),
{
"role": "tool",
"tool_call_id": "stale_call",
"content": [
{"type": "text", "text": "image description"},
{"type": "image_url", "image_url": {"url": "x"}},
],
},
_msg("user", "follow up"),
_long("assistant", 100),
]
out = SlidingWindowCompact(keep_recent = 2).compact(msgs, budget_tokens = 5)
asst_ids, tool_ids = _surviving_tool_ids(out)
assert asst_ids == tool_ids, (asst_ids, tool_ids)