* Studio: client-tool passthrough healing for safetensors and MLX
PR 6801 made response-side tool-call healing default-on for the client-tool
passthrough, but only on the GGUF path: the passthrough branch in
/v1/chat/completions is gated on using_gguf, and the safetensors section never
reads payload.tools, so a client-tools request against a safetensors or MLX
model silently dropped the tool schemas and returned prose with no tool_calls.
Add the missing leg. When a non-GGUF model is loaded, the request declares
client tools (or carries tool-role history), server-side tools are off, and the
template supports tools, the route now:
- renders the tools into the chat template for a single turn via the existing
backend.generate_chat_response(..., tools=...) seam (worker templating
already accepts role=tool and assistant.tool_calls messages, normalized with
_openai_messages_for_passthrough);
- non-streaming: promotes text-form calls with heal_openai_message, honors the
opt-in nudge single retry (nudge_should_retry / nudge_messages), caps healed
calls when parallel_tool_calls=false (covers the nudge retry too), and sets
finish_reason=tool_calls with content null on a pure tool-call turn;
- streaming: derives deltas from the worker's cumulative snapshots and feeds
StreamToolCallHealer, emitting healed tool-call deltas and the correct
finish chunk, guarded against repeated or shrinking snapshots.
heal_gate semantics are identical to the GGUF passthrough: default on,
auto_heal_tool_calls=false or UNSLOTH_DISABLE_TOOL_CALL_HEALING=1 relays
verbatim, tool_choice narrows promotion, undeclared names stay text. MLX rides
the same orchestrator seam, so both local backends gain the behavior.
CompletionMessage.content becomes Optional so a promoted pure tool-call turn
matches the OpenAI contract (content null when only tool_calls return).
Adds tests/test_sf_client_tools_passthrough.py (22 cases: healing, gating,
opt-outs, streaming deltas, tool-role history, dict-arguments history, forced
tool_choice, parallel cap, usage, nudge on/off/double-failure, generator error
hygiene, disconnect reset, empty output, MLX path).
* Address review: tool_choice none, developer folding, retry fallback, monitor reply
Four review follow-ups on the safetensors/MLX client-tool passthrough leg:
- tool_choice="none" keeps the tool-history templating but no longer
advertises the tools, so a forced final-answer turn is not prompted into
emitting markup that the (correctly disabled) healer would relay as prose.
Mirrors the GGUF passthrough where llama-server honors tool_choice itself.
- OpenAI "developer" messages fold into a single leading system message via
_set_or_prepend_system_message before templating; local templates reject the
role and the fallback formatter drops it.
- A nudge retry that fails or is cancelled after the original answer exists
falls back to the first response instead of surfacing a 500, matching the
GGUF nudge path.
- The API monitor records the healed tool call summary instead of the raw
markup on a promoted turn.
Adds four regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: forced tool_choice templating, content-part flattening, stream monitor parity
- A forced tool_choice function is now the only schema rendered into the
local template, so the advertised tools and the healer allowlist can no
longer disagree (llama-server enforces tool_choice itself on the GGUF path).
- Content-part lists are flattened to their text parts before templating.
Remote image URLs are not decodable locally, so such requests reached this
path with part lists that raise inside apply_chat_template on text-only
templates; the plain non-GGUF path has always flattened them.
- The streaming monitor entry is now fed from the healed events the client
actually receives, recording promoted calls as the [tool_calls] summary
the non-streaming path records.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate passthrough on the engaged server path, deserialize templated arguments
- The client-tools gate now keys on _sf_use_tools (whether the server-side
tool path actually claimed the request) instead of the raw mcp_enabled
flag: with an empty MCP registry or a CLI --disable-tools policy, a client
that sets mcp_enabled while declaring its own tools fell through to plain
generation with the tools silently dropped. The GGUF passthrough gate has
no mcp_enabled clause either.
- New _structured_tool_history_for_local_template deserializes assistant
tool_calls[].function.arguments JSON strings into mappings for the
templated copy only: spec-compliant clients send strings, but local chat
templates iterate arguments as a mapping or raise on strings, which
crashed or misrendered multi-turn tool history. The HTTP response and the
GGUF wire shape keep strings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments and docstrings in the client-tools passthrough
* Report first-attempt usage when a nudge retry is discarded
When nudge_should_retry fires but the retry produces no healable tool call
(or raises), the first response is still delivered to the client. The retry's
generate() had already overwritten stats_holder, so _monitor_usage recorded
the unseen retry's token counts against the request instead of the first
attempt that was actually returned. Capture the first attempt's stats before
the retry and restore them on both the no-heal and exception paths so the
monitor reports the usage of the response the caller received.
* Do not promote buffered tool markup when a stream is cancelled
The streaming client-tool heal path breaks out of the token loop when
cancel_event is set (the registry "Stop" path), but then still fell through to
healer.finalize(), which heals incomplete tool markup at EOF (allow_incomplete)
and emits a tool_calls delta plus finish_reason=tool_calls. Because the Stop
request only sets the event and leaves the SSE socket open, the client received
that promoted call and executed a tool the user had just cancelled. The disconnect
path already returns before finalize; guard finalize and the finish_reason on
cancel_event too, so a cancelled stream ends with finish_reason=stop and no tool
call. Adds a regression test driving a Stop mid-emission with buffered markup.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the client-tools passthrough
* Trim client-tools passthrough comments further
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
46ab683065