* studio: tool calling for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5615)
Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe.
* studio: tool-call healing parity between safetensors / MLX and GGUF
After the multi-format parser landed in #5615, the safetensors / MLX
agentic loop and the GGUF loop still differed on healing behaviour.
This commit closes the gaps in both directions so the two backends
react the same way to identical model output.
Changes:
1. core/inference/llama_cpp.py -- the GGUF BUFFERING state machine
now wakes on every emission marker the shared parser knows. Was
("<tool_call>", "<function="); is now the five-tuple imported
from core.inference.tool_call_parser (Qwen / Qwen3.5 / Llama-3
<|python_tag|> / Mistral [TOOL_CALLS] / Gemma 4 <|tool_call>).
Stream cleanup is delegated to the same shared strip_tool_markup
so leaked markup from any family is removed from assistant
content.
2. core/inference/llama_cpp.py -- per-tool canonical heal key. When
a tool arguments field is a bare string and JSON parsing fails,
the GGUF path now heals to {"code": raw_args} for python,
{"command": raw_args} for terminal, and {"query": raw_args} for
everything else. Was hard-coded to {"query": raw_args}, which
silently routed every python / terminal emission through
web_search. Mirrors safetensors_agentic._CANONICAL_HEAL_ARG.
3. core/inference/safetensors_agentic.py -- re-prompt on plan-
without-action. When the model emits a short forward-looking
intent ("I'll search for that", "Let me check", "First, I
will...") and no tool call, the loop nudges the model to act
instead of silently returning a plan-only answer. Up to
_MAX_REPROMPTS=3 (matches GGUF). The intent regex, character
cap, and instruction text are byte-identical to the GGUF path.
The buffer-end fall-through is unified so a buffered intent
emission that never exits the BUFFERING state still triggers
the re-prompt.
4. core/inference/safetensors_agentic.py -- extra iteration slots
for re-prompts. The loop now budgets max_tool_iterations +
_MAX_REPROMPTS + 1 total iterations and tracks the tool-call
count separately, so a stalling model can be nudged 3x without
eating the caller's tool-call budget. Mirrors the _extra slot
reservation in the GGUF path.
Tests (14 new safetensors-side units; 5 GGUF parity pins):
TestLoopRePrompt -- intent-trigger, plain-answer,
no-tools, cap-at-three, budget
preserved, buffer-end intent.
TestLoopCanonicalHealKey -- python / terminal / unknown.
TestGGUFSafetensorsHealingParity -- shared markers used, shared
strip used, canonical heal keys
identical, intent regex matches
same phrases, _MAX_REPROMPTS
equal on both backends.
All 110 targeted tests pass locally; the broader tool / inference /
model-config / sandbox / anthropic / mlx suites stay green.
Why this matters
Without this parity, Llama-3.2 / Mistral / Gemma 4 emissions on Mac
(MLX) and Linux-safetensors stop the agentic loop as soon as the
model says "Let me...", because the GGUF re-prompt logic never
existed on these backends. The two-marker GGUF BUFFERING tuple also
let non-Qwen tool emissions stream out as plain prose when
llama-server's structured channel did not pick them up. Both paths
now drain the same way, heal the same way, and re-prompt the same
way -- so a tool call that works on GGUF works identically on
safetensors / MLX.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix tool-call parser bugs from gemini review on #5620
Three high-priority gemini findings on the tool-call parsing additions:
1. unicode_escape on UTF-8 bytes corrupts non-ASCII literals
(e.g. ✨ becomes â\x9c¨). Replace with json.loads on a quoted
string -- preserves emoji / CJK / RTL while still handling
\n \t \uXXXX escapes.
2. Llama-3 sentinel stripping is order-dependent. A leading
`<|eot_id|><|begin_of_text|>` left `<|begin_of_text|>` behind
because the loop had already passed that sentinel. Loop until
no sentinel matches at the start.
3. Mistral v11+ `[TOOL_CALLS] name { json }` regex uses non-greedy
`\{.*?\}` which truncates at the first `}` of a nested JSON
argument, leaking the tail (e.g. `}}`) into user-visible
streamed text. Same problem for the v0.3 array pattern with
nested brackets. Strip those with balanced brace/bracket
scanning via a new `_strip_mistral_closed_calls` helper called
from `strip_tool_markup`.
Also fix the inference routes' parallel `_TOOL_XML_RE`:
- Same nested-JSON truncation in the Mistral patterns; route the
strip through the parser's balanced-scan helper via a thin
`_strip_tool_xml` wrapper that all existing callers now use.
- Llama-3 `<|python_tag|>[^\n<]*` stopped at any `<`, leaking the
tail of any tool call whose argument contained a literal `<`
(queries, code snippets). Relax to `[^\n]*` which keeps the
strip confined to the actual end-of-line.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tool calling for DeepSeek (R1/V3/V3.1), GLM 4.x, Kimi K2
Adds three more emission-family parsers to tool_call_parser.py so the
shared safetensors / MLX / GGUF agentic loop covers the major open-
weight reasoning families. Patterns ported from llama.cpp
(common/chat-parser.cpp legacy pre-PEG branch), vLLM
(tool_parsers/deepseekv3*, glm4_moe, kimi_k2), and SGLang
(function_call/deepseekv31_detector, glm4_moe_detector, kimik2_detector).
All three references are MIT (llama.cpp) or Apache-2.0 (vLLM, SGLang).
Formats covered:
DeepSeek R1 <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function
<｜tool▁sep｜>NAME\n```json\n{...}\n```<｜tool▁call▁end｜>
<｜tool▁calls▁end｜>
-- args wrapped in a Markdown json fence, ``function``
literal prefix per llama.cpp common_chat_parse_
deepseek_r1 (chat-parser.cpp:801-820)
DeepSeek V3/V3.1
<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>NAME
<｜tool▁sep｜>{json}<｜tool▁call▁end｜><｜tool▁calls▁end｜>
-- bare JSON, no code fence, no ``function`` prefix
per llama.cpp common_chat_parse_deepseek_v3_1
(chat-parser.cpp:822-879)
GLM 4.5/4.6/4.7 <tool_call>NAME\n<arg_key>k1</arg_key>
\n<arg_value>v1</arg_value>...</tool_call>
-- strings raw, non-strings JSON-encoded per
chat_template.jinja; multi-call is back-to-back
blocks. Per llama.cpp common_chat_parse_glm_4_5
(chat-parser.cpp:1040-1052)
Kimi K2 <|tool_calls_section_begin|><|tool_call_begin|>
functions.NAME:IDX<|tool_call_argument_begin|>{json}
<|tool_call_end|><|tool_calls_section_end|>
-- bare name recovered by stripping ``functions.``
prefix and ``:IDX`` suffix; full id preserved as
tool_calls[i].id so the roundtrip replays verbatim.
Per llama.cpp common_chat_parse_kimi_k2
(chat-parser.cpp:896-913)
Marker collisions
GLM uses the same ``<tool_call>`` opener as Qwen but with a bare
function name + ``<arg_key>`` body (Qwen has ``\s*{`` after the tag).
The dispatch keeps Qwen first; Qwen's _TC_JSON_START_RE returns no
matches on a GLM emission, so the fall-through to _parse_glm_tool_
calls handles it correctly. Existing Qwen tests confirm zero
regression.
Streaming buffer
TOOL_XML_SIGNALS extended from 5 markers to 12 so the BUFFERING state
machine wakes on every new family's section opener. Added the
DeepSeek alternative markers (ASCII underscores, short ``<｜tool▁calls｜>``
form) because real checkpoints emit those variants.
Strip patterns
_TOOL_CLOSED_PATS adds DeepSeek envelope (``<｜tool▁calls▁begin｜>...
<｜tool▁calls▁end｜>``) and Kimi section (``<|tool_calls_section_begin|>
...<|tool_calls_section_end|>``). _TOOL_ALL_PATS adds the same plus
the unclosed-tail variants so a truncated stream does not leak
markup.
Route gate
_detect_safetensors_features._PARSER_MARKERS grows to include
DeepSeek and Kimi markers plus ``<arg_key>`` (the unique GLM signal).
_TOOL_XML_RE (the route-layer markup-strip regex) gets DeepSeek and
Kimi closed-pair patterns. _TOOL_TEMPLATE_MARKERS in llama_cpp.py
adds ``message['role'] == 'tool'``, ``message['tool_calls']``, and
``tool_calls is defined`` so the classifier recognises DeepSeek's
subscripted-access template style (it has no top-level
``{% if tools %}`` block).
Tests (39 new):
TestParserDeepSeek (7) -- R1 fence, short-form opener, V3.1 bare,
multi-call, with-reasoning, strip,
signal-wakes-streaming
TestParserGLM (6) -- single, mixed types, multi-call,
unclosed-heal, no-Qwen-regression, strip
TestParserKimi (6) -- single, multi-call, dotted-name, unclosed,
strip, signal-wakes-streaming
TestParserCrossFormatRouting (2) -- dispatch routing, signal coverage
TestLoopBasic loop integration (3) -- DeepSeek / GLM / Kimi end-to-end
Capability advertise (3) -- DeepSeek / GLM / Kimi templates flip
supports_tools=True
All 398 targeted tests pass locally (115 safetensors + 27 capability
+ rest of tool / inference / sandbox / model-config suites). Builds
on PR #5620 (parser + healing parity for Llama-3 / Mistral / Gemma 4);
will rebase cleanly onto main once #5620 lands. PR opened as draft -
do not merge until validated against real models for each family.
Sources
- llama.cpp common/chat-parser.cpp lines 801-913, 1040-1052 (MIT)
- vLLM vllm/tool_parsers/deepseekv31_tool_parser.py (Apache-2.0)
- vLLM vllm/tool_parsers/glm4_moe_tool_parser.py (Apache-2.0)
- vLLM vllm/tool_parsers/kimi_k2_tool_parser.py (Apache-2.0)
- SGLang python/sglang/srt/function_call/{deepseekv31,glm4_moe,kimik2}_
detector.py (Apache-2.0)
- Live chat templates: deepseek-ai/DeepSeek-V3.1, zai-org/GLM-4.6,
moonshotai/Kimi-K2-Instruct, unsloth/DeepSeek-V3-0324,
unsloth/GLM-4.5-Air, unsloth/Kimi-K2-Instruct
* studio/routes: make python_tag strip multi-line aware
Earlier revisions of _TOOL_XML_RE in studio.backend.routes.inference
oscillated between two bug shapes:
5615 r"<\|python_tag\|>[^\n<]*" -- stopped at any literal "<"
so code='if x < 10: pass'
leaked '< 10: pass)' to the
user.
5620.1 r"<\|python_tag\|>[^\n]*" -- single-line only; the second
line of
python.call(code="a\nb")
leaked.
The full parser (_parse_llama3_python_tag) already handles both via
balanced-brace scanning, so the parsing path was fine; the LEAK was
in the streaming strip path that runs on every cumulative emission
while content is still arriving.
Switch to r"<\|python_tag\|>(?:[^<]|<(?!\|))*" so the strip consumes:
* any character that is not a "<" (newlines, JSON, code, ...),
* a "<" only when it is NOT followed by "|" (i.e. NOT a Llama-3
sentinel start like <|eot_id|>, <|eom_id|>, <|begin_of_text|>).
This means:
* code='if x < 10' stays inside the strip (5615 fix preserved),
* multi-line code stays inside the strip (5620 round 2),
* the strip terminates at the next Llama-3 sentinel so trailing
assistant content survives.
Tests: TestRoutesPythonTagStrip (8 cases)
pytest test_safetensors_tool_loop.py test_safetensors_capability_advertise.py
-> 118 passed in 1.81s (was 110).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: review follow-ups for DeepSeek / GLM / Kimi tool calling
Four fixes addressing review of the parent commit:
1. GLM <arg_value> coercion: tighten the
json.loads -> ast.literal_eval -> raw cascade to only deserialize
when the body unambiguously looks like a JSON literal (object,
array, JSON-encoded string, true/false/null, or numeric). Strings
like ``True`` / ``None`` (Python literals, not JSON) and arbitrary
prose now stay raw. The bare-numeric / bare-boolean ambiguity with
string args remains an inherent limitation of the template without
schema access -- documented in the new comment. Drops the ast
import entirely (closes Gemini's :1036 suggestion).
2. Kimi K2 bare-counter ids (e.g. ``<|tool_call_begin|>3``) are now
dropped rather than surfaced as a tool literally named "3". Matches
vLLM behaviour; SGLang's schema-infer fallback is out of scope at
the parse site. Real Kimi K2 emissions use ``functions.NAME:IDX``
so this is the exception path.
3. Restore the elaborate ``<|python_tag|>(?:[^<]|<(?!\|))*`` clause in
routes.inference._TOOL_XML_RE -- the simpler ``[^\n<]*`` form
regressed PR #5620's multi-line / literal-``<`` python_tag fix.
Restore ``TestRoutesPythonTagStrip`` (8 tests) adapted to call
``_TOOL_XML_RE.sub`` directly since the ``_strip_tool_xml`` helper
was inlined this PR.
4. Add the spaced and backslash-escaped DeepSeek opener variants
(``<｜tool calls begin｜>``, ``<｜tool\_calls\_begin｜>``) to
``TOOL_XML_SIGNALS`` for streaming-gate parity with
``_DEEPSEEK_BEGIN_RE``.
Also updates the llama.cpp / vLLM citations in the parser docstrings:
``common/chat-parser.cpp`` was split into ``common/chat.cpp`` +
``common/chat-peg-parser.cpp`` by llama.cpp PR #18675, and vLLM
moved the tool parsers from ``vllm/entrypoints/openai/tool_parsers/``
to ``vllm/tool_parsers/``. Pin to pre-refactor commit ``51fa458a92d6``
where the cited line numbers still resolve.
New regression tests in ``test_pr5624_regressions.py`` cover the GLM
coercion heuristic shapes, GLM literal-``<`` in arg_value, Kimi K2
dotted name, Kimi K2 bare-counter drop, DeepSeek V3.1 truncated
mid-stream, and routes-layer strip across all three new families.
Tests:
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 170 passed in 1.91s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten verbose comments in tool-call parser sections
Comments were narrating what the code already says. Cut historical
"earlier revisions used X, then Y" narratives down to one-line WHY
notes where the footgun still matters (canonical heal-key parity,
balanced-brace vs non-greedy regex, ``(?:[^<]|<(?!\|))*`` over
``[^\n<]*``/``[^\n]*``). Drop section-header banners.
No behaviour change. Re-ran:
pytest studio/backend/tests/test_safetensors_tool_loop.py \
studio/backend/tests/test_safetensors_capability_advertise.py -q
-> 118 passed.
Regression replay (parser + _coerce_arguments on the 5 #5615 inputs)
-> 21/21.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: GLM 4.7 no-newline emission + Kimi multi-section parity
Two fixes surfaced by triple-confirm verification against the live
HF chat templates and upstream llama.cpp / vLLM / SGLang parsers.
1. GLM 4.7 silent drop
``zai-org/GLM-4.7/chat_template.jinja`` line 65 uses
``{{- '<tool_call>' + tc.name -}}`` which Jinja strips trailing
whitespace from, so the first ``<arg_key>`` follows the function
name with NO ``\n`` between them. Real emissions look like
``<tool_call>get_weather<arg_key>city</arg_key><arg_value>London
</arg_value></tool_call>``. The previous ``_GLM_TC_OPEN_RE`` ended
the name with ``\n`` so GLM-4.7 calls were silently dropped
(parser returned ``[]``).
Fix: relax the name terminator to a lookahead that accepts EITHER
``\n`` OR the next ``<arg_key>``:
_GLM_TC_OPEN_RE = re.compile(
r"<tool_call>\s*([^\n<{][^\n<]*?)\s*(?=\n|<arg_key>)"
)
The first-char restriction ``[^\n<{]`` still excludes Qwen's
``<tool_call>{json}`` form so the Qwen-vs-GLM dispatch remains
mutually exclusive.
2. Kimi multi-section parity with vLLM / SGLang
``vllm/tool_parsers/kimi_k2_tool_parser.py`` and SGLang's
``kimik2_detector.py`` both use ``re.findall`` and so collect every
``<|tool_calls_section_begin|>...<|tool_calls_section_end|>`` block
in a single stream. The previous implementation stopped at the
first ``<|tool_calls_section_end|>``. Kimi K2 doesn't emit
multi-section in practice, but parity is cheap.
Fix: wrap the existing per-call body parser in an outer loop that
advances past each ``<|tool_calls_section_end|>`` and continues to
the next ``<|tool_calls_section_begin|>``. Body parsing extracted
to ``_parse_kimi_section_body`` for clarity. Truncated final
section is still surfaced via the existing in-body balanced-brace
walk.
Verified independently against the live HF templates:
* GLM-4.7 emission constructed from the live template parses to the
expected ``{name, arguments}`` shape.
* GLM-4.5 / 4.6 newline shape continues to parse (the lookahead also
matches ``\n``).
* Qwen ``<tool_call>{json}`` still dispatches to the Qwen path -- the
first-char restriction stops the GLM regex from biting JSON bodies.
* Kimi two-section stream surfaces both calls in order with full ids
preserved.
* Bare-counter Kimi ids still drop.
Tests added in ``test_pr5624_regressions.py``:
* ``test_glm_4_7_no_newlines_between_name_and_arg_key``
* ``test_glm_4_7_no_newlines_multi_call``
* ``test_glm_4_7_does_not_break_qwen_path``
* ``test_kimi_two_sections_in_one_stream_both_parse``
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 174 passed in 1.93s
pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration'
-> 2038 passed, 15 failed (pre-existing CI gaps).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: parser robustness fixes for PR #5620
Three surgical extensions to the multi-format tool-call parser, each
covering a real fine-tune / template emission shape that the current
parser silently drops. No path narrows; all changes widen what is
accepted.
1. `_parse_tool_call_json` now accepts both `arguments` and
`parameters` keys. A Hermes / Qwen `<tool_call>{json}</tool_call>`
wrapper around a Llama-3.2 fine-tune that emits the `parameters`
key was extracting the tool name and silently discarding the
args, producing a working-shaped call with an empty payload. The
bare-JSON and python_tag paths already accepted both keys; this
path now matches them.
2. `_TC_FUNC_START_RE`, `_TC_PARAM_START_RE`, and `_TC_PARAM_CLOSE_RE`
now also match the attribute form
`<function name="..."><param name="...">v</param></function>` used
by MiniCPM-5 and MiniMax-M2. Names land in either capture group,
and `</param>` is accepted as a short close.
3. `_parse_llama3_bare_json` sentinel-strip now consumes the role
label inserted between `<|start_header_id|>` and
`<|end_header_id|>` by Meta's official Llama-3.x chat template.
Without this, every assistant turn re-fed through the template
prefix `<|start_header_id|>assistant<|end_header_id|>\n\n{json}`
parsed to zero calls, so any history-with-tool-call round-trip
in production silently dropped.
Tests in `studio/backend/tests/test_safetensors_tool_loop.py`:
* `TestParserRobustness::test_tool_call_json_accepts_parameters_key`
* `TestParserRobustness::test_function_xml_attribute_form`
* `TestParserRobustness::test_function_xml_attribute_form_multi_param`
* `TestParserRobustness::test_function_xml_legacy_equals_form_still_works`
(regression guard for the existing `<function=name>` syntax)
* `TestParserRobustness::test_llama3_chat_template_round_trip`
* `TestParserRobustness::test_llama3_round_trip_all_roles`
* `TestParserRobustness::test_llama3_round_trip_with_eot_prefix`
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 118 to 125 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose comments in tool-call parser sections for PR #5624
Pure comment / docstring tightening on top of the GLM 4.7 + Kimi
multi-section fixes. No behavioural change.
* Drop multi-paragraph prelude and post-refactor citation chatter in
the DeepSeek, GLM and Kimi parser docstrings; keep the shape and
upstream-commit pin.
* Collapse ``parse_tool_calls_from_text``'s 9 per-family blocks into
a single ordered loop with one combined comment.
* Tighten the GLM coercion, Kimi bare-counter and ``_TOOL_XML_RE``
comments to one or two lines each.
* Same trim pass on ``_PARSER_MARKERS`` and the regression-test
docstrings.
Tests:
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 174 passed in 2.00s
* Fix O(N^2) DeepSeek V3.1 backtracking for PR #5624
Adversarial input ``<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>fn<｜tool▁sep｜>``
followed by a long body that does NOT contain a closing brace caused
the V3 path's ``([^\n<]+?)<｜tool▁sep｜>`` regex to backtrack
quadratically: at each position the lazy quantifier extends one char
at a time looking for a sep that isn't there, taking ~19s on 50k
chars.
Replace the regex search with ``str.find`` on the sep marker plus a
left-walk to recover the name. ``str.find`` is O(N); the walk stops
on ``\n`` (turn boundary), ``<`` (start of a tag), or ``>`` (end of
an optional ``<｜tool▁call▁begin｜>`` prefix). Same observable
behaviour as the regex on every canonical input.
Tests:
test_deepseek_v3_1_huge_truncated_body_is_linear (new) -- 50k chars
must parse in < 1s.
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 175 passed in 1.97s
pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration'
-> 2038 passed, 15 pre-existing failures unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: terminate function-XML body at </function>, not just </tool_call>
`_parse_function_xml` was looking for `</tool_call>` (the Hermes
wrapper) as the body terminator. When a model emits a standalone
`<function=NAME><parameter=K>v</parameter></function>` followed by
explanatory prose (which models routinely do), no `</tool_call>` is
present, so the body extended to end-of-string and the trailing
prose leaked into the LAST parameter value.
Pre-existing on main (the legacy `<function=NAME>` form had this
bug too). Same affects PR #5620's new attribute-form
`<function name="NAME"><param name="K">v</param></function>`
emission used by MiniCPM-5 / MiniMax-M2.
Fix: `_TC_END_TAG_RE` now matches either `</tool_call>` OR
`</function>`. The existing `_TC_FUNC_CLOSE_RE` / `_TC_PARAM_CLOSE_RE`
strips are unchanged. Multi-call inputs still bound each function
at the next `<function=` start, so no over-eager consumption.
New tests:
* `test_function_xml_followed_by_prose` (legacy form + prose)
* `test_function_attribute_xml_followed_by_prose` (attribute form + prose)
Existing `test_code_with_embedded_xml` still passes (a parameter
value containing literal `<a></a>` is preserved because the
embedded close tag is `</a>`, not `</function>`).
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 125 to 127 passed.
* Studio: tighten Llama-3.2 bare-JSON guard
A fuzz pass on PR #5811 turned up that ``_parse_llama3_bare_json``
accepted ``parameters`` as a string, contradicting the docstring's
"parameters or arguments is a dict" guard. Prose JSON like
``{"name":"foo","parameters":"a sentence"}`` would wrongly fire the
parser, which the agentic loop would then heal into a real
``foo(query="a sentence")`` call.
Same code lives on this branch, so the same fix applies here.
Tightened guard:
- ``parameters`` must be a dict (Llama-3 spec).
- ``arguments`` may be a dict, or a JSON-encoded string that
decodes to a dict (OpenAI shape, e.g.
``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings or
JSON-strings of lists / scalars / null no longer pass.
Mirrors the fix landed in PR #5811 commit 615b8608. Adds the same
4 regression tests under TestParserMultiFormat.
Existing test suite stays green: 127 -> 131 passing.
* Studio: skip non-scalar args in python_tag JSON form
The JSON sub-path of ``_parse_llama3_python_tag`` was fabricating
``{"value": args}`` when the model emitted a non-dict / non-string
``arguments`` value (e.g. ``42``, ``[1,2,3]``, ``null``, ``true``).
This silently turned a malformed emission into a real tool call,
which the agentic loop would then execute with arguments the model
never intended.
Tightened: skip the call instead of fabricating. The same
behaviour now matches the bare-JSON guard tightened earlier
(strict-guard merge from PR #5620, inherited via merge here).
Added a regression test covering the four non-scalar shapes.
Pass count on this branch: 158 -> 159.
Sites in ``_parse_tool_call_json`` and ``_consume_mistral_call``
keep the existing looser behaviour for now; both are reached
only after explicit ``<tool_call>`` / ``[TOOL_CALLS]`` markers
so the false-positive surface there is much narrower.
* studio: fix safetensors tool-call parser gaps vs llama.cpp (Mistral CALL_ID / THINK, attribute-form signal)
Three GGUF-parity fixes to the safetensors tool-call parser, each matching
llama.cpp's reference behaviour:
- Mistral Small 3.2 emits [TOOL_CALLS]name[CALL_ID]<id>[ARGS]{json}. The
parser stopped after the name on seeing [CALL_ID] (neither [ARGS] nor {),
dropping the call. Skip an optional [CALL_ID]<id> segment in both the
parse and strip paths. llama.cpp parses this (test-chat.cpp:4785).
- Magistral wraps reasoning in [THINK]...[/THINK]. A [TOOL_CALLS] inside the
reasoning was parsed as a real call, producing a phantom call. Strip a
leading [THINK] block before scanning so only the post-reasoning call
counts (test-chat.cpp:2285); a literal [THINK] inside a later argument is
left intact.
- The standalone MiniCPM-5 / MiniMax-M2 <function name="..."> attribute form
parsed correctly but was absent from TOOL_XML_SIGNALS and the markup strip
patterns, so the streaming safety-net parse was gated off (dropping the
call) and markup leaked into displayed text. Add the signal and broaden
the strip regexes.
Adds regression tests for all three.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix GLM and Kimi K2 safetensors tool-call parser gaps vs llama.cpp
Four GGUF-parity fixes for the GLM and Kimi K2 families:
- GLM 4.7 zero-argument inline call <tool_call>name</tool_call> was dropped:
the open-tag lookahead only allowed \n or <arg_key> after the name. Allow
</tool_call> too so a no-arg call parses to empty args (vLLM / SGLang /
llama.cpp all parse it).
- GLM string argument values were stripped, losing significant leading /
trailing whitespace in code / diff arguments. Keep the raw value for the
string fallback and only strip the copy used to probe for a JSON literal,
matching vLLM glm4_moe which never strips string args.
- Kimi K2 calls emitted without the <|tool_calls_section_begin|> wrapper
were dropped. llama.cpp makes the section optional (Kimi can call a tool
straight after reasoning without opening a section); parse a bare
<|tool_call_begin|> when no section is present.
- Kimi K2 malformed / truncated JSON in one call dropped every later call in
the section. Skip the bad call and keep parsing so valid subsequent calls
are recovered (vLLM parity).
Adds regression tests for all four.
* studio: fire safetensors tool calls for the bare-JSON (Llama-3.2) form
The agentic loop's streaming safety-net parse was gated on
has_tool_signal(), which is False for the Llama-3.1 / 3.2 bare-JSON tool
form {"name":..,"parameters":..} (no XML marker). Real tool calls were
therefore dropped: the loop logged "model planned without calling tools",
re-prompted three times, then gave up with zero tool calls, while GGUF's
llama-server parses the same emission natively.
Run parse_tool_calls_from_text() unconditionally in the safety net. The
parser is strict (only fires on a valid tool-call shape) so plain answers
are unaffected. Reproduced on a real unsloth/Llama-3.1-8B-Instruct run:
the model emits {"name":"web_search","parameters":{...}} which now
executes the tool instead of being re-prompted into a no-op.
Adds a loop regression test for the bare-JSON form.
* studio: fire safetensors tool calls for Gemma 4 (native template + stripped parser)
Gemma-4 safetensors fired no tools while its GGUF fired reliably. Three gaps:
- The Studio swaps in the Unsloth "gemma-4" chat template, which does not
render the tools schema (the model's native template does), so the model
never saw the tools. Fall back to the model's native template when the
override template renders identically with and without tools. Same fix
helps any family whose override template drops tools.
- skip_special_tokens strips the <|tool_call> wrapper and <|"|> string
markers, so a streamed Gemma-4 call arrives as a bare call:NAME{k:v, ...}
with unquoted values. Parse that form, keeping commas/braces inside a
code or command value, normalising surrounding quotes, and stripping the
leaked markup from the final answer.
- Without a grammar a small model can loop, repeating one call for the whole
tool budget. Collapse exact-duplicate calls within a turn and force a final
answer after a turn that made no new tool progress (llama-server's lazy
grammar prevents this loop on the GGUF side).
Adds parser tests for the bare/stripped Gemma-4 form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: complete strict-mode contract and fix parser import paths
Address review findings on the multi-format tool-call parser:
- Honor allow_incomplete=False in the remaining sub-parsers. The Llama-3
<|python_tag|>NAME.call(...) parser, the pre-v11 Mistral [TOOL_CALLS] array
parser, and the Gemma 4 <|tool_call> parser ignored strict mode, so a
truncated call (missing closing paren, ], or <tool_call|>) was still healed
and executed with Auto-Heal disabled. Thread strictness through and reject
the unclosed forms, matching the JSON and function-XML paths.
- Drop the duplicate tool_call_parser import block in llama_cpp.py and the
redundant un-aliased TOOL_XML_SIGNALS; only the _SHARED_TOOL_XML_SIGNALS
alias is used as a value.
- Import _strip_mistral_closed_calls from core.inference.tool_call_parser in
routes/inference.py instead of studio.backend.core... The self-contained
run.py launch mode only puts studio/backend on sys.path, so the absolute
package path raised ModuleNotFoundError on the server-tool strip path.
Add strict-mode regression tests for the truncated Llama-3 dot-call and the
unclosed Mistral array.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DeepSeek/Kimi tool-call parsing and strip
Address review findings on the DeepSeek and Kimi parsers:
- Honor allow_incomplete=False for DeepSeek. An envelope with no closing
<｜tool▁calls▁end｜> is truncated mid-stream; reject it in strict mode
instead of healing the body out to EOF, matching the strict XML and Mistral
paths.
- Do not skip a following tool call when the current call's end marker is
missing. The DeepSeek V3 and Kimi loops advanced by searching forward for the
next <｜tool▁call▁end｜> / <|tool_call_end|>, which could land on a later
call's end marker and drop the call in between. Advance by the JSON end; the
loop re-locates the next call marker from there.
- Strip truncated DeepSeek and Kimi section blocks in the route-level display
regex. The patterns required the closing marker; add the end-of-text
alternative so a block truncated by EOS does not leak raw markup to the UI.
Add regression tests for the truncated DeepSeek envelope, and for DeepSeek and
Kimi multi-call recovery when the first call's end marker is missing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve XML param indentation and alias Mistral array parameters
Two parser-correctness fixes found by auditing against the model chat templates
and the SGLang / vLLM reference parsers:
- Qwen3.5 XML parameter values lost their leading indentation. The chat template
emits <parameter=k>\nVALUE\n</parameter>, but the parameter-start regex ate the
wrapping newline AND the value's first-line indentation with a trailing \s*,
then str.strip() removed the rest. Narrow the trailing class to horizontal
whitespace only and trim exactly one wrapping newline (via _trim_param_value),
preserving indentation in code/diff arguments. Matches SGLang's qwen3_coder
detector. Applies to both _parse_function_xml (tool_call_parser.py) and the XML
path in tool_healing.py.
- Mistral pre-v11 array objects keyed on parameters dropped their payload.
_consume_mistral_call read only the arguments key; alias parameters the same way
the JSON/XML paths and SGLang's base detector do.
Add regression tests for preserved multi-line indentation and the array
parameters alias.
* Studio: DeepSeek strip sync, Gemma nested args, GLM/Kimi strict mode
Parser-correctness fixes found by auditing DeepSeek/GLM/Kimi against vLLM,
SGLang, and the model chat templates:
- DeepSeek: the short <｜tool▁calls｜> opener (and the space / escaped-underscore
spellings) was parsed but never stripped, so a short-opener envelope leaked raw
markup to the UI. Share one opener alternation between _DEEPSEEK_BEGIN_RE and
the strip patterns (and the route-level display regex) so a signal we parse can
never be left un-stripped.
- Gemma wrapper-less stream: a nested object/array argument (loc:{city:NYC},
labels:[bug,ui]) was kept as a literal string. Parse it recursively when the
bare value is a balanced {} / [], falling back to the raw string for a
truncated value.
- GLM and Kimi ignored allow_incomplete. With Auto-Heal off, a GLM block with no
</tool_call>, a Kimi section with no <|tool_calls_section_end|>, or a Kimi call
with no <|tool_call_end|> are truncated and must be rejected, matching the
strict behavior of the JSON/XML/Mistral/DeepSeek paths and vLLM/SGLang.
Add regression tests for the short-opener strip, the Gemma nested args, and GLM /
Kimi strict-mode rejection.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten tool-call parser comments
Make the comments in the multi-format tool-call parser and its callers succinct:
compress verbose docstrings/blocks to one or two lines, drop ones that restate the
code, and trim the tiny balanced-scanner helpers. Correctness rationale and
upstream provenance (SGLang/llama.cpp parity, the strict-mode / Auto-Heal
contract, whitespace-preservation, and the Unicode / full-width-pipe notes) are
kept in compact form.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; parser suite green).
* Studio: tighten DeepSeek/GLM/Kimi parser comments
Compress the comments added for the DeepSeek/GLM/Kimi parsers and the Gemma
wrapper-less helpers to one or two lines, keeping the upstream provenance
(llama.cpp 51fa458a92d6), the O(N^2) / strict-mode rationale, and the vLLM parity
notes intact.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; parser suite green).
* Studio: make DeepSeek R1 / GLM parsing linear and close routes strip gaps
Review follow-up for the DeepSeek/GLM/Kimi parser:
- DeepSeek R1 detection used a greedy ``([^\n]+)\n```json`` regex that backtracks
O(N^2) on a fence-less truncated body; scan with str.find instead (mirrors the
V3 path).
- GLM arg pairs used a lazy-group finditer that rescanned to EOF from each bare
<arg_key> in an unclosed body (O(N^2)); walk pairs with str.find.
- The route display strip (_TOOL_XML_RE) accepted fewer DeepSeek openers than the
parser (missed the space / escaped-underscore spellings) and missed bare
section-less Kimi calls, so a call we parse could leak raw markup to the UI.
Reuse the parser's shared _DEEPSEEK_OPEN_RE_SRC and add a bare-Kimi arm.
Add ReDoS-linearity regressions for the R1 and GLM paths, a positive R1
fenced-json parse test, and routes-strip tests for the space/escaped DeepSeek
openers and the bare Kimi call.
* Studio: fix test_mcp_servers _TOOL_XML_RE reconstruction after _DS_OPEN_SRC reuse
The routes strip fix made _TOOL_XML_RE reference the module-level
_DS_OPEN_SRC variable. test_mcp_servers reconstructs the regex by exec-ing
the extracted compile() source in a namespace that only defined _re, so it
raised NameError. Inject _DS_OPEN_SRC into that namespace, matching the same
fix already applied in test_tool_xml_strip.
* Studio: make Llama-3 .call and Mistral-array healing parsing linear
Two more O(n^2) ReDoS paths in the multi-format parser, both reachable from
the agentic loop on a long truncated body with no length cap:
- _LLAMA3_KV_RE.finditer over a .call(...) body retried at every offset of a
long word run / unterminated quote (40K -> 14s). Replace with a hand-scan
that reuses the same key/number/literal sub-regexes via anchored match and
walks the string body by hand, so an unterminated quote is O(n). Verified
byte-identical to the old regex over 200K fuzzed inputs.
- _parse_mistral_array healing ran _balanced_brace_end from every { in the
body (20K -> 17s). Walk top-level objects, advancing past each balanced
{...}; this also drops the phantom call the old scan emitted from a nested
argument object.
Add adversarial-length linearity regressions plus positive .call kwargs and
unclosed-array recovery coverage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strengthen #5624 regression assertions and strip-test harness guards
- test_strip_tool_markup_handles_deepseek_envelope used `A or B` where B was the
preservation property the next line already asserts, masking the real check.
Replace with an explicit assertion that the call name and args are stripped.
- The test_tool_xml_strip source-extraction harness reconstructs _TOOL_XML_RE and
_strip_tool_xml_for_display from routes/inference.py via lazy regexes that could
silently grab a shorter slice. Assert the extracted regex carries the DeepSeek /
bare-Kimi arms and the helper body reached the _TOOL_XML_RE.sub call.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor strict mode in safety-net, keep empty Gemma args, strip attribute-form function XML
- safetensors safety-net parser now forwards allow_incomplete=auto_heal_tool_calls,
matching the draining path, so a late incomplete tool call is not healed and
executed when Auto-Heal is off.
- Gemma empty bare value ({k:}) now serialises as "" instead of invalid {"k":},
which previously dropped the whole call.
- Route _TOOL_XML_RE also strips the <function name="..."> attribute form
(MiniCPM-5 / MiniMax-M2) so it no longer leaks to the UI.
* Studio: linearize wrapper-less Gemma nested-arg parsing and correct parser provenance
- _gemma_parse_value/_gemma_parse_mapping/_gemma_parse_array now parse nested
{}/[] in a single forward pass instead of pre-scanning each subtree with a
balanced-brace walk and re-parsing it. Deeply nested wrapper-less Gemma args
were O(n^2); they are now ~linear (and ~40x faster at depth 400).
- Correct the DeepSeek/GLM/Kimi provenance comments: the cited commit
51fa458a92d6 is unrelated, and GLM/Kimi were never standalone
common_chat_parse_* functions (llama.cpp uses common_chat_params_init_glm_4_5
plus a generalized XML parser, PRs #15904 / #16932).
- Add tests: Gemma deep-nesting linearity, nested object/array preservation,
same-turn distinct-call cap, and the native-template tool-render fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: guard Gemma value parser against non-advancement and missing tokenizer
Addresses Gemini review:
- _gemma_parse_value now consumes one character when a stray }/]/, sits where a
value is expected, so _gemma_parse_array can never stall at the same index on
malformed input (a latent infinite loop).
- _render_with_native_template returns None when neither a tokenizer nor a
processor is present instead of raising AttributeError.
- Tests for both.
* Studio: fix attribute-form function-XML literal close tag and zero-arg strict call
Addresses Codex review of the <function name="..."> attribute form in
_parse_function_xml (MiniCPM-5 / MiniMax-M2):
- End the call body at the LAST </function> / </tool_call> within the call's
window, so a literal close tag inside a code/search argument (e.g.
print("</function>")) is preserved instead of truncating the call.
- Accept a closed call with no parameters as a valid zero-argument call in strict
mode (the function close is already required), instead of rejecting it as a
truncated call.
- Tests for both, mirroring the legacy <function=...> coverage.
* Studio: drop scratch review/planning artifacts from the branch
* Studio: fix tool-call parser/loop review findings on the multi-format path
Address the live code-review findings on the safetensors/MLX + GGUF tool path:
- routes: include the attribute form <function name="..."> in the safetensors
capability whitelist so MiniCPM-5 / MiniMax-M2 templates keep the tool pill
(parser already handles the form; the post-filter wrongly suppressed it).
- safetensors loop: build the plan-without-action re-prompt from the active
tools instead of a hardcoded web_search/python string, and gate it on
auto_heal_tool_calls, matching the GGUF loop.
- safetensors loop: hold a leading bare-JSON object ({"name":..,"parameters":..})
during BUFFERING until it closes, then drain it as a tool call instead of
streaming the raw JSON to clients. The DRAINING/STREAMING resolvers still
recover a plain JSON answer, so this can never drop content.
- parser: anchor the Llama-3 <|python_tag|>NAME.call(...) scan to the tag and
chain ; -separated calls, so all semicolon-separated built-ins parse and a
literal <|python_tag|>x.call(...) inside a JSON string argument no longer
fires the wrong tool.
- parser: consume the optional trailing </s> after a named Mistral
[TOOL_CALLS]name{json} call, mirroring the array shape.
- GGUF streaming strip: use the shared parser patterns (which know
[TOOL_CALLS] and <|python_tag|>) so a textual tool call entering DRAINING is
stripped instead of leaking the marker to streaming clients.
- routes: hoist the _strip_mistral_closed_calls import to module level.
Adds regression tests covering each fix; existing parser suite stays green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix DeepSeek/GLM/Gemma tool-call review findings
Address the live code-review findings specific to the DeepSeek / GLM / Kimi
and native-template additions:
- parser: in strict mode (Auto-Heal off) require the per-call
<｜tool▁call｜end｜> terminator for DeepSeek V3 calls instead of executing on
a bare balanced object closed only by the envelope end.
- parser: keep GLM string arguments that begin with a quote verbatim (drop
the leading-quote case from the JSON-decode probe) so a quoted search query
is not decoded down to its inner text.
- parser: reject a GLM call with an unclosed <arg_value> in strict mode, and
under Auto-Heal keep the partial value rather than dropping it to a no-arg
call.
- parser: add a balanced wrapper-less Gemma strip (call:NAME{...}) so a nested
object/array argument is removed whole instead of leaving a trailing brace;
run the balanced Mistral and Gemma strips on the streaming display paths too.
- safetensors loop: buffer a leading wrapper-less Gemma call:NAME{...} so it
drains and executes instead of streaming the raw call text.
- inference: render the native-template fallback on a shallow tokenizer copy
instead of mutating the shared tokenizer outside the generation lock, and
load the native template from base_model for LoRA adapters.
Adds regression tests for each; existing parser suite stays green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden multi-format tool-call detection from review findings
Apply five targeted fixes from the review pass over the multi-format tool
path:
- routes: route display strip delegates to _strip_tool_xml so Mistral
[TOOL_CALLS] blocks with nested JSON are removed from streamed display
text, not just the XML forms.
- tool_call_parser: skip function/parameter starts that fall inside an
already-open parameter block (_inside_open_parameter) so nested example
payloads are not mis-parsed as new calls; extract
strip_llama3_leading_sentinels so the bare-JSON guard is shared.
- safetensors_agentic: probe bare JSON through strip_llama3_leading_sentinels
before the balanced-brace check so a leaked header sentinel does not defeat
the guard.
- tool_healing: allow dotted tool names in the Gemma wrapped start pattern.
- llama_cpp (GGUF): buffer wrapper-less Llama-3.2 {"name":..} calls that carry
no XML signal, drain a complete object silently and hold an incomplete one,
and run the end-of-stream safety net unconditionally so markerless calls are
detected and never leak the raw JSON (including truncated fragments).
Adds regression tests for the GGUF bare-JSON streaming path and the Mistral
display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop bare-JSON tool calls leaking at EOF, oversized, and into history
The second review pass flagged that the Llama-3.2 bare-JSON tool-call handling
still leaked raw JSON in several spots; ``strip_tool_markup`` only knows
XML/bracket markup, so the bare-JSON form survived it. Fix them symmetrically
across the safetensors and GGUF loops:
- Safetensors stream-end resolver now routes a held bare-JSON fragment to
DRAINING (mirroring GGUF) so a truncated ``{"name":..`` cut off by the end of
the stream is dropped instead of flushed as assistant content. The 7/10
reviewer finding.
- Both loops now drain (suppress) an oversized still-open bare-JSON call once it
passes ``_MAX_BARE_JSON_BUFFER`` instead of streaming the raw prefix, gated on
a ``"name"`` key so a giant plain JSON answer still streams; a complete
oversized call still executes via the safety net.
- Add a shared ``strip_leading_bare_json_call`` helper and apply it to the
content kept for the assistant turn in both loops, so an executed bare-JSON
call is not replayed as visible text or fed back as next-turn history.
Plain JSON answers without a ``"name"`` key are untouched throughout. Adds
regression tests for the EOF, oversized, and next-turn cases on both backends
plus unit tests for the helper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the Llama-3 python_tag strip on real control sentinels
The route display strip's <|python_tag|> arm ran to the next <| of any kind.
A tool-call argument carrying a literal <|...|> token (for example <|cite|>
inside a string value) truncated the strip early and leaked the call tail into
the visible response. Narrow the stop condition to the genuine Llama control
sentinels (eot_id, eom_id, python_tag, start/end_header_id, begin_of_text,
finetune_right_pad_id) so embedded markup and JSON are consumed while real
header/turn boundaries still bound the strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden GLM/Gemma parsing, cap GGUF textual calls, share native-template fallback
GLM 4.x parser walked a body pre-bounded by the first </tool_call>, so a string
argument containing a literal </tool_call> (e.g. code that prints it) was
truncated. Walk arg_key/arg_value pairs against the full content instead, since
each <arg_value> is delimited by its own </arg_value> and the call's real close
is the </tool_call> that precedes the next <arg_key>.
Add a truncated wrapper-less Gemma pattern (call:NAME{... with no closing brace)
to the markup strip so a call cut off mid-arguments does not leak raw into the
visible stream. It runs after the closed form, so a complete call keeps trailing
prose.
Cap and dedup tool calls parsed from the GGUF TEXTUAL fallback at
_MAX_TOOL_CALLS_PER_TURN, mirroring the safetensors loop. Structured
delta.tool_calls are grammar-bounded by llama-server, but text parsed straight
from content is not, so one runaway turn could fan out into dozens of
executions.
Extract the native-chat-template fallback into chat_template_helpers
(render_native_template / render_with_native_template_fallback) so the
transformers and MLX text backends share one implementation. The MLX text path
now applies it too, so an Unsloth override template that drops the tools schema
no longer silently stops MLX from advertising tools. The MLX VLM path renders
via the processor for image tokens and is intentionally left on its own render.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate markerless bare JSON on enabled tools and close parser/strip asymmetries
The Llama-3.2 custom_tools bare-JSON form has no marker, so any JSON object with a
name key was read as a tool call. An ordinary JSON answer like
{"name":"Alice","parameters":{"age":30}} was misclassified as a call to a
disabled tool and dropped from the visible response. Gate the markerless form on
the enabled tool names (threaded through parse_tool_calls_from_text and
strip_leading_bare_json_call, supplied by both streaming loops): an object whose
name is not an enabled tool is ordinary content. The marker-based forms keep
their name-agnostic behaviour (an explicit signal is a real call attempt), and
unrestricted mode stays ungated.
Also fix two parser/strip asymmetries the parser already tolerated:
- A literal </function> inside a parameter value (print("</function>")) truncated
both the core and route strips at the first close, leaking the tail. Extend the
strip to the call's real close (last </function> before the next opener),
mirroring the parser, without merging separate calls.
- The single-object Mistral [TOOL_CALLS]{...} shape parsed but _strip_mistral_closed_calls
left it, leaking the raw object into display. Strip the balanced object while
keeping trailing prose, matching the array and name shapes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: fix strip/parse symmetry and native-template token for DeepSeek/GLM/Kimi
Pass-3 review follow-ups on the multi-format tool parser:
- Bare Kimi call (<|tool_call_begin|>...<|tool_call_end|> with no section
wrapper) is accepted by the parser, so add it to the closed strip patterns
so the streaming (non-final) display strip removes it instead of leaking the
markup mid-generation.
- Route display strip now also runs the wrapper-less Gemma cleanup, so a
Gemma 4 call:NAME{..} no longer leaks into the visible answer.
- MLX model record carries base_model for a LoRA adapter so the native-template
fallback loads the base repo template rather than the adapter's
(often template-less) tokenizer.
- Native-template reload forwards the load-time HF token so a gated/private
model's repo template can still be fetched (transformers and MLX text paths).
- GGUF end-of-stream bare-call heuristic is gated on the enabled tool names so a
truncated ordinary JSON object ({"name":"Alice","age":) streams as the answer
instead of being dropped as a tool call.
Adds regression tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: gate GGUF bare-JSON suppression on enabled tools and fix python-tag exponent parsing
Pass-4 review follow-ups on the GGUF tool loop and Llama-3 parser:
- The GGUF bare-JSON suppression sites still keyed off a raw "name" substring,
so an ordinary JSON answer whose name is not an enabled tool was dropped when
it was truncated, oversized, or reached the no-tool DRAINING fallback (the
parser, helper, and safetensors paths were already gated). All three sites now
use the shared enabled-name gate, and a held bare-JSON buffer that turns out not
to be an enabled call is shown as the answer instead of dropped at stream end.
- The Llama-3 python-tag numeric kwarg regex matched only the mantissa, so
scientific notation was truncated to its leading digits (1e-3 parsed as 1) and a
tool executed with the wrong value. The regex now accepts exponent and decimal
forms, and the int/float classification keys off the exponent too.
Adds regression tests for the truncated / oversized disabled-name JSON cases (and
a counterpart that a truncated enabled call still does not leak) plus the
scientific-notation kwargs.
* Studio: drop accidentally committed async worker transcripts
Eight generated reviewer / async-worker transcripts were committed under
studio/backend/async_task_outputs/. They are not imported or referenced by any
code and carry only internal task state, so they should never ship in the repo.
Remove them and gitignore the directory so they cannot be re-added.
* Studio tools: gate safetensors bare-JSON drain, fix nested-name gate and function-XML strip
Pass-4 review follow-ups on the shared parser / safetensors loop:
- The safetensors oversized and end-of-stream bare-JSON drain branches keyed off
a raw "name" substring, so a large or truncated ordinary JSON answer whose name
is not an enabled tool was drained instead of streamed. Both now use the shared
enabled-tool-name gate, matching the GGUF path.
- strip_leading_bare_json_call matched the first "name" anywhere, so a plain JSON
answer with a nested name equal to an enabled tool ({"result":{"name":"web_search"}})
was wrongly suppressed. It now extracts the TOP-LEVEL name only, walking past
nested objects/arrays and keeping the text when a top-level value is truncated.
- The function-XML display strip used a regex negative-lookahead that stopped at a
literal <function=...> opener inside a parameter value and then dropped the rest
of the answer to EOF. A scan-based strip mirrors the parser (ignores openers
inside an open <parameter> via _inside_open_parameter) and closes each call at its
real </function>, so trailing assistant text after such a call survives.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep tools prompt when native-template probe raises; make helper tests hermetic
Pass-4 review follow-ups on the native-template fallback:
- render_with_native_template_fallback re-renders the live template with tools=None
to detect whether it dropped the schema. A template that requires tools can raise
on that probe; that must not discard the already-valid tools prompt. The probe is
now wrapped so any error returns the original formatted_prompt (transformers would
otherwise fall back to manual formatting and lose the schema; MLX would let the
exception escape).
- The native-template helper tests imported InferenceBackend just to reach the
thin wrapper, which pulls in unsloth and its optional vllm package metadata. They
now call the dependency-light render_native_template helper directly so they pass
in a backend/test environment without vllm. Adds a probe-raises regression test.
* Tool parsing: 3.9 import safety, disabled-Auto-Heal contract, capability gate
Round-2 review follow-ups on the multi-format tool-call parser:
- tool_call_parser: add `from __future__ import annotations`. The module
is dependency-light by design (external llama-server wrappers import it
standalone) and the package targets python >=3.9, where its PEP 604
`int | None` return annotations would raise TypeError on import.
- safetensors + GGUF drain fallback: gate the leading bare-JSON strip on
auto_heal_tool_calls. With Auto-Heal off, a truncated enabled-name
fragment that did not parse now stays visible, matching the XML strip
in the same branch and the disabled-Auto-Heal contract. With Auto-Heal
on it is still suppressed.
- safetensors capability gate: match the bare-JSON `{"name":` template
marker with a whitespace/escape-tolerant regex so a pretty-printed
`{ "name" :` or JSON-escaped `{\"name\":` template is not mis-classified
as tool-less. The parser already accepts that whitespace via
raw_decode, so the gate must too.
Regression tests added for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GLM tool-call display strip: treat literal close tag in arg value as data
Round-2 review follow-up on the GLM 4.x tool-call format.
The GLM call shape is <tool_call>NAME<arg_key>k</arg_key><arg_value>v
</arg_value>...</tool_call>. The parser was hardened to walk arg_key /
arg_value pairs so a literal </tool_call> inside an argument value (e.g.
print("</tool_call>")) is treated as data and the call's real close is the
</tool_call> that precedes the next <arg_key>. The display strips still used a
non-greedy <tool_call>.*?</tool_call> regex, which stopped at the literal and
leaked the call's tail into visible content and stale history.
Add _strip_glm_calls, a scan that mirrors the parser's close detection, and run
it before the regex arms in every strip pipeline: the core strip_tool_markup,
the route _strip_tool_xml display/history cleanup, and the safetensors + GGUF
streaming strips. Qwen / Hermes <tool_call>{json} has no NAME token after the
opener, so it is left to the regex arms unchanged.
Regression tests cover the literal-close-tag leak (core + route), normal GLM
calls, back-to-back GLM calls, zero-arg GLM, truncated GLM, and untouched Qwen.
* Tool parsing: symmetric "function" bare-JSON alias and route strip parity
Round-3 review follow-ups, all parser/strip symmetry fixes.
- Bare-JSON "function" alias: the markerless parser accepts a call name via
obj.get("name") or obj.get("function"), but the strip/gates only knew "name",
so a {"function":<enabled tool>} call executed while its raw JSON leaked. Teach
_top_level_bare_json_name the alias (with "name" precedence and the same nested
and truncated-name guards), and widen the guards in strip_leading_bare_json_call,
the safetensors and GGUF _looks_like_enabled_bare_json gates, and the route
capability marker regex.
- Route display/history cleanup: strip a tail-only </param> alias close (the
parser accepts <param name="...">...</param>), and run the parser's guarded
function-XML scan (_inside_open_parameter) before _TOOL_XML_RE so a literal
nested <function=...></function> inside an argument value does not truncate the
strip and leak the tail.
Regression tests added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: fix DeepSeek strict recovery, Kimi dotted names, Gemma spaced streaming
Round 3 review fixes for the DeepSeek / GLM / Kimi tool-call parsing path.
- DeepSeek R1 and V3/V3.1 strict parsing (Auto-Heal off): when a call is
truncated (missing closing fence or <tool_call_end> terminator), skip it
and keep scanning for later well-formed calls instead of breaking out and
dropping the rest of the envelope. This matches the Kimi strict parser's
recovery behaviour.
- Kimi dotted tool names: keep the full name after stripping only the
functions. prefix and :idx suffix, e.g. functions.mcp.server-list:0 stays
mcp.server-list. The previous split on "." truncated dotted MCP names to
their last segment. This matches current vLLM
(tool_id.split(":")[0].removeprefix("functions.")) and SGLang
(^(?:functions\.)?(?P<name>[\w.\-]+):(?P<index>\d+)$).
- Gemma wrapper-less call streaming: hold the whitespace-tolerant prefix
(call : NAME) in the streaming suppression buffer, matching the parser's
_GEMMA_BARE_TC_RE, so the spaced spelling split across chunks is buffered
instead of leaking as visible text. Applied to both the safetensors and
llama.cpp streaming paths.
- Remove dead _render_with_native_template method and the now-unused copy
import from inference.py; the live path uses render_with_native_template_fallback.
Adds regression tests for DeepSeek R1/V3 strict recovery, Kimi full dotted
name preservation, and the Gemma spaced-call streaming suppression.
* Studio tools: honor tool budget in GGUF loop and guard function-XML streaming strip
Round 4 review fixes. Both are asymmetric-fix bugs where the final/steady path got a
guard the analogous streaming/loop path did not.
- GGUF tool-call budget: the safetensors loop counts real tool-call turns against
max_tool_iterations (re-prompt stalls excepted), but the GGUF loop only bounded the
turn count by the enlarged range (max_tool_iterations + _MAX_REPROMPTS). Since this
PR raised _MAX_REPROMPTS from 1 to 3, a model that keeps making valid tool calls
could run up to three extra tool rounds (with max_tool_iterations=1, four rounds
instead of one). Add a _tool_iters_done counter that increments only when a tool
actually executed in the turn, and stop once the caller's budget is spent so the
post-loop final-answer nudge fires. A duplicate/disabled no-op turn is a correction
turn (like a plan-without-action re-prompt) and does not consume budget, preserving
the existing "already completed" re-prompt behavior.
- Streaming display strip: the final strip runs the guarded _strip_function_xml_calls
scanner (a literal <function=...> inside a parameter value is data, not a nested
call), but the GGUF and safetensors streaming strips still used only the open-ended
regex arms. When a tool-call argument contained literal function markup, the regex
tail ate everything to end-of-text and dropped the real trailing prose after the
call's true </function>. Run the guarded scanner (and the balanced Mistral strip)
before the regex arms in both streaming paths so streaming and final display agree.
Adds regression tests: GGUF valid tool calls respect max_tool_iterations, and the
streaming strip keeps trailing prose after a function-XML call with a literal marker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: safetensors tool budget counts only executed turns (GGUF parity)
Follow-up to the GGUF budget fix. The safetensors loop charged max_tool_iterations
per non-re-prompt iteration (iteration + 1 - reprompt_count), so a duplicate/disabled
no-op turn spent a budget slot even though no tool ran. With a small cap this dropped
real work: for max_tool_iterations=2, a model that made a valid call, repeated it (an
internal no-op correction turn), then made a distinct valid call executed only the
first -- the third turn was sent with no tools and the distinct call was ignored.
Track whether a turn actually executed a tool (set on record_result) and count only
those turns against the cap, matching the GGUF loop. A duplicate/disabled no-op is a
correction turn -- like a plan-without-action re-prompt -- and no longer consumes
budget, so the model still gets its "already completed" nudge and another tool-enabled
turn. Adds a regression test for the small-cap duplicate-then-distinct-call flow.
* Studio tools: fix stale Kimi dotted-name regression test
test_pr5624_regressions.py still expected functions.my.tool:0 to resolve to the last
segment (tool). The parser now preserves the full dotted name (my.tool) after removing
only the functions. prefix and :idx suffix, matching current vLLM/SGLang so dotted MCP
names like mcp.server-list survive. Update the assertion, name, and module docstring to
the corrected contract (the raw id is still preserved on the call).
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: don't force a tool re-prompt on a negated intent (safetensors parity)
The safetensors _INTENT_SIGNAL claimed to mirror GGUF but was missing the
negative lookahead, so a refusal like "I will not search the web for that"
matched the "i will" intent and triggered the plan-without-action re-prompt
(STOP... you MUST call a tool), overriding a valid no-tool answer. GGUF already
excludes not/never. Add the same (?!\s+(?:not|never)\b) lookahead so both
backends agree. Extends the intent parity test with negated refusals.
* studio: parse the outer envelope before DeepSeek/Kimi markers embedded in its args
parse_tool_calls_from_text ran the DeepSeek/Kimi marker pre-pass before the shared
<tool_call>/<function=...> parser. When a Qwen/Hermes call's argument contained
literal Kimi/DeepSeek markup (for example a user asking the model to explain that
syntax), the pre-pass matched the embedded marker and returned it, executing the
wrong tool and dropping the real call. Skip the pre-pass when a <tool_call> or
<function=...> envelope opens before the first DeepSeek/Kimi marker, so the shared
parser takes the outer call; a genuine marker-led call (no leading envelope) still
goes through the pre-pass. Tests for the embedded-marker case and the control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: trim redundant comments (comment-only, AST-verified)
* Studio: trim redundant comments (comment-only, AST-verified)
* Studio: prevent Gemma tool-parser DoS on stray delimiters
_gemma_parse_value returned the input index unchanged when text[i] was a
stray delimiter (,}]), so the list and mapping caller loops that advance
on the returned index spun forever at 100% CPU on malformed input such as
[},]. Advance past the delimiter so parsing always terminates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip Magistral [THINK] reasoning from final display/history
strip_tool_markup removed [TOOL_CALLS] and <function> markup but left a
leading Magistral [THINK]...[/THINK] block intact, so its bracket-form
reasoning (not the <think> the reasoning channel renders) leaked into the
safetensors display and conversation history while GGUF/llama.cpp routes
it natively. Drop the leading reasoning block at end-of-turn (final=True)
via the existing _strip_mistral_reasoning helper; streaming is untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep times in wrapper-less Gemma tool arguments
The wrapper-less Gemma value scanner used _GEMMA_KEY_RE = [\w.\-]+ for keys,
which also matches a digit-leading token, so a comma followed by a time or
ratio inside a value (call:web_search{query:meet at 10:00, 11:00 tomorrow})
was misread as a new 11: key, truncating the query and injecting a bogus
argument. Require keys to start with a letter or underscore, matching the
identifier-start rule the wrapped path already uses (_GEMMA_NEXT_KEY_RE).
Add a regression test.
* Studio: treat markers/close-tags inside tool-call arguments as data
Four parser correctness fixes where a valid argument string was mistaken for
structure:
- DeepSeek: find the envelope-end token outside JSON strings, so a query/code
argument containing the literal token no longer truncates the body and drops
the whole call.
- GLM: locate the real </arg_value> as the one whose next token is <arg_key> /
</tool_call> / end, so a value containing a literal </arg_value> (or
</tool_call>) is kept instead of executing the tool with corrupted arguments.
- Attribute-form <function name="..."> envelopes now count in the embedded-marker
guard, so a DeepSeek/Kimi marker inside a parameter value does not hijack the
outer call and run the wrong tool.
- Wrapper-less Gemma call:NAME{...} is gated on the enabled tool names (parse and
display strip), mirroring the Llama bare-JSON gate, so a disabled/example name in
prose is not stolen as a call and the real answer is preserved.
Add regression tests for each.
* Gate route Gemma wrapperless strip by enabled tools; make Kimi section-end search string-aware
Route-level display stripping now threads the enabled tool-name set into the
Gemma wrapperless-call strip, so prose that mentions a disabled tool
(call:foo{...}) is preserved while active tool calls are still stripped. This
mirrors the parser-level gate already used in tool_call_parser.
The Kimi section-end lookup now searches outside JSON string literals, so a
section-end marker appearing inside an argument string no longer triggers a
false truncation that drops a valid tool call.
* Run DeepSeek/Kimi pre-pass when a closed tool-call example precedes a real block
The marker pre-pass was skipped whenever any <tool_call>/<function> opener
appeared before the first DeepSeek/Kimi marker, even when that opener was a
CLOSED syntax example in prose that ends before the real block. In that case
parse_tool_calls_from_text skipped the DeepSeek/Kimi parsers and the genuine
tool call was dropped while a phantom tool named in the example ran instead.
Only treat a marker as embedded in a leading envelope when removing the closed
outer <tool_call>/<function> envelopes also removes every marker (the marker
actually sat inside one). A marker left standing is a real call, so the pre-pass
runs. The legitimate case of a marker inside a closed outer envelope's arguments
is preserved.
* Honor reasoning_effort none in safetensors prefill; strip Magistral reasoning while streaming
Two safetensors/MLX reasoning fixes surfaced in review:
_sf_reasoning_prefill_mode only checked enable_thinking, so an
enable_thinking_effort (GLM-5.2) request that disables thinking via
reasoning_effort=none (without enable_thinking=False) still began in
prefilled-<think> mode. A plain answer with no </think> was then swallowed
whole into reasoning_content and the visible response came back empty. Thread
reasoning_effort into the predicate and treat none as disabled, mirroring
_request_reasoning_kwargs.
strip_tool_markup_streaming stripped tool markup but not the leading Magistral
[THINK]...[/THINK] bracket block, so the raw chain-of-thought leaked into the
streamed safetensors content instead of the reasoning drawer (GGUF routes it
natively). Apply _strip_mistral_reasoning first, matching the final strip; an
unclosed [THINK] is held from the marker on so nothing flickers.
* Heal truncated outer tool envelopes and keep quoted Gemma args intact
Two follow-ups from review of the marker pre-pass and Gemma parsing:
The leading-envelope guard only removed CLOSED outer <tool_call>/<function>
envelopes before deciding whether a DeepSeek/Kimi marker was embedded, so a
truncated outer call missing its close tag (whose argument embeds a marker) was
treated as a standalone marker and the embedded sample ran instead of the
intended outer call being Auto-Healed. Decide on the last outer opener before the
marker and whether it closed before the marker instead, so a closed syntax
example still runs the pre-pass while a real closed-or-truncated outer call keeps
it.
The wrapper-less Gemma argument scan tracked bracket depth but not quotes, so a
quoted value containing a comma followed by a key-like token (a search query such
as "weather, location: Boston") was split mid-string, truncating the value and
fabricating an extra argument. Track quote state (with escapes) so the top-level
comma boundary is only taken outside quoted spans.
* Span outer envelopes to their real close when locating embedded markers
Locating the DeepSeek/Kimi marker relative to a leading outer envelope used the
FIRST close tag after the opener, so a literal </function> or </tool_call> inside
an argument value (for example python code that contains the text) was mistaken
for the envelope boundary. The marker after it was then treated as a standalone
call and the embedded sample ran instead of the intended outer call.
Match the closed outer envelopes with the shared patterns that already extend to
the real final close (a literal close inside a value is data), and treat a marker
that survives their removal as embedded only when a still-open (truncated) outer
opener precedes it, so Auto-Heal still repairs a truncated outer call. A closed
syntax example before a genuine block still runs the pre-pass.
* Span the tool_call outer envelope to its real close in the marker guard
The leading-envelope check reused the lazy <tool_call>.*?</tool_call> strip
pattern, so a Qwen/Hermes JSON argument containing a literal </tool_call> ended
the span early. A DeepSeek/Kimi sample later in that same string then survived
the closed-envelope removal, and the pre-pass executed the embedded call instead
of the outer <tool_call>. The <function> arm already spanned to its real close;
give <tool_call> the same real-close pattern (with the negative lookahead that
keeps back-to-back calls separate) so a literal close inside a value is data.
* Preserve no-tool Gemma prose and keep later R1 calls when healing a close
Two review follow-ups:
_gemma_strip_gate returned None when no tools were enabled, and None means
strip every markerless call:NAME{...} block, so a no-tool answer that documents
the syntax (or the Anthropic display path, which passes an empty tool list as
None) had that prose deleted. It is a display/history gate, so return the
enabled-name set instead -- an empty set when no tool is enabled, which strips
nothing because every call:NAME{...} is then prose.
The DeepSeek R1 heal path located the close fence with an unbounded forward
search, so when a first call had balanced JSON but omitted its fence the search
landed on a LATER call's terminator and pos advanced past that valid call,
dropping it. Match the close immediately after the JSON (whitespace-skipped) like
the strict path, and advance by just the JSON when it is absent, so a multi-call
turn keeps its later well-formed calls (heal is now a superset of strict).
* Resume wrapper-less Gemma scan past a consumed call's balanced body
The markerless call:NAME{...} scan used finditer, which resumes right after the
opening call: token, so a nested call:OTHER{...} mentioned inside the first
call's own quoted string argument (for example a web_search query that quotes the
Gemma tool syntax) was re-matched and returned as a spurious second tool call,
executing an unintended tool. Walk with a manual cursor that resumes after the
outer call's balanced body (brace matching already skips quoted braces), so a
call's arguments are never rescanned. Genuinely separate back-to-back calls and
disabled/example prose are unaffected.
* Mistral outer call wins over XML literals; align healer signals with its parser
Two follow-ups on the shared-parser ordering after the healing-passthrough
merge:
- A well-formed [TOOL_CALLS] call whose JSON arguments quote tool XML parsed
the literal instead of the outer call (executing the wrong tool). When the
first XML signal sits inside a leading balanced Mistral body it is argument
data, so the Mistral parser now runs first; an XML signal before the trigger
keeps the normal order, so a [TOOL_CALLS] literal inside an XML call's
arguments still stays data.
- passthrough_healing buffered streams on the parser module's broadened signal
list (now including <|python_tag|> and [TOOL_CALLS]) but promotes with
core.tool_healing, which does not parse those forms: a streamed Mistral or
Llama text call was held until finalization and flushed as prose. The healer
keeps its own signal list limited to the formats it can promote, restoring
immediate streaming for the rest.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Gemma wrapper-less marker literals and quotes, GLM embedded close pair
- The Gemma fallback deferral now keys on an actual wrapped opener
(_GEMMA_TC_RE), not the wrapper literal anywhere in content: a wrapper-less
call whose argument merely mentions <|tool_call> has nothing tool_healing
can parse, and deferring it lost the call entirely (not executed and
stripped from display).
- New _gemma_body_brace_end boundary scanner honors single- and double-quoted
strings like _gemma_parse_stripped_body, shared by parse and strip, so a
quoted brace in a code argument (code:print('}')) no longer truncates the
executed arguments or the strip span.
- _glm_value_close now requires a structural </arg_value> to sit at balanced
quote state: the full pair </arg_value></tool_call> embedded inside a string
literal is data, not an early close. When no candidate balances, the first
token-valid close wins as before.
* Address review: leading envelopes win over rehearsed literals
- New _first_foreign_tool_signal shared by the leading-envelope guards adds
<|python_tag|> to the protected signal set: the spelled-out literal inside a
Mistral call's arguments (a query about Llama built-in tool syntax) executed
the inner literal instead of the outer call.
- New _xml_signal_inside_leading_bare_json guard, sibling of the Mistral one:
a leading bare-JSON call whose string argument quotes tool XML (a code value
citing <function=...>) had the literal promoted by the shared XML pass
before the bare-JSON parser ran.
- Magistral [THINK]...[/THINK] is dropped once at parse entry instead of only
inside the Mistral parser, so a call rehearsed in the think block in a
foreign format can no longer be promoted while the real call after the
block is lost. Parse now agrees with the display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: a disabled leading bare-JSON object keeps its literals as data
When the leading bare-JSON object is ordinary content (name not an enabled
tool), the guard proved the first tool signal sits inside it, so falling
through to the XML/python_tag passes promoted quoted string data as a real
call. Drop the object and parse only the tail: a real call after the object
still parses, nothing inside it can be promoted.
* Address review: apostrophes in raw Gemma values, GLM strict key contract, per-model template token
- Quote openers in the wrapper-less Gemma boundary and body scanners now
require value-start context (after : { [ ( , =): an apostrophe inside an
unquoted value (query:what's the weather) opened quote mode, swallowed the
real closing brace, and lost the whole call on common contraction queries.
Quoted values keep hiding delimiters as before.
- A GLM <arg_key> with no <arg_value> tag now rejects the call in strict
mode, matching the unclosed-value contract, instead of executing the tool
with the argument silently dropped; Auto-Heal keeps the lenient skip.
- The native-template fallback reads the hf_token stored on the model record
instead of the instance-wide last-load token, so a later token-less load
cannot break template fetches for a previously loaded gated model (both
the transformers and MLX backends).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Mistral literals inside leading JSON, whitespace-tolerant wrapped Gemma opener
- The leading bare-JSON guard now treats the [TOOL_CALLS] trigger as a
foreign signal: the Mistral parser runs before the bare-JSON one, so a
literal quoted inside the leading object's strings was promoted over the
outer call (or over ordinary JSON content).
- tool_healing's wrapped Gemma opener tolerates whitespace around call and
the colon: sampling drift emits call: name{ and call : name{, and
rejecting those lost the call entirely because no fallback re-parses the
wrapped form. Strict mode still requires the closing tag.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: DeepSeek/Kimi markers inside leading JSON and Mistral envelopes stay data
The DeepSeek/Kimi pre-pass runs before the outer-call parsers, and
_marker_inside_leading_envelope only protected XML envelopes: a marker
quoted inside a leading bare-JSON or Mistral call's argument strings was
promoted as a separate no-arg call and the real outer call dropped. The
guard now recognizes those two leading envelopes as well; standalone
DeepSeek/Kimi calls keep parsing.
* Address review: accept dotted Gemma argument keys in the key-quoting scanner
The scanner quoted keys of [alnum_-] only, so a dotted key (user.name:...)
was left unquoted, json.loads failed, and the whole wrapped call was lost
(parse empty, strip wipes the markup). Dots now match the parser's own
key/name charset.
* Address review: a real DeepSeek/Kimi call after a disabled leading JSON object still parses
DeepSeek/Kimi markers are foreign signals for the leading bare-JSON guard
too: a marker literal inside a disabled leading object made the envelope
guard skip the pre-pass for the whole message, so a real DeepSeek/Kimi call
after the object was dropped. Routing the case through the guard's
drop-and-parse-the-tail recursion reaches the real call while the literal
inside the object stays data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: leading Mistral call owns the turn, dotted keys after bare values
- A LEADING parseable [TOOL_CALLS] call now runs the Mistral parser first
unconditionally: literal XML in trailing prose after the call was promoted
by the earlier shared XML pass, executing the quoted example instead of
the real leading call. XML leading keeps the normal order.
- _GEMMA_NEXT_KEY_RE accepts dots so a dotted key after a bare value
(query:foo,user.name:bob) ends the value at the comma instead of being
swallowed into it, matching the round-earlier key-quoting charset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: a leading wrapper-less Gemma call owns the turn
A quoted foreign literal inside a leading wrapper-less Gemma call's
argument (a query citing another tool syntax) was promoted by tool_healing
before the Gemma fallback ran, executing the quoted example and dropping
the outer call. New leading guard, sibling of the Mistral and bare-JSON
ones, gated on an enabled name since the form is markerless. Foreign markup
leading keeps the normal order.
* Fix merge resolution: restore both leading-guard test classes intact
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: markup quoted inside a nameless leading JSON answer stays data
The leading bare-JSON guard required a top-level name, so a structured JSON
answer quoting tool markup in its strings (a response_format turn
documenting a tool's syntax) had the literal promoted by the later passes.
A nameless leading object that parses as real JSON now routes through the
same decline-then-parse-the-tail path; non-JSON braced prose keeps the old
behaviour, and a real call after the answer still parses.
* Address review: JSON answers stay data, nested Gemma quotes, earliest envelope, no failure caching
- A whole-content JSON value is a structured answer: the markerless Gemma
scan and its strip no longer promote or strip a quoted example of an
enabled tool's syntax inside it.
- Nested stripped-stream Gemma values now unquote quoted string leaves
recursively, so {loc:{city:"New York"}} hands the tool New York, matching
the top-level coercion.
- The DeepSeek/Kimi pre-pass dispatches by earliest envelope opener, so a
leading real call wins over a trailing example of the sibling format in
either direction.
- A failed native-template fetch is no longer cached as no-template: the
next call retries after the model record's token is fixed or a transient
Hub error clears; only definitive loads are cached.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: closed calls precede the marker pre-pass, truncated Gemma scan stops, quoted nested delimiters
- A closed non-DeepSeek/Kimi call preceding the first DS/Kimi marker owns
the turn: a trailing syntax example, or one quoted inside a wrapped Gemma
argument, was promoted by the pre-pass and dropped the real leading call.
Wrapped Gemma joins the outer-envelope pattern sets.
- An unbalanced wrapper-less Gemma call now stops the scan (mirroring the
strip contract) instead of resuming inside its own argument text, where a
quoted enabled call would be promoted.
- Raw-quoted strings in nested stripped-stream Gemma values hide delimiters,
so {city:"New, York"} is one value instead of a split pair, returned
unquoted like the top-level coercion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: string-marker literals in wrapper-less args, mid-value quoted phrases
- The wrapper-less deferral guard no longer keys on the <|"|> literal: a
real call whose argument merely mentions the string marker was deferred to
tool_healing, which has no wrapped opener to parse, losing the call. The
wrapped-opener check alone owns the deferral.
- Double quotes now also open at the start of a word, so a quoted phrase
mid-value (query:find "weather, location: Boston", limit:3) hides its
delimiters instead of splitting the value into garbage keys; apostrophes
keep the value-start-only rule so contractions stay prose.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: strict GLM refuses in-quote close fallback, Gemma guard covers preambles
- _glm_value_close gains a strict flag: a truncated value whose only close
candidates sit inside a string literal rejects the call in strict mode
(Auto-Heal keeps the lenient partial), restoring the strict contract the
quote-aware fallback had weakened.
- The leading wrapper-less Gemma guard no longer requires the call to open
the response: a visible preamble before call:NAME{...} is the normal
shape, and the quoted foreign literal inside the argument was promoted
again in that shape. An enabled balanced call beginning before the first
foreign signal owns it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: contextual GLM quote openers, disabled Gemma examples stay prose, JSON array answers
- The GLM value-close quote tracker uses the same contextual openers as the
Gemma scanners (single quote after punctuation context, double quote also
at word start), so strict mode accepts a normal apostrophe value again
while still rejecting a truncated value whose only close candidates sit
inside a string literal.
- A disabled wrapper-less Gemma call is prose by design, so a tool literal
quoted inside it no longer promotes: the span is dropped for parsing and
the tail parsed, mirroring the nameless-JSON guard.
- Leading JSON ARRAY answers join the leading-JSON envelope guard, so a
marker quoted inside a structured array response stays data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align closed-envelope regression test with the document-order contract
The test asserted the pre-round-13 behavior (trailing DeepSeek/Kimi block
wins over a leading closed envelope) while the shipped rule is document
order: the leading closed call owns the turn. Rename the test and assert
the leading call so the suite matches the contract exercised by
test_leading_xml_call_wins_over_trailing_kimi_example.
* Parse a leading Llama-3.2 bare-JSON call before the markerless Gemma scan
The bare-JSON form only ever matches a leading call object, and document
order says that call owns the turn. Running the Gemma wrapper-less scan
first let an enabled call:NAME{...} snippet quoted inside the leading
call's string arguments steal the turn when the JSON was not the whole
content (trailing prose or a second ;-separated call), executing the
quoted tool instead of the real one. Reordering cannot take a leading
Gemma call's turn since that content never starts with an object brace.
* Leading-call ownership: Mistral trigger in Gemma guards, closed bare JSON before markers, depth-aware nested Gemma values
Three parser gaps against the document-order contract:
The wrapperless Gemma leading guards did not count [TOOL_CALLS] as a
foreign signal, so a leading Gemma call quoting a Mistral snippet in its
argument lost the turn to the quoted literal. Both the enabled-call and
disabled-example guards now include the trigger, matching the bare-JSON
guard's local inclusion.
_marker_inside_leading_envelope required the DeepSeek/Kimi marker to sit
inside the first closed bare-JSON or Mistral call. A marker after that
closed call (a trailing example or data in a later ;-chained call's
strings) now also defers to the leading call, the same inside-or-after
rule the closed XML envelope patterns already applied.
The nested Gemma primitive value scan split on every comma, corrupting
arguments like opts:{code:print(1,2),lang:py}. It now applies the same
paren/brace depth, contextual quote openers, and comma-only-before-a-key
mapping rule as the top-level scan.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gemma leading guard: a closed enabled call preceding the signal owns the turn
The wrapperless Gemma guard only claimed the turn when the first foreign
signal sat inside the first enabled balanced call. When that call closed
before the signal (a second call quoting a Mistral or Kimi literal, or a
trailing prose example), the guard forfeited the turn and the foreign
parser promoted the quoted literal, dropping the real Gemma calls. Apply
the same inside-or-after ownership rule as the closed bare-JSON and
Mistral envelopes, gated on an enabled name so the name-agnostic legacy
path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Marker guard: only an executable leading bare-JSON call owns the turn
The bare-JSON branch of the leading-envelope marker guard claimed the
turn for any NAMED leading object. A disabled-name object is prose by
design (the bare-JSON parser will not execute it), so deferring the
DeepSeek/Kimi pre-pass to it lost the real later call entirely. Gate the
ownership claim on the enabled set (or the name-agnostic None path). A
marker inside the disabled object's own strings stays data, matching the
tail-exclusion contract; a marker after it now falls through so the
pre-pass parses the real call. The Mistral branch stays ungated since
[TOOL_CALLS] parsing is never name-gated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gemma scan skips leading JSON answers; GLM heal bounds values at structural tags
Two fixes to the document-order data contracts:
The markerless Gemma scan only exempted whole-content JSON, so a leading
JSON answer followed by prose had an enabled call:NAME{...} snippet
inside its strings promoted to a real executed call and stripped from
the displayed answer. Both the parse and strip scans now start after a
balanced json-valid leading value span, keeping parse and strip
mirrored. Real calls after the answer still parse; mid-prose JSON gets
no exemption.
The GLM heal fallback for a missing closing arg_value tag took the
entire remainder as the value, executing markup-contaminated arguments
like city="NYC</tool_call>" and swallowing trailing prose. The healed
value now stops at the next arg_key or tool_call close and the pair walk
resumes there. EOF-truncated values keep the partial heal, strict mode
still rejects, and closed values holding a literal close tag in quotes
are untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Compress docstrings in the multi-format tool parser to their contract essence
* Condense parser guard comments and test narration to contract essentials
* verify_import_hoist: exempt __future__ imports and same-diff relocations
Two false positives fired on this PR's refactor. A from __future__ import
is a compiler directive whose name never appears as a runtime load, so
HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for
PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate
move of the strip-pattern constants into core.inference.tool_call_parser
as a silent re-point even though the old module-level target was removed
and the new one added in the same diff. Both get narrow exemptions; a
re-point to a pre-existing target is still caught, and the self-test
negative controls all pass unchanged.
* Leading bare-JSON calls own the turn; function calls end at the first balanced close
The XML-signal guard for a leading bare-JSON call required the signal
strictly inside the object, so a trailing XML example stole the turn
from the leading call; it now applies the same inside-or-after rule as
the Mistral guard. Function-XML calls also ended at the LAST close tag,
which let prose after a closed call that mentions a literal close tag
get swallowed into the final parameter value; calls now end at the
first close tag that is not inside an open parameter, and the strip
mirrors the same rule so parse and strip agree.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Attribute-form calls end at the first balanced close; bare-JSON strip requires the call shape
The attribute form parser still kept the last close tag in the call
window, folding prose after a closed call into the final parameter
value. It now takes the first close not inside an open parameter, the
same rule the equals form and the strip already use.
The leading bare-JSON strip deleted any closed object whose top-level
name matched an enabled tool, including plain JSON answers the parser
correctly rejects as non-calls. The strip (and the drain gate that
delegates to it) now requires the parser's exact call shape, so answers
like {"name":"web_search","result":...} stream and display intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* False-alarm markers keep the answer; the bare-JSON strip consumes the whole chain
The trailing strip arms dropped everything from a bare marker to EOF,
so a normal answer that mentions [TOOL_CALLS] or another marker
literally was truncated (or fully swallowed when it started with the
literal) after the no-call drain fallback. Those arms now require a
call-shaped lookahead or marker-at-EOF before dropping; truncated real
calls still strip.
Chained bare-JSON turns executed both calls but stripped only the first
object, so the second call's raw JSON replayed into the next assistant
history message alongside the structured tool_calls. The strip now
consumes the entire chained run of call-shaped enabled objects while
non-call answers, disabled names, and trailing prose stay intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* DeepSeek and Kimi trailing strip arms require a call-shaped lookahead
Same false-alarm rule as the bare-word markers: a prose answer that
mentions a DeepSeek or Kimi marker literally keeps its tail, while
truncated real envelopes and bare end-of-text fragments still drop.
* Attribute-form containment, parameter-close-decides rule, preamble-tolerant Mistral guard, strict strip shape
Four document-order and containment fixes. A leading attribute-form
call now parses before the shared XML pass, so markup quoted in its
parameter stays data. The open-parameter scan lets the parameter's own
close tag decide, so any number of literal function closes inside one
value stay data, restoring the pre-close-scan behavior for multi-close
arguments. The leading-Mistral guard tolerates a visible preamble, with
the leading-bare-JSON guard running first so a trigger quoted inside a
leading JSON object stays data. The bare-JSON strip requires the
parser's top-level name in every mode, so nested-name JSON answers
survive name-agnostic stripping.
* Keep buffering long wrapper-less Gemma tool names instead of leaking the prefix
The streaming buffer stopped holding a call:NAME prefix at a fixed
32-char cap, so a Gemma wrapper-less call to a tool whose name exceeds
that (OpenAI allows 64 chars, MCP names run longer) streamed its raw
call:longname text as visible content before the end-of-turn parser
executed it. Hold the variable-length prefix while it still matches the
call: shape, bounded like the bare-JSON path and self-terminating into
prose, draining once the opening brace arrives.
* Keep prose that only mentions DeepSeek/Kimi markers in the route display strip
The route-level _TOOL_XML_RE DeepSeek/Kimi arms consumed from an opener up to
the end of text whenever the marker appeared, so an answer that merely refers
to a marker (for example "See <|tool_call_begin|> in the docs") had the rest
of the reply truncated. The parser-level _TOOL_ALL_PATS already gates these
arms with a call-shaped lookahead. Mirror it here so a marker is only stripped
when a real call follows it or it is a bare fragment at end of text.
* Tighten tool-calling parser and backend comments
* Pass trust_remote_code when reloading native tokenizers
The native-template fallback re-fetches a model's native chat template from
its repo when an Unsloth override template drops the tools schema. The
secondary AutoTokenizer.from_pretrained threaded hf_token but not
trust_remote_code, so for a model loaded with trust_remote_code=True whose
tokenizer repo carries custom code the reload raised, was swallowed, and the
request silently kept the tool-dropping prompt for a model that supports tools.
Store the loaded trust_remote_code on each backend's per-model info dict and
source it in render_native_template, so the reload re-uses exactly the consent
granted at load. For a LoRA adapter the reload targets the base model, whose
remote code was gated and loaded under the same stored flag, so re-passing it
executes no unconsented code. Falsy stored flag preserves the prior behaviour.
Adds a regression test that fails without the flag (custom-code reload raises,
returns None) and passes with it (tools-advertising native prompt returned).
* Treat <|python_tag|> as an outer marker envelope
A Llama-3 <|python_tag|> tool call (built-in NAME.call(...) or custom
{json} form) whose argument quotes a complete DeepSeek/Kimi example was
hijacked by the DeepSeek/Kimi marker pre-pass: the embedded example (for
example delete_all) executed instead of the real outer call. python_tag
is Llama-3's tool-call envelope, so a marker quoted inside its arguments
is data, the same as for <tool_call>, <function=...>, bare JSON, Mistral
and wrapper-less Gemma, which the guard already covers.
Add <|python_tag|> to _OUTER_ENVELOPE_OPEN_RE with a call-shaped
lookahead (mirroring the _TOOL_ALL_PATS python_tag arm) so the marker
pre-pass is suppressed when a python_tag call opens before the first
marker, while a bare prose <|python_tag|> mention is left untouched.
* Tighten tool-call parser comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
* Quote-aware Gemma strip, symmetric unstarted cleanup, ReDoS anchor
Address review findings on the tool-strip and streaming paths:
- strip_tool_call_markup stripped Gemma-native spans with a plain regex that
stops at the first <tool_call|>, so a literal close marker inside a
<|"|>-quoted argument truncated the span and leaked its suffix into visible
text. A brace/quote-aware _strip_gemma_native_spans now removes complete
spans (keeping an incomplete one unless final), matching the parser's own
balance logic.
- The Gemma close pattern this PR added (<\|tool_call>.*?<tool_call\|>) had no
\Z fallback, so a run of unclosed markers backtracked from every open
position (quadratic, and the streaming stripper re-scans per token). It is
now anchored to (?:<tool_call|>|\Z) like routes/inference.py's _TOOL_XML_RE,
linear with identical output on well-formed input.
- _SameTaskStreamingResponse added unstarted_cleanup for the OpenAI passthrough,
but the local GGUF/safetensors streams that enter _TrackedCancel before
returning only unregister in the generator finally, which never runs if the
client disconnects before the body iterator starts, leaking cancel-registry
entries. Each such stream now passes unstarted_cleanup to exit its tracker.
- __call__ reads _unstarted_cleanup via getattr so a response built through
__new__ (the cancel-timing test) without __init__ does not raise
AttributeError; the test also sets the attribute explicitly.
- Document that the verbatim /v1/chat/completions passthrough delegates
<think>/<|tool_call> splitting to llama-server (--jinja, --reasoning-format
auto) and is intentionally not re-parsed locally, noting the llama.cpp
dependency.
Adds a regression test for the close-marker-inside-quoted-argument strip.
* Tighten comments on the tool-strip and streaming paths
Compress the verbose comment blocks added with the Gemma tool-call / streaming
work to crisp one or two liners, drop restatements of obvious code, and shorten
docstrings, keeping the load-bearing rationale (ReDoS anchor, quote-aware strip,
unstarted-cleanup, llama.cpp passthrough dependency). Code is unchanged
(verified comment-only via AST/ast signature, docstrings stripped).
* Harden Gemma parse/strip: span-aware XML fallback and quote-aware streaming
- Security: the XML fallback in parse_tool_calls_from_text scanned the whole
content for <function=...> markers and only skipped those inside an open XML
parameter, not those inside a collected JSON/Gemma candidate span. A balanced
but unparsable Gemma call whose argument data contained XML tool markup
(<|tool_call>call:outer{code:<function=terminal>...}<tool_call|>) therefore
fell through to the fallback and returned an executable terminal call. The
fallback now also excludes <function=> markers inside any candidate span,
including ones that failed to parse.
- strip_tool_call_markup no longer skips the generic Gemma regex after running
the quote-aware _strip_gemma_native_spans, so a closed Gemma span the helper
cannot match (malformed, e.g. <|tool_call>{"name":"x"}<tool_call|>) is still
stripped instead of leaking its opener and payload into visible text.
- _strip_gemma_native_spans stops at the first unbalanced start instead of
re-scanning every later start to EOF, keeping it linear on a run of unclosed
markers rather than quadratic.
- The GGUF and safetensors streaming strippers run _strip_gemma_native_spans
before the regex patterns, so a well-formed streamed call whose quoted
argument contains a literal close marker no longer leaks its suffix into
incremental display.
Adds regression tests for the nested-XML escape and the malformed-span strip.
* Avoid remainder copy in _strip_gemma_native_spans
Match the Gemma close marker with re pos directly on the buffer instead
of slicing tail = text[brace_end + 1:] on every span. The streaming
strippers re-scan a growing cumulative buffer per token, so the per-span
remainder copy was quadratic. Behavior is unchanged.
* Exclude unclosed Gemma/JSON starts from the XML tool-call fallback
The nested-XML guard only skipped <function=> markers inside recorded
candidate spans, but a span is recorded only when the braces balance. An
unbalanced call such as <|tool_call>call:outer{code:<function=terminal>...
recorded no span, so the fallback still promoted the inner <function=> to
an executable terminal call. Treat unclosed JSON/Gemma starts as exclusion
spans through EOF before scanning. Standalone <function=> calls with no
preceding unclosed start still parse. Regression tests added.
* Skip doomed tool-strip passes to avoid quadratic rescans
The lazy closed-pair strip patterns (<tool_call>.*?</tool_call>,
<function=...>.*?</function>) rescan to EOF from every opener when their
close token is absent, which is O(n^2) and re-runs per streamed token. Add
strip_tool_patterns, which skips a pass whose close token is not present in
the text; output is identical to the per-pattern loop (verified by fuzz),
and a degenerate run drops from ~minutes to milliseconds. Used by
strip_tool_call_markup and the GGUF/safetensors streaming strippers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use full tool-call envelopes to close nested-XML escape variants
Key the parser and stripper off the full <|tool_call>...<tool_call|> /
<tool_call>...</tool_call> envelope (start to close marker, searched after
the braces; EOF if unclosed) instead of just the braces:
- XML between the closing brace and the close marker
(call:outer{broken:{x}}<function=terminal>...<tool_call|>) is now inside
the envelope, so the fallback no longer promotes it to a tool call.
- A balanced inner call inside an unclosed outer
(call:outer{code:<|tool_call>call:terminal{...}<tool_call|>) is skipped
via the envelope nested check, not just the XML fallback.
- strip_tool_call_markup searches for the close marker after the braces, so
junk before <tool_call|> is stripped through the close and text after it is
preserved instead of truncated to EOF; a no-close run stops early (linear).
Regression tests added; standalone XML and well-formed calls unaffected.
* Fix non-final Gemma strip and missing-close recovery for PR #6611
Split the nested-skip from the XML fallback exclusion: nesting is decided by
each marker's brace region, so a balanced call after one with a missing close
marker is recovered instead of being swallowed to EOF. Only the XML fallback
keeps the search-to-close envelope, so trailing nested markup still cannot
escape as an executable call.
Use a closed-only Gemma pattern in the non-final strip list so an incomplete
block is preserved (matching the JSON and function paths); the final list keeps
the close-or-EOF Gemma pattern in its original position, so streaming display
output is byte-for-byte unchanged.
Add regression tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Block gap-nested tool markers and fix XML strip order for PR #6611
Decide candidate nesting by a per-marker coverage region paired with a
per-format stack (a close after the braces pops the nearest still-open marker
of that format). A closed outer call now covers up to its own close marker, so a
JSON or Gemma tool marker smuggled between the outer braces and that close is
treated as data instead of being executed. An outer that balances but has no
close of its own covers only its brace region, so a later sibling after an
omitted close marker is still recovered (adjacent calls use an exclusive end
bound so the next call is not misread as nested).
Strip every closed pair (JSON, Gemma, function) before any to-EOF sweep, so a
closed function call whose parameter text contains a bare Gemma opener is
removed as a unit and the to-EOF sweep can no longer drop the visible text after
the close.
Add regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip closed tool blocks before the Gemma final sweep for PR #6611
The final display strip ran the quote-aware Gemma helper before the closed
JSON/function patterns. A closed <tool_call>...</tool_call> or
<function=...>...</function> block whose argument data held a call-form Gemma
opener (e.g. a "<|tool_call>call:t{" string) was read as an incomplete Gemma
span and truncated to EOF, dropping the block's close and any visible text after
it.
Strip closed JSON/function blocks first, so such a block is removed as a unit
before the helper runs. Centralize the final strip order in a shared
strip_tool_markup_final so strip_tool_call_markup and both streaming display
wrappers (safetensors, llama_cpp) stay in sync, and apply the same closed-block
pre-pass to the non-final path.
Add regression tests for the JSON and function variants.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Recover XML/JSON siblings after a close-less tool marker for PR #6611
Two fixes so the XML fallback and marker coverage recover a later valid call
after an earlier marker omits its close, matching the candidate loop:
Reuse the candidate marker-coverage in the XML fallback instead of a separate
search-to-close-or-EOF envelope. A balanced but close-less marker now covers
only its brace region there too, so a following <function=...> sibling is
recovered rather than filtered as nested data; an unbalanced marker still covers
to EOF and a closed one still covers through its close, so nested XML stays
blocked.
Ignore a close token that falls inside another call's balanced braces when
pairing closes in _marker_coverage. Such a token is that call's quoted argument
data, so it no longer pops an earlier close-less marker and extends its coverage
over a later valid sibling.
Add regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the closed-block strip pre-pass Gemma-span-aware
The final display strip ran the closed JSON/function regex pre-pass before
removing Gemma-native spans, so a literal <function=...> quoted inside a Gemma
argument plus any later </function> (a real call's close or even prose) was
deleted across the Gemma boundary. That mangled the Gemma close marker, the
quote-aware helper then saw an unclosed opener, and the whole visible tail
after the call was truncated.
The pre-pass now skips matches that start inside a complete Gemma span (that
text is the span's argument data) and resumes scanning at the end of the
covering span, so a real function-XML call after the Gemma call is still
stripped. The original ordering rationale is preserved: a Gemma opener inside
a JSON or function argument still cannot truncate that block, covered by
regression tests for both directions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the Gemma streaming and strip pipeline to essentials
* Tighten comments in the Gemma strip and streaming disconnect paths
* Fold marker-collection comment to two lines
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: coerce tool_call arguments to dict before chat templating
Strict tool chat templates (e.g. mlx-community Qwen3.5 checkpoints) iterate
arguments.items() and raise "TypeError: Can only get item pairs from a mapping"
when a prior assistant tool call is re-rendered on the next turn. The agentic
loop stores arguments in the OpenAI JSON-string form (as_assistant_tool_call),
which is correct on the wire and for llama-server, but the transformers / MLX
paths apply_chat_template directly and hit the strict Jinja templates.
Normalize each assistant tool_call's function.arguments from a JSON string to a
dict inside apply_chat_template_for_generation (shared by both the MLX and
safetensors paths). A dict renders on strict and lenient templates alike;
non-JSON / non-dict values are left untouched, and the OpenAI-format
as_assistant_tool_call (used by the GGUF path + API responses) is unchanged.
Verified against the real mlx-community/Qwen3.5-2B-8bit template: string args
raised the tester's error, the fix renders cleanly, and the lenient
unsloth/Qwen3.5-0.8B template still works.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make tool-arg coercion a string-first fallback (non-regressive)
Render the original OpenAI string-arg form first and only coerce arguments to a
dict when the template raises the mapping TypeError, instead of always coercing.
Any template that already renders is now byte-identical (a template that emits
arguments verbatim keeps the JSON string, not a Python dict repr).
Verified across Llama-3, Qwen2.5, Qwen3, Qwen3.5, Phi-3.5 (byte-identical) and
mlx-community/Qwen3.5-2B-8bit (strict -> fixed). Gemma-3 / Mistral tool-template
errors are unrelated (role alternation / tool-id length) and identical with or
without the change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make core.inference package init lazy so dependency-light helpers import standalone
Importing any core.inference submodule ran the package __init__, which
eagerly imported orchestrator and llama_cpp; both pull loggers ->
structlog (and httpx), so a dependency-light helper like
chat_template_helpers dragged in the full heavy stack and its unit test
failed to collect in a backend env without structlog. Defer those
imports to attribute access via PEP 562 __getattr__, mirroring the lazy
pattern already in core/__init__.py. The re-exports resolve unchanged on
first access.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Retry dict-coercion for strict templates that raise non-TypeError
apply_chat_template_for_generation only retried the OpenAI JSON-string arguments
coercion when the first render raised TypeError (the arguments.items() form). The
bundled gemma-4.jinja instead rejects string arguments with raise_exception, which
surfaces as a Jinja error, so a second tool turn with string function.arguments
propagated and failed rather than retrying with the parsed dict.
Broaden the outer catch to Exception, still gated on there being a string arg to
normalize (normalized is messages -> re-raise), so unrelated template errors and
templates that already render are unaffected.
* Tighten comments in tool-call argument coercion helper and tests
* Tighten tool-call argument coercion comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: deterministic backend tool-calling wiring test
Add a deterministic, download-free test that exercises the shared tool-calling
seam both inference backends use. InferenceBackend (transformers) and
MLXInferenceBackend both render the prompt through
apply_chat_template_for_generation(..., tools=...) and stream cumulative text
into run_safetensors_tool_loop. The existing test_safetensors_tool_loop.py
covers the parser and the loop state machine with fake generators but does not
cover the backend's own tool-injection seam, so a regression that drops the
tool schema before the tokenizer, or fails to feed a tool result back into
generation, would slip through.
The test drives that seam with fakes: a tokenizer that records the tools it is
handed, a canned tool-call generation, and a stub executor. It asserts the full
chain: tools reach the chat template, the loop parses the call, the tool is
dispatched once with the parsed arguments, the result is fed back, generation
re-enters, and the final answer streams after the tool result. It also guards
that the raw tool-call markup never leaks to the client as content.
The test imports no torch, unsloth, or mlx, so it runs in the portable Backend
CI alongside the tool-call parser tests and stays sub-second. Follow-up to the
parser test PRs #5620 and #5704.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: assert the tool result is fed back before the final turn
Strengthen the wiring test so single_turn records each turn's conversation and
the test asserts the tool result message is present in the conversation handed
to the final generation turn. Event ordering alone did not catch a loop that
stops appending the tool output before re-entering generation, because the fake
generation ignores the conversation; this closes that gap.
* studio: tighten comments in tool-calling wiring test
* studio: shorten comments in tool-calling wiring test
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: stop chat generation on the assistant-turn-end token
A small chat model (e.g. Qwen3.5-0.8B) looped on the safetensors path: it emitted
a valid response or tool call, then ran past its turn and re-emitted the call,
hallucinating <|im_start|>user turns. Root cause: the model's tokenizer.eos_token
is synced to the config document terminator (<|endoftext|>, 248044) while chat
turns actually end with <|im_end|> (248046), so generate_stream's single
eos_token_id never stopped at the turn boundary.
Stop on every assistant-turn-end marker the vocab defines (tokenizer.eos plus
<|im_end|>, <|eot_id|>, <end_of_turn>, ...). Verified on the real weights: the
single-eos control loops (400 tokens) while the fixed set yields a clean 38-token
tool call and a clean answer from the tool result. No-op when eos is already the
turn-ender (the id just dedups).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: repair chat generation_config.eos_token_id at load time
Qwen3.5 / Qwen3.6 small chat checkpoints declare the chat turn-end as
tokenizer.eos_token (<|im_end|>) but ship config.eos_token_id = <|endoftext|>
and no generation_config.json (upstream shipped generation_config only on the
large chat models). So every .generate() path that reads generation_config -- the
vision path and tool loops, not just generate_stream -- never stops at the turn
boundary and loops.
At load time, when the tokenizer's own eos is a chat turn-end marker but
generation_config.eos_token_id omits it, add it. This fixes the config once for
all generation paths and complements the generate_stream turn-end stop. No-op for
base models (eos is a plain document terminator) and already-correct configs.
Verified on unsloth/Qwen3.5-0.8B: 248044 -> [248044, 248046].
* Studio: derive chat turn-end eos from the template, resolve once at load
Address PR review of the turn-end stop handling:
- Do not call tokenizer.get_vocab() per generation request (serializes the whole
100k+ vocab). Resolve the turn-end tokens once at load and cache them on
model_info; generate_stream reads the cache.
- Derive turn-end markers from the chat_template the model actually uses, not raw
vocab membership, so a base/coder model that merely carries ChatML control
tokens in a shared vocab is not stopped early, and a loader that synced
tokenizer.eos to the document terminator is still covered.
- Skip harmony/gpt-oss templates: <|end|> there is an intra-message channel
delimiter, not the turn end (dropped <|return|> from the marker list too).
- Move the logic to a dependency-light module (core.inference.chat_eos) so the
unit test does not import the full unsloth/torch inference stack.
Verified on unsloth/Qwen3.5-0.8B (gen_config 248044 -> [248044, 248046], clean
38-token tool call with generation_config-only stopping), Phi-3.5 (adds <|end|>),
Llama-3 / Qwen3 (unchanged), and a harmony template (left untouched).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: refresh turn-end eos after the mapper installs its template
For a MODEL_TO_TEMPLATE_MAPPER model whose own tokenizer ships no
chat_template, the effective template is applied at generate time via
get_chat_template, but the turn-end eos ids were resolved once at load when
the template was still empty, so only the document eos was cached. Qwen2.5 /
Yi base checkpoints (eos <|endoftext|>, ChatML turns end with <|im_end|>)
then run past the assistant boundary in generate_stream and loop.
Re-resolve the turn-end eos from the now-templated tokenizer and refresh the
cached ids right after applying the mapper template, so generate_stream stops
at the ChatML turn end. Add a regression test.
* Studio: union turn-end eos refresh into load-time cache instead of overwriting
get_chat_template can return a different tokenizer whose vocab was remapped
(Gemma folds <end_of_turn> onto the eos id), while generate_stream re-reads the
original model_info tokenizer. Overwriting the cache with the refreshed set
dropped a valid load-time id (e.g. <end_of_turn>=107) and let generation run
past the real turn marker. Union the refresh into the existing cache so it can
only add ids, never drop a valid one. Add a regression test covering the
destructive-swap case the prior test missed.
* Studio: resolve refreshed turn-end ids on the generation tokenizer, add Gemma-4 marker
Two residual gaps in the turn-end eos refresh:
- For map_eos_token=True mapped templates (e.g. chatml on a Yi-6B base), get_chat_template
returns a tokenizer whose vocab folds the turn-end token onto the document eos id, while
generate_stream re-reads the original tokenizer. The refresh resolved ids on the returned
tokenizer, so it stored the doc eos and missed the real turn-end id, and generation ran
past the boundary. Read the turn-end marker strings from the mapped template but resolve
their ids on the original generation tokenizer (new resolve_chat_turn_end_eos_ids_using).
- Add Gemma-4's <turn|> turn terminator to the marker allowlist; those templates keep a
document eos so resolve otherwise missed the real turn marker.
Add regression tests for both.
* Fix turn-end detection for Starling, multi-variant and vision templates; keep tests collectable
The turn-end marker set missed OpenChat/Starling's barred <|end_of_turn|>
(distinct from Gemma's unbarred form), so Starling generations ran past
the assistant boundary. A dict/list chat_template (Hermes-3 style
default+tool_use variants) hit an early non-string return and skipped
detection; flatten and scan every variant. Vision models carry the
chat_template on the ProcessorMixin, not the unwrapped inner tokenizer,
so read markers from the template-carrying container while resolving ids
on the generation tokenizer.
The refresh test constructs the real backend, so it is guarded with a
module-level skip when unsloth/unsloth_zoo is absent (the lightweight
pytest matrix), and core.inference package init is made lazy so the
dependency-light chat_eos tests collect without the heavy stack.
* Studio: tighten chat turn-end eos comments
* Studio: condense chat turn-end eos comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: tool calling for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5615)
Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe.
* studio: tool-call healing parity between safetensors / MLX and GGUF
After the multi-format parser landed in #5615, the safetensors / MLX
agentic loop and the GGUF loop still differed on healing behaviour.
This commit closes the gaps in both directions so the two backends
react the same way to identical model output.
Changes:
1. core/inference/llama_cpp.py -- the GGUF BUFFERING state machine
now wakes on every emission marker the shared parser knows. Was
("<tool_call>", "<function="); is now the five-tuple imported
from core.inference.tool_call_parser (Qwen / Qwen3.5 / Llama-3
<|python_tag|> / Mistral [TOOL_CALLS] / Gemma 4 <|tool_call>).
Stream cleanup is delegated to the same shared strip_tool_markup
so leaked markup from any family is removed from assistant
content.
2. core/inference/llama_cpp.py -- per-tool canonical heal key. When
a tool arguments field is a bare string and JSON parsing fails,
the GGUF path now heals to {"code": raw_args} for python,
{"command": raw_args} for terminal, and {"query": raw_args} for
everything else. Was hard-coded to {"query": raw_args}, which
silently routed every python / terminal emission through
web_search. Mirrors safetensors_agentic._CANONICAL_HEAL_ARG.
3. core/inference/safetensors_agentic.py -- re-prompt on plan-
without-action. When the model emits a short forward-looking
intent ("I'll search for that", "Let me check", "First, I
will...") and no tool call, the loop nudges the model to act
instead of silently returning a plan-only answer. Up to
_MAX_REPROMPTS=3 (matches GGUF). The intent regex, character
cap, and instruction text are byte-identical to the GGUF path.
The buffer-end fall-through is unified so a buffered intent
emission that never exits the BUFFERING state still triggers
the re-prompt.
4. core/inference/safetensors_agentic.py -- extra iteration slots
for re-prompts. The loop now budgets max_tool_iterations +
_MAX_REPROMPTS + 1 total iterations and tracks the tool-call
count separately, so a stalling model can be nudged 3x without
eating the caller's tool-call budget. Mirrors the _extra slot
reservation in the GGUF path.
Tests (14 new safetensors-side units; 5 GGUF parity pins):
TestLoopRePrompt -- intent-trigger, plain-answer,
no-tools, cap-at-three, budget
preserved, buffer-end intent.
TestLoopCanonicalHealKey -- python / terminal / unknown.
TestGGUFSafetensorsHealingParity -- shared markers used, shared
strip used, canonical heal keys
identical, intent regex matches
same phrases, _MAX_REPROMPTS
equal on both backends.
All 110 targeted tests pass locally; the broader tool / inference /
model-config / sandbox / anthropic / mlx suites stay green.
Why this matters
Without this parity, Llama-3.2 / Mistral / Gemma 4 emissions on Mac
(MLX) and Linux-safetensors stop the agentic loop as soon as the
model says "Let me...", because the GGUF re-prompt logic never
existed on these backends. The two-marker GGUF BUFFERING tuple also
let non-Qwen tool emissions stream out as plain prose when
llama-server's structured channel did not pick them up. Both paths
now drain the same way, heal the same way, and re-prompt the same
way -- so a tool call that works on GGUF works identically on
safetensors / MLX.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix tool-call parser bugs from gemini review on #5620
Three high-priority gemini findings on the tool-call parsing additions:
1. unicode_escape on UTF-8 bytes corrupts non-ASCII literals
(e.g. ✨ becomes â\x9c¨). Replace with json.loads on a quoted
string -- preserves emoji / CJK / RTL while still handling
\n \t \uXXXX escapes.
2. Llama-3 sentinel stripping is order-dependent. A leading
`<|eot_id|><|begin_of_text|>` left `<|begin_of_text|>` behind
because the loop had already passed that sentinel. Loop until
no sentinel matches at the start.
3. Mistral v11+ `[TOOL_CALLS] name { json }` regex uses non-greedy
`\{.*?\}` which truncates at the first `}` of a nested JSON
argument, leaking the tail (e.g. `}}`) into user-visible
streamed text. Same problem for the v0.3 array pattern with
nested brackets. Strip those with balanced brace/bracket
scanning via a new `_strip_mistral_closed_calls` helper called
from `strip_tool_markup`.
Also fix the inference routes' parallel `_TOOL_XML_RE`:
- Same nested-JSON truncation in the Mistral patterns; route the
strip through the parser's balanced-scan helper via a thin
`_strip_tool_xml` wrapper that all existing callers now use.
- Llama-3 `<|python_tag|>[^\n<]*` stopped at any `<`, leaking the
tail of any tool call whose argument contained a literal `<`
(queries, code snippets). Relax to `[^\n]*` which keeps the
strip confined to the actual end-of-line.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/routes: make python_tag strip multi-line aware
Earlier revisions of _TOOL_XML_RE in studio.backend.routes.inference
oscillated between two bug shapes:
5615 r"<\|python_tag\|>[^\n<]*" -- stopped at any literal "<"
so code='if x < 10: pass'
leaked '< 10: pass)' to the
user.
5620.1 r"<\|python_tag\|>[^\n]*" -- single-line only; the second
line of
python.call(code="a\nb")
leaked.
The full parser (_parse_llama3_python_tag) already handles both via
balanced-brace scanning, so the parsing path was fine; the LEAK was
in the streaming strip path that runs on every cumulative emission
while content is still arriving.
Switch to r"<\|python_tag\|>(?:[^<]|<(?!\|))*" so the strip consumes:
* any character that is not a "<" (newlines, JSON, code, ...),
* a "<" only when it is NOT followed by "|" (i.e. NOT a Llama-3
sentinel start like <|eot_id|>, <|eom_id|>, <|begin_of_text|>).
This means:
* code='if x < 10' stays inside the strip (5615 fix preserved),
* multi-line code stays inside the strip (5620 round 2),
* the strip terminates at the next Llama-3 sentinel so trailing
assistant content survives.
Tests: TestRoutesPythonTagStrip (8 cases)
pytest test_safetensors_tool_loop.py test_safetensors_capability_advertise.py
-> 118 passed in 1.81s (was 110).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten verbose comments in tool-call parser sections
Comments were narrating what the code already says. Cut historical
"earlier revisions used X, then Y" narratives down to one-line WHY
notes where the footgun still matters (canonical heal-key parity,
balanced-brace vs non-greedy regex, ``(?:[^<]|<(?!\|))*`` over
``[^\n<]*``/``[^\n]*``). Drop section-header banners.
No behaviour change. Re-ran:
pytest studio/backend/tests/test_safetensors_tool_loop.py \
studio/backend/tests/test_safetensors_capability_advertise.py -q
-> 118 passed.
Regression replay (parser + _coerce_arguments on the 5 #5615 inputs)
-> 21/21.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: parser robustness fixes for PR #5620
Three surgical extensions to the multi-format tool-call parser, each
covering a real fine-tune / template emission shape that the current
parser silently drops. No path narrows; all changes widen what is
accepted.
1. `_parse_tool_call_json` now accepts both `arguments` and
`parameters` keys. A Hermes / Qwen `<tool_call>{json}</tool_call>`
wrapper around a Llama-3.2 fine-tune that emits the `parameters`
key was extracting the tool name and silently discarding the
args, producing a working-shaped call with an empty payload. The
bare-JSON and python_tag paths already accepted both keys; this
path now matches them.
2. `_TC_FUNC_START_RE`, `_TC_PARAM_START_RE`, and `_TC_PARAM_CLOSE_RE`
now also match the attribute form
`<function name="..."><param name="...">v</param></function>` used
by MiniCPM-5 and MiniMax-M2. Names land in either capture group,
and `</param>` is accepted as a short close.
3. `_parse_llama3_bare_json` sentinel-strip now consumes the role
label inserted between `<|start_header_id|>` and
`<|end_header_id|>` by Meta's official Llama-3.x chat template.
Without this, every assistant turn re-fed through the template
prefix `<|start_header_id|>assistant<|end_header_id|>\n\n{json}`
parsed to zero calls, so any history-with-tool-call round-trip
in production silently dropped.
Tests in `studio/backend/tests/test_safetensors_tool_loop.py`:
* `TestParserRobustness::test_tool_call_json_accepts_parameters_key`
* `TestParserRobustness::test_function_xml_attribute_form`
* `TestParserRobustness::test_function_xml_attribute_form_multi_param`
* `TestParserRobustness::test_function_xml_legacy_equals_form_still_works`
(regression guard for the existing `<function=name>` syntax)
* `TestParserRobustness::test_llama3_chat_template_round_trip`
* `TestParserRobustness::test_llama3_round_trip_all_roles`
* `TestParserRobustness::test_llama3_round_trip_with_eot_prefix`
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 118 to 125 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: terminate function-XML body at </function>, not just </tool_call>
`_parse_function_xml` was looking for `</tool_call>` (the Hermes
wrapper) as the body terminator. When a model emits a standalone
`<function=NAME><parameter=K>v</parameter></function>` followed by
explanatory prose (which models routinely do), no `</tool_call>` is
present, so the body extended to end-of-string and the trailing
prose leaked into the LAST parameter value.
Pre-existing on main (the legacy `<function=NAME>` form had this
bug too). Same affects PR #5620's new attribute-form
`<function name="NAME"><param name="K">v</param></function>`
emission used by MiniCPM-5 / MiniMax-M2.
Fix: `_TC_END_TAG_RE` now matches either `</tool_call>` OR
`</function>`. The existing `_TC_FUNC_CLOSE_RE` / `_TC_PARAM_CLOSE_RE`
strips are unchanged. Multi-call inputs still bound each function
at the next `<function=` start, so no over-eager consumption.
New tests:
* `test_function_xml_followed_by_prose` (legacy form + prose)
* `test_function_attribute_xml_followed_by_prose` (attribute form + prose)
Existing `test_code_with_embedded_xml` still passes (a parameter
value containing literal `<a></a>` is preserved because the
embedded close tag is `</a>`, not `</function>`).
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 125 to 127 passed.
* Studio: tighten Llama-3.2 bare-JSON guard
A fuzz pass on PR #5811 turned up that ``_parse_llama3_bare_json``
accepted ``parameters`` as a string, contradicting the docstring's
"parameters or arguments is a dict" guard. Prose JSON like
``{"name":"foo","parameters":"a sentence"}`` would wrongly fire the
parser, which the agentic loop would then heal into a real
``foo(query="a sentence")`` call.
Same code lives on this branch, so the same fix applies here.
Tightened guard:
- ``parameters`` must be a dict (Llama-3 spec).
- ``arguments`` may be a dict, or a JSON-encoded string that
decodes to a dict (OpenAI shape, e.g.
``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings or
JSON-strings of lists / scalars / null no longer pass.
Mirrors the fix landed in PR #5811 commit 615b8608. Adds the same
4 regression tests under TestParserMultiFormat.
Existing test suite stays green: 127 -> 131 passing.
* studio: fix safetensors tool-call parser gaps vs llama.cpp (Mistral CALL_ID / THINK, attribute-form signal)
Three GGUF-parity fixes to the safetensors tool-call parser, each matching
llama.cpp's reference behaviour:
- Mistral Small 3.2 emits [TOOL_CALLS]name[CALL_ID]<id>[ARGS]{json}. The
parser stopped after the name on seeing [CALL_ID] (neither [ARGS] nor {),
dropping the call. Skip an optional [CALL_ID]<id> segment in both the
parse and strip paths. llama.cpp parses this (test-chat.cpp:4785).
- Magistral wraps reasoning in [THINK]...[/THINK]. A [TOOL_CALLS] inside the
reasoning was parsed as a real call, producing a phantom call. Strip a
leading [THINK] block before scanning so only the post-reasoning call
counts (test-chat.cpp:2285); a literal [THINK] inside a later argument is
left intact.
- The standalone MiniCPM-5 / MiniMax-M2 <function name="..."> attribute form
parsed correctly but was absent from TOOL_XML_SIGNALS and the markup strip
patterns, so the streaming safety-net parse was gated off (dropping the
call) and markup leaked into displayed text. Add the signal and broaden
the strip regexes.
Adds regression tests for all three.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fire safetensors tool calls for the bare-JSON (Llama-3.2) form
The agentic loop's streaming safety-net parse was gated on
has_tool_signal(), which is False for the Llama-3.1 / 3.2 bare-JSON tool
form {"name":..,"parameters":..} (no XML marker). Real tool calls were
therefore dropped: the loop logged "model planned without calling tools",
re-prompted three times, then gave up with zero tool calls, while GGUF's
llama-server parses the same emission natively.
Run parse_tool_calls_from_text() unconditionally in the safety net. The
parser is strict (only fires on a valid tool-call shape) so plain answers
are unaffected. Reproduced on a real unsloth/Llama-3.1-8B-Instruct run:
the model emits {"name":"web_search","parameters":{...}} which now
executes the tool instead of being re-prompted into a no-op.
Adds a loop regression test for the bare-JSON form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: complete strict-mode contract and fix parser import paths
Address review findings on the multi-format tool-call parser:
- Honor allow_incomplete=False in the remaining sub-parsers. The Llama-3
<|python_tag|>NAME.call(...) parser, the pre-v11 Mistral [TOOL_CALLS] array
parser, and the Gemma 4 <|tool_call> parser ignored strict mode, so a
truncated call (missing closing paren, ], or <tool_call|>) was still healed
and executed with Auto-Heal disabled. Thread strictness through and reject
the unclosed forms, matching the JSON and function-XML paths.
- Drop the duplicate tool_call_parser import block in llama_cpp.py and the
redundant un-aliased TOOL_XML_SIGNALS; only the _SHARED_TOOL_XML_SIGNALS
alias is used as a value.
- Import _strip_mistral_closed_calls from core.inference.tool_call_parser in
routes/inference.py instead of studio.backend.core... The self-contained
run.py launch mode only puts studio/backend on sys.path, so the absolute
package path raised ModuleNotFoundError on the server-tool strip path.
Add strict-mode regression tests for the truncated Llama-3 dot-call and the
unclosed Mistral array.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve XML param indentation and alias Mistral array parameters
Two parser-correctness fixes found by auditing against the model chat templates
and the SGLang / vLLM reference parsers:
- Qwen3.5 XML parameter values lost their leading indentation. The chat template
emits <parameter=k>\nVALUE\n</parameter>, but the parameter-start regex ate the
wrapping newline AND the value's first-line indentation with a trailing \s*,
then str.strip() removed the rest. Narrow the trailing class to horizontal
whitespace only and trim exactly one wrapping newline (via _trim_param_value),
preserving indentation in code/diff arguments. Matches SGLang's qwen3_coder
detector. Applies to both _parse_function_xml (tool_call_parser.py) and the XML
path in tool_healing.py.
- Mistral pre-v11 array objects keyed on parameters dropped their payload.
_consume_mistral_call read only the arguments key; alias parameters the same way
the JSON/XML paths and SGLang's base detector do.
Add regression tests for preserved multi-line indentation and the array
parameters alias.
* Studio: tighten tool-call parser comments
Make the comments in the multi-format tool-call parser and its callers succinct:
compress verbose docstrings/blocks to one or two lines, drop ones that restate the
code, and trim the tiny balanced-scanner helpers. Correctness rationale and
upstream provenance (SGLang/llama.cpp parity, the strict-mode / Auto-Heal
contract, whitespace-preservation, and the Unicode / full-width-pipe notes) are
kept in compact form.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; parser suite green).
* Studio: make Llama-3 .call and Mistral-array healing parsing linear
Two more O(n^2) ReDoS paths in the multi-format parser, both reachable from
the agentic loop on a long truncated body with no length cap:
- _LLAMA3_KV_RE.finditer over a .call(...) body retried at every offset of a
long word run / unterminated quote (40K -> 14s). Replace with a hand-scan
that reuses the same key/number/literal sub-regexes via anchored match and
walks the string body by hand, so an unterminated quote is O(n). Verified
byte-identical to the old regex over 200K fuzzed inputs.
- _parse_mistral_array healing ran _balanced_brace_end from every { in the
body (20K -> 17s). Walk top-level objects, advancing past each balanced
{...}; this also drops the phantom call the old scan emitted from a nested
argument object.
Add adversarial-length linearity regressions plus positive .call kwargs and
unclosed-array recovery coverage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor strict mode in safety-net, keep empty Gemma args, strip attribute-form function XML
- safetensors safety-net parser now forwards allow_incomplete=auto_heal_tool_calls,
matching the draining path, so a late incomplete tool call is not healed and
executed when Auto-Heal is off.
- Gemma empty bare value ({k:}) now serialises as "" instead of invalid {"k":},
which previously dropped the whole call.
- Route _TOOL_XML_RE also strips the <function name="..."> attribute form
(MiniCPM-5 / MiniMax-M2) so it no longer leaks to the UI.
* Studio: fix attribute-form function-XML literal close tag and zero-arg strict call
Addresses Codex review of the <function name="..."> attribute form in
_parse_function_xml (MiniCPM-5 / MiniMax-M2):
- End the call body at the LAST </function> / </tool_call> within the call's
window, so a literal close tag inside a code/search argument (e.g.
print("</function>")) is preserved instead of truncating the call.
- Accept a closed call with no parameters as a valid zero-argument call in strict
mode (the function close is already required), instead of rejecting it as a
truncated call.
- Tests for both, mirroring the legacy <function=...> coverage.
* Studio: fix tool-call parser/loop review findings on the multi-format path
Address the live code-review findings on the safetensors/MLX + GGUF tool path:
- routes: include the attribute form <function name="..."> in the safetensors
capability whitelist so MiniCPM-5 / MiniMax-M2 templates keep the tool pill
(parser already handles the form; the post-filter wrongly suppressed it).
- safetensors loop: build the plan-without-action re-prompt from the active
tools instead of a hardcoded web_search/python string, and gate it on
auto_heal_tool_calls, matching the GGUF loop.
- safetensors loop: hold a leading bare-JSON object ({"name":..,"parameters":..})
during BUFFERING until it closes, then drain it as a tool call instead of
streaming the raw JSON to clients. The DRAINING/STREAMING resolvers still
recover a plain JSON answer, so this can never drop content.
- parser: anchor the Llama-3 <|python_tag|>NAME.call(...) scan to the tag and
chain ; -separated calls, so all semicolon-separated built-ins parse and a
literal <|python_tag|>x.call(...) inside a JSON string argument no longer
fires the wrong tool.
- parser: consume the optional trailing </s> after a named Mistral
[TOOL_CALLS]name{json} call, mirroring the array shape.
- GGUF streaming strip: use the shared parser patterns (which know
[TOOL_CALLS] and <|python_tag|>) so a textual tool call entering DRAINING is
stripped instead of leaking the marker to streaming clients.
- routes: hoist the _strip_mistral_closed_calls import to module level.
Adds regression tests covering each fix; existing parser suite stays green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden multi-format tool-call detection from review findings
Apply five targeted fixes from the review pass over the multi-format tool
path:
- routes: route display strip delegates to _strip_tool_xml so Mistral
[TOOL_CALLS] blocks with nested JSON are removed from streamed display
text, not just the XML forms.
- tool_call_parser: skip function/parameter starts that fall inside an
already-open parameter block (_inside_open_parameter) so nested example
payloads are not mis-parsed as new calls; extract
strip_llama3_leading_sentinels so the bare-JSON guard is shared.
- safetensors_agentic: probe bare JSON through strip_llama3_leading_sentinels
before the balanced-brace check so a leaked header sentinel does not defeat
the guard.
- tool_healing: allow dotted tool names in the Gemma wrapped start pattern.
- llama_cpp (GGUF): buffer wrapper-less Llama-3.2 {"name":..} calls that carry
no XML signal, drain a complete object silently and hold an incomplete one,
and run the end-of-stream safety net unconditionally so markerless calls are
detected and never leak the raw JSON (including truncated fragments).
Adds regression tests for the GGUF bare-JSON streaming path and the Mistral
display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop bare-JSON tool calls leaking at EOF, oversized, and into history
The second review pass flagged that the Llama-3.2 bare-JSON tool-call handling
still leaked raw JSON in several spots; ``strip_tool_markup`` only knows
XML/bracket markup, so the bare-JSON form survived it. Fix them symmetrically
across the safetensors and GGUF loops:
- Safetensors stream-end resolver now routes a held bare-JSON fragment to
DRAINING (mirroring GGUF) so a truncated ``{"name":..`` cut off by the end of
the stream is dropped instead of flushed as assistant content. The 7/10
reviewer finding.
- Both loops now drain (suppress) an oversized still-open bare-JSON call once it
passes ``_MAX_BARE_JSON_BUFFER`` instead of streaming the raw prefix, gated on
a ``"name"`` key so a giant plain JSON answer still streams; a complete
oversized call still executes via the safety net.
- Add a shared ``strip_leading_bare_json_call`` helper and apply it to the
content kept for the assistant turn in both loops, so an executed bare-JSON
call is not replayed as visible text or fed back as next-turn history.
Plain JSON answers without a ``"name"`` key are untouched throughout. Adds
regression tests for the EOF, oversized, and next-turn cases on both backends
plus unit tests for the helper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the Llama-3 python_tag strip on real control sentinels
The route display strip's <|python_tag|> arm ran to the next <| of any kind.
A tool-call argument carrying a literal <|...|> token (for example <|cite|>
inside a string value) truncated the strip early and leaked the call tail into
the visible response. Narrow the stop condition to the genuine Llama control
sentinels (eot_id, eom_id, python_tag, start/end_header_id, begin_of_text,
finetune_right_pad_id) so embedded markup and JSON are consumed while real
header/turn boundaries still bound the strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate markerless bare JSON on enabled tools and close parser/strip asymmetries
The Llama-3.2 custom_tools bare-JSON form has no marker, so any JSON object with a
name key was read as a tool call. An ordinary JSON answer like
{"name":"Alice","parameters":{"age":30}} was misclassified as a call to a
disabled tool and dropped from the visible response. Gate the markerless form on
the enabled tool names (threaded through parse_tool_calls_from_text and
strip_leading_bare_json_call, supplied by both streaming loops): an object whose
name is not an enabled tool is ordinary content. The marker-based forms keep
their name-agnostic behaviour (an explicit signal is a real call attempt), and
unrestricted mode stays ungated.
Also fix two parser/strip asymmetries the parser already tolerated:
- A literal </function> inside a parameter value (print("</function>")) truncated
both the core and route strips at the first close, leaking the tail. Extend the
strip to the call's real close (last </function> before the next opener),
mirroring the parser, without merging separate calls.
- The single-object Mistral [TOOL_CALLS]{...} shape parsed but _strip_mistral_closed_calls
left it, leaking the raw object into display. Strip the balanced object while
keeping trailing prose, matching the array and name shapes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: gate GGUF bare-JSON suppression on enabled tools and fix python-tag exponent parsing
Pass-4 review follow-ups on the GGUF tool loop and Llama-3 parser:
- The GGUF bare-JSON suppression sites still keyed off a raw "name" substring,
so an ordinary JSON answer whose name is not an enabled tool was dropped when
it was truncated, oversized, or reached the no-tool DRAINING fallback (the
parser, helper, and safetensors paths were already gated). All three sites now
use the shared enabled-name gate, and a held bare-JSON buffer that turns out not
to be an enabled call is shown as the answer instead of dropped at stream end.
- The Llama-3 python-tag numeric kwarg regex matched only the mantissa, so
scientific notation was truncated to its leading digits (1e-3 parsed as 1) and a
tool executed with the wrong value. The regex now accepts exponent and decimal
forms, and the int/float classification keys off the exponent too.
Adds regression tests for the truncated / oversized disabled-name JSON cases (and
a counterpart that a truncated enabled call still does not leak) plus the
scientific-notation kwargs.
* Studio tools: gate safetensors bare-JSON drain, fix nested-name gate and function-XML strip
Pass-4 review follow-ups on the shared parser / safetensors loop:
- The safetensors oversized and end-of-stream bare-JSON drain branches keyed off
a raw "name" substring, so a large or truncated ordinary JSON answer whose name
is not an enabled tool was drained instead of streamed. Both now use the shared
enabled-tool-name gate, matching the GGUF path.
- strip_leading_bare_json_call matched the first "name" anywhere, so a plain JSON
answer with a nested name equal to an enabled tool ({"result":{"name":"web_search"}})
was wrongly suppressed. It now extracts the TOP-LEVEL name only, walking past
nested objects/arrays and keeping the text when a top-level value is truncated.
- The function-XML display strip used a regex negative-lookahead that stopped at a
literal <function=...> opener inside a parameter value and then dropped the rest
of the answer to EOF. A scan-based strip mirrors the parser (ignores openers
inside an open <parameter> via _inside_open_parameter) and closes each call at its
real </function>, so trailing assistant text after such a call survives.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tool parsing: 3.9 import safety, disabled-Auto-Heal contract, capability gate
Round-2 review follow-ups on the multi-format tool-call parser:
- tool_call_parser: add `from __future__ import annotations`. The module
is dependency-light by design (external llama-server wrappers import it
standalone) and the package targets python >=3.9, where its PEP 604
`int | None` return annotations would raise TypeError on import.
- safetensors + GGUF drain fallback: gate the leading bare-JSON strip on
auto_heal_tool_calls. With Auto-Heal off, a truncated enabled-name
fragment that did not parse now stays visible, matching the XML strip
in the same branch and the disabled-Auto-Heal contract. With Auto-Heal
on it is still suppressed.
- safetensors capability gate: match the bare-JSON `{"name":` template
marker with a whitespace/escape-tolerant regex so a pretty-printed
`{ "name" :` or JSON-escaped `{\"name\":` template is not mis-classified
as tool-less. The parser already accepts that whitespace via
raw_decode, so the gate must too.
Regression tests added for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tool parsing: symmetric "function" bare-JSON alias and route strip parity
Round-3 review follow-ups, all parser/strip symmetry fixes.
- Bare-JSON "function" alias: the markerless parser accepts a call name via
obj.get("name") or obj.get("function"), but the strip/gates only knew "name",
so a {"function":<enabled tool>} call executed while its raw JSON leaked. Teach
_top_level_bare_json_name the alias (with "name" precedence and the same nested
and truncated-name guards), and widen the guards in strip_leading_bare_json_call,
the safetensors and GGUF _looks_like_enabled_bare_json gates, and the route
capability marker regex.
- Route display/history cleanup: strip a tail-only </param> alias close (the
parser accepts <param name="...">...</param>), and run the parser's guarded
function-XML scan (_inside_open_parameter) before _TOOL_XML_RE so a literal
nested <function=...></function> inside an argument value does not truncate the
strip and leak the tail.
Regression tests added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: honor tool budget in GGUF loop and guard function-XML streaming strip
Round 4 review fixes. Both are asymmetric-fix bugs where the final/steady path got a
guard the analogous streaming/loop path did not.
- GGUF tool-call budget: the safetensors loop counts real tool-call turns against
max_tool_iterations (re-prompt stalls excepted), but the GGUF loop only bounded the
turn count by the enlarged range (max_tool_iterations + _MAX_REPROMPTS). Since this
PR raised _MAX_REPROMPTS from 1 to 3, a model that keeps making valid tool calls
could run up to three extra tool rounds (with max_tool_iterations=1, four rounds
instead of one). Add a _tool_iters_done counter that increments only when a tool
actually executed in the turn, and stop once the caller's budget is spent so the
post-loop final-answer nudge fires. A duplicate/disabled no-op turn is a correction
turn (like a plan-without-action re-prompt) and does not consume budget, preserving
the existing "already completed" re-prompt behavior.
- Streaming display strip: the final strip runs the guarded _strip_function_xml_calls
scanner (a literal <function=...> inside a parameter value is data, not a nested
call), but the GGUF and safetensors streaming strips still used only the open-ended
regex arms. When a tool-call argument contained literal function markup, the regex
tail ate everything to end-of-text and dropped the real trailing prose after the
call's true </function>. Run the guarded scanner (and the balanced Mistral strip)
before the regex arms in both streaming paths so streaming and final display agree.
Adds regression tests: GGUF valid tool calls respect max_tool_iterations, and the
streaming strip keeps trailing prose after a function-XML call with a literal marker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: safetensors tool budget counts only executed turns (GGUF parity)
Follow-up to the GGUF budget fix. The safetensors loop charged max_tool_iterations
per non-re-prompt iteration (iteration + 1 - reprompt_count), so a duplicate/disabled
no-op turn spent a budget slot even though no tool ran. With a small cap this dropped
real work: for max_tool_iterations=2, a model that made a valid call, repeated it (an
internal no-op correction turn), then made a distinct valid call executed only the
first -- the third turn was sent with no tools and the distinct call was ignored.
Track whether a turn actually executed a tool (set on record_result) and count only
those turns against the cap, matching the GGUF loop. A duplicate/disabled no-op is a
correction turn -- like a plan-without-action re-prompt -- and no longer consumes
budget, so the model still gets its "already completed" nudge and another tool-enabled
turn. Adds a regression test for the small-cap duplicate-then-distinct-call flow.
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: don't force a tool re-prompt on a negated intent (safetensors parity)
The safetensors _INTENT_SIGNAL claimed to mirror GGUF but was missing the
negative lookahead, so a refusal like "I will not search the web for that"
matched the "i will" intent and triggered the plan-without-action re-prompt
(STOP... you MUST call a tool), overriding a valid no-tool answer. GGUF already
excludes not/never. Add the same (?!\s+(?:not|never)\b) lookahead so both
backends agree. Extends the intent parity test with negated refusals.
* Studio: trim redundant comments (comment-only, AST-verified)
* Studio: prevent Gemma tool-parser DoS on stray delimiters
_gemma_parse_value returned the input index unchanged when text[i] was a
stray delimiter (,}]), so the list and mapping caller loops that advance
on the returned index spun forever at 100% CPU on malformed input such as
[},]. Advance past the delimiter so parsing always terminates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip Magistral [THINK] reasoning from final display/history
strip_tool_markup removed [TOOL_CALLS] and <function> markup but left a
leading Magistral [THINK]...[/THINK] block intact, so its bracket-form
reasoning (not the <think> the reasoning channel renders) leaked into the
safetensors display and conversation history while GGUF/llama.cpp routes
it natively. Drop the leading reasoning block at end-of-turn (final=True)
via the existing _strip_mistral_reasoning helper; streaming is untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor reasoning_effort none in safetensors prefill; strip Magistral reasoning while streaming
Two safetensors/MLX reasoning fixes surfaced in review:
_sf_reasoning_prefill_mode only checked enable_thinking, so an
enable_thinking_effort (GLM-5.2) request that disables thinking via
reasoning_effort=none (without enable_thinking=False) still began in
prefilled-<think> mode. A plain answer with no </think> was then swallowed
whole into reasoning_content and the visible response came back empty. Thread
reasoning_effort into the predicate and treat none as disabled, mirroring
_request_reasoning_kwargs.
strip_tool_markup_streaming stripped tool markup but not the leading Magistral
[THINK]...[/THINK] bracket block, so the raw chain-of-thought leaked into the
streamed safetensors content instead of the reasoning drawer (GGUF routes it
natively). Apply _strip_mistral_reasoning first, matching the final strip; an
unclosed [THINK] is held from the marker on so nothing flickers.
* Mistral outer call wins over XML literals; align healer signals with its parser
Two follow-ups on the shared-parser ordering after the healing-passthrough
merge:
- A well-formed [TOOL_CALLS] call whose JSON arguments quote tool XML parsed
the literal instead of the outer call (executing the wrong tool). When the
first XML signal sits inside a leading balanced Mistral body it is argument
data, so the Mistral parser now runs first; an XML signal before the trigger
keeps the normal order, so a [TOOL_CALLS] literal inside an XML call's
arguments still stays data.
- passthrough_healing buffered streams on the parser module's broadened signal
list (now including <|python_tag|> and [TOOL_CALLS]) but promotes with
core.tool_healing, which does not parse those forms: a streamed Mistral or
Llama text call was held until finalization and flushed as prose. The healer
keeps its own signal list limited to the formats it can promote, restoring
immediate streaming for the rest.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: leading envelopes win over rehearsed literals
- New _first_foreign_tool_signal shared by the leading-envelope guards adds
<|python_tag|> to the protected signal set: the spelled-out literal inside a
Mistral call's arguments (a query about Llama built-in tool syntax) executed
the inner literal instead of the outer call.
- New _xml_signal_inside_leading_bare_json guard, sibling of the Mistral one:
a leading bare-JSON call whose string argument quotes tool XML (a code value
citing <function=...>) had the literal promoted by the shared XML pass
before the bare-JSON parser ran.
- Magistral [THINK]...[/THINK] is dropped once at parse entry instead of only
inside the Mistral parser, so a call rehearsed in the think block in a
foreign format can no longer be promoted while the real call after the
block is lost. Parse now agrees with the display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: a disabled leading bare-JSON object keeps its literals as data
When the leading bare-JSON object is ordinary content (name not an enabled
tool), the guard proved the first tool signal sits inside it, so falling
through to the XML/python_tag passes promoted quoted string data as a real
call. Drop the object and parse only the tail: a real call after the object
still parses, nothing inside it can be promoted.
* Address review: Mistral literals inside leading JSON, whitespace-tolerant wrapped Gemma opener
- The leading bare-JSON guard now treats the [TOOL_CALLS] trigger as a
foreign signal: the Mistral parser runs before the bare-JSON one, so a
literal quoted inside the leading object's strings was promoted over the
outer call (or over ordinary JSON content).
- tool_healing's wrapped Gemma opener tolerates whitespace around call and
the colon: sampling drift emits call: name{ and call : name{, and
rejecting those lost the call entirely because no fallback re-parses the
wrapped form. Strict mode still requires the closing tag.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept dotted Gemma argument keys in the key-quoting scanner
The scanner quoted keys of [alnum_-] only, so a dotted key (user.name:...)
was left unquoted, json.loads failed, and the whole wrapped call was lost
(parse empty, strip wipes the markup). Dots now match the parser's own
key/name charset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: leading Mistral call owns the turn, dotted keys after bare values
- A LEADING parseable [TOOL_CALLS] call now runs the Mistral parser first
unconditionally: literal XML in trailing prose after the call was promoted
by the earlier shared XML pass, executing the quoted example instead of
the real leading call. XML leading keeps the normal order.
- _GEMMA_NEXT_KEY_RE accepts dots so a dotted key after a bare value
(query:foo,user.name:bob) ends the value at the comma instead of being
swallowed into it, matching the round-earlier key-quoting charset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: markup quoted inside a nameless leading JSON answer stays data
The leading bare-JSON guard required a top-level name, so a structured JSON
answer quoting tool markup in its strings (a response_format turn
documenting a tool's syntax) had the literal promoted by the later passes.
A nameless leading object that parses as real JSON now routes through the
same decline-then-parse-the-tail path; non-JSON braced prose keeps the old
behaviour, and a real call after the answer still parses.
* Compress docstrings in the multi-format tool parser to their contract essence
* verify_import_hoist: exempt __future__ imports and same-diff relocations
Two false positives fired on this PR's refactor. A from __future__ import
is a compiler directive whose name never appears as a runtime load, so
HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for
PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate
move of the strip-pattern constants into core.inference.tool_call_parser
as a silent re-point even though the old module-level target was removed
and the new one added in the same diff. Both get narrow exemptions; a
re-point to a pre-existing target is still caught, and the self-test
negative controls all pass unchanged.
* Leading bare-JSON calls own the turn; function calls end at the first balanced close
The XML-signal guard for a leading bare-JSON call required the signal
strictly inside the object, so a trailing XML example stole the turn
from the leading call; it now applies the same inside-or-after rule as
the Mistral guard. Function-XML calls also ended at the LAST close tag,
which let prose after a closed call that mentions a literal close tag
get swallowed into the final parameter value; calls now end at the
first close tag that is not inside an open parameter, and the strip
mirrors the same rule so parse and strip agree.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Attribute-form calls end at the first balanced close; bare-JSON strip requires the call shape
The attribute form parser still kept the last close tag in the call
window, folding prose after a closed call into the final parameter
value. It now takes the first close not inside an open parameter, the
same rule the equals form and the strip already use.
The leading bare-JSON strip deleted any closed object whose top-level
name matched an enabled tool, including plain JSON answers the parser
correctly rejects as non-calls. The strip (and the drain gate that
delegates to it) now requires the parser's exact call shape, so answers
like {"name":"web_search","result":...} stream and display intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* False-alarm markers keep the answer; the bare-JSON strip consumes the whole chain
The trailing strip arms dropped everything from a bare marker to EOF,
so a normal answer that mentions [TOOL_CALLS] or another marker
literally was truncated (or fully swallowed when it started with the
literal) after the no-call drain fallback. Those arms now require a
call-shaped lookahead or marker-at-EOF before dropping; truncated real
calls still strip.
Chained bare-JSON turns executed both calls but stripped only the first
object, so the second call's raw JSON replayed into the next assistant
history message alongside the structured tool_calls. The strip now
consumes the entire chained run of call-shaped enabled objects while
non-call answers, disabled names, and trailing prose stay intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Attribute-form containment, parameter-close-decides rule, preamble-tolerant Mistral guard, strict strip shape
Four document-order and containment fixes. A leading attribute-form
call now parses before the shared XML pass, so markup quoted in its
parameter stays data. The open-parameter scan lets the parameter's own
close tag decide, so any number of literal function closes inside one
value stay data, restoring the pre-close-scan behavior for multi-close
arguments. The leading-Mistral guard tolerates a visible preamble, with
the leading-bare-JSON guard running first so a trigger quoted inside a
leading JSON object stays data. The bare-JSON strip requires the
parser's top-level name in every mode, so nested-name JSON answers
survive name-agnostic stripping.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Let a leading <|python_tag|> call own the turn over quoted XML literals
The leading-call ownership contract (a leading executable call owns the turn;
foreign markup quoted in its string arguments or trailing prose stays data) was
enforced for the bare-JSON, Mistral and attribute-form leading calls but not
for the Llama-3 <|python_tag|> form. The shared tool_healing XML pass runs
before _parse_llama3_python_tag and does not recognise <|python_tag|>, so a
<function=...> / <tool_call> / [TOOL_CALLS] literal quoted inside a
<|python_tag|> .call(...) string argument (or its JSON parameters) was promoted
and the wrong tool executed. Well-formed single-format examples:
<|python_tag|>web_search.call(query="... <function=foo> ...") -> foo
<|python_tag|>python.call(code="<function=render_html>..</function>") -> render_html
both returned the phantom inner tool instead of the real leading call.
Add a leading-<|python_tag|> guard mirroring the other leading-call guards:
when the tag is the first tool signal, parse it before tool_healing so quoted
foreign markup stays data. A foreign signal before the tag keeps normal
document order. Added TestPythonTagOuterOverXmlLiteral (7 cases).
* studio: tighten tool-calling comments to be shorter and clearer
* studio: shorten tool-format comments in changed files
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
ruff-format requires two blank lines before a top-level function.
loader.py carried only one, so the ruff-format-with-kwargs pre-commit
hook reformats it and the run fails. This restores the expected spacing.
* Fix llama3 RoPE scaling dropped on transformers v5
transformers v5 loads on meta then blanks non-persistent buffers, so
_fix_rope_inv_freq rebuilds inv_freq after load. It recomputed a vanilla
inv_freq and applied _apply_inv_freq_scaling, a no-op on the base
LlamaRotaryEmbedding used by the config/llama3 path, so inv_freq ended up
divided by 1 instead of the config factor (8 for Llama 3.1, 32 for Llama
3.2). This corrupts long-range positions and inflates long-context loss
about 3-5x. transformers 4.x was unaffected.
Route __init__ and the v5 repair through one _unsloth_recompute_inv_freq
so they cannot diverge, and stash the config on the rotary module so the
repair can rebuild the same scaled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add test for llama3 RoPE scaling under the transformers v5 repair
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update RoPE drift guard for the recompute refactor and guard the v5 repair
The drift guard's AST tripwire asserted the config-scaling call lived in the
if config is not None branch of LlamaRotaryEmbedding.__init__. The fix moved
that into _unsloth_recompute_inv_freq, so follow it there (with a fallback to
the old inline branch) and add a guard that loader._fix_rope_inv_freq rebuilds
inv_freq through the same helper. Also add a CPU functional check of the helper
and drop the redundant standalone test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The live resource monitor and GPU readouts derive memory from binary byte
counts (bytes / 1024**3 for torch and psutil, MiB / 1024 for the nvidia-smi
path), which is GiB, but the UI labeled the values "GB". On a B200 this
showed "178.35 GB" for a card whose nvidia-smi total is 183359 MiB
(179 GiB), so it looked like memory was missing.
Relabel the measured RAM and VRAM readouts to GiB across the floating
monitor, the resources tab, the studio live GPU panel, the hub header, the
about tab and the onboarding summary. The numeric values are unchanged, so
the training GPU selection and memory-fit logic that read the same fields
are unaffected. Disk stays labeled GB because the backend reports it in
decimal GB (bytes / 1e9), and model file sizes and download progress keep
their decimal GB labels to match Hugging Face.
* Note the bundled flash-linear-attention kernels for gated-deltanet models
Unsloth Zoo now bundles the flash-linear-attention (fla) gated-delta Triton
kernels and injects them automatically, so gated-deltanet models (Qwen3-Next,
Qwen3.5, Kimi-Linear) get the fast path with no pip install. Replace the old
install advisory with a one-time note that fires only when the bundled kernels
could not be enabled on the current setup (no CUDA, or torch < 2.7 / triton < 3.3),
i.e. exactly when transformers falls back to the slow pure PyTorch path.
* Tighten comments
* Normalize model_types in fla install advisory for None and single string
* Cover olmo_hybrid in the gated-deltanet fla advisory
* Add gemma4, glm4_moe and qwen3_moe to the FORCE_FLOAT32 fallback list
Keeps the fallback list (used only if the unsloth_zoo import fails) in sync with
unsloth_zoo/model_lists.py, which now force-float32s these MoE archs so a float16
request loads bf16 and trains finite instead of NaNing the grad_norm.
* Union FORCE_FLOAT32 fallback so new archs force float32 with older unsloth_zoo
A model path ending in -bf16 unconditionally forced 16-bit loading, so a
LOCAL checkpoint directory whose name happens to end in -bf16 could never be
loaded in 4-bit, 8-bit or fp8: the suffix rule silently overrode the caller's
quantization flags. Hub repo ids keep the existing behavior (the suffix is a
publishing convention there), but for a local directory (expanduser-aware, so
tilde paths are detected too) the requested quantization is preserved unless
the caller explicitly passes load_in_16bit=True.
* Auto-enable grouped MoE on loaded / PEFT'd models via loader hook
Wraps the FastLlamaModel and FastBaseModel from_pretrained / get_peft_model leaves with wrap_loader_for_grouped_moe so the grouped-GEMM MoE forward is installed on the live instance after the model and its compiled module are built. Gated by UNSLOTH_MOE_GROUPED and wrapped in try/except, so it is a no-op when the unsloth_zoo module is absent or no eligible MoE block exists.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Install grouped-MoE loader wrappers before PatchFastRL
* Re-evaluate grouped MoE after loading a PEFT adapter
When loading an existing adapter through FastLanguageModel.from_pretrained,
the base model is evaluated for grouped MoE when the wrapped from_pretrained
leaf returns, but the adapter is attached afterwards via PeftModel and
patch_peft_model. Re-run auto_enable_grouped_moe on the final model so
blocks whose experts gained LoRA are restored to the original loop,
attention-only adapters keep the grouped path on their frozen experts, and
recompute is re-derived from the final gradient-checkpointing state. Guarded
so it never blocks adapter loading.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the grouped MoE loader hooks
Shorten the loader re-eval and llama.py wrapper comments; code is unchanged
(verified comment-only).
* Re-evaluate grouped MoE after loading a PEFT adapter on the vision path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Honor an explicit sdpa or flex_attention request when flash is disabled
When flash attention is disabled for a model, the fallback selection could
downgrade a caller who explicitly passed attn_implementation='sdpa' or
'flex_attention' to a different backend, because the disable reason is
flash-specific. Keep an explicit non-flash request as-is; flash requests
still fall back as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Gate honor-explicit attention on provenance and flex support
Only honor an explicit non-flash attention request when it comes from the
caller argument, not from a config value the loaders synthesize (the language
path seeds attn_implementation=sdpa). Honor explicit flex_attention only when
supports_flex_attention is True so excluded/broken configs (e.g. gpt_oss) fall
back instead of selecting a known-broken backend. Explicit sdpa stays honored.
* Honor explicit sdpa through the resolver guard
* Keep SDPA exclusions when honoring an explicit sdpa request
An explicit attn_implementation="sdpa" was re-enabling sdpa for models in
_SDPA_EXCLUDED_MODELS (e.g. gpt_oss) where sdpa is known-broken: the helper
honored the request and the resolver's final not-supports_sdpa guard skipped
the eager downgrade for any explicit request. Honor an explicit sdpa only when
the model is not sdpa-excluded, mirroring the flex guard that already falls
back for _FLEX_EXCLUDED_MODELS via supports_flex_attention. Conservative
supports_sdpa=False (large head dim / attention-sink models) still honors an
explicit sdpa; a synthesized/default sdpa still downgrades to eager.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor DISABLE_SDPA_MODEL_NAMES when honoring explicit sdpa
The honor-explicit-sdpa guard only skipped the sdpa->eager downgrade for models
in _SDPA_EXCLUDED_MODELS (gpt_oss). Gemma3/Gemma3Text disable SDPA through the
loader's DISABLE_SDPA_MODEL_NAMES (their bundled SDPA modules are wrong), so an
explicit sdpa request bypassed the downgrade and re-enabled a known-wrong path.
Extend _is_sdpa_excluded to also treat DISABLE_SDPA_MODEL_NAMES membership as
excluded, replicating the loader's trailing-comma substring match so gemma3 and
gemma3_text match but gemma3n does not. Move the constant into _utils.py (single
source of truth, re-exported from loader.py) to avoid a loader -> _utils cycle.
Conservative supports_sdpa=False models not in either list still honor explicit
sdpa.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Scope MoE expert LoRA detection to actual MLP projection targets
_moe_target_set_from_string treated any regex containing the substring mlp
or ffn as targeting the expert MLP projections. Unsloth's auto-generated
attention-only regex lists mlp, ffn and feed_forward as allowed intermediate
path segments while its final group matches only q_proj/k_proj/v_proj/o_proj,
so attention-only finetuning on MoE models silently enabled expert LoRA as
well: the experts were trained and every MoE layer paid the extra expert LoRA
grouped matmuls. Detect expert intent from the projection names themselves
(gate_proj/up_proj/down_proj/gate_up_proj) instead of the mlp substring.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Detect MoE expert LoRA via mlp path segment, not proj names
The auto-generated target regex always lists every projection leaf
(q/k/v/o and gate/up/down), so keying detection on a proj name mis-fired:
it enabled expert LoRA for attention-only regexes and dropped the
mlp/ffn path regexes. Key on the mlp/ffn/feed_forward/experts path
segment instead, which is present only when the MLP/experts are actually
targeted. Add a regression test for the attention-only case.
* Scope expert LoRA targets to the leaves a regex names
An mlp path alternative with attention-only leaves, for example
(mlp|self_attn).(q_proj|o_proj), no longer enables expert LoRA, and a
regex naming a single expert leaf such as .*experts.*down_proj now
targets only that projection instead of the whole broad set. Generic
mlp projections (.*mlp.*proj) and the auto regex mlp tag block keep the
broad set for fused-expert models whose leaves are plain Parameters.
* Route explicit leaf list into MoE expert detection
An attention-only explicit target_modules list routed through get_peft_regex
for family scoping (e.g. FastVisionModel with vision layers off) yields a
regex carrying the full mlp|feed_forward|ffn|dense component block even though
its leaf group only names q/k/v/o_proj. Keying expert detection on that regex
trained the experts for a language-only/attention-only request. Use the
caller's original leaf list for detection; only the auto path uses the regex,
where the mlp block is the sole MLP-intent signal on fused-expert models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Respect finetune_mlp_modules and finetune_language_layers scope for MoE expert detection
When an explicit leaf list that names MLP projections (gate_proj/up_proj/down_proj)
is routed through get_peft_regex under finetune_mlp_modules=False, the scoped regex
correctly drops the MLP leaves, but MoE expert detection was still keyed on the
original list and re-added mlp.experts.* via target_parameters, training the experts
the caller had frozen. Same gap for finetune_language_layers=False on vision-only runs.
Prefer the original list only when MLP and language families are both in scope
(preserving the attention-only fix); otherwise honor the scoped result so the frozen
family is respected. Factored the choice into _select_moe_detection_targets with unit
tests over the full selection matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Handle odd shapes and non-float scales in FP8BlockQuantLinear
Small fp8 checkpoints (e.g. tiny test models) break the block-quantized
linear in three ways: weight scales stored in a float8 dtype such as
float8_e8m0fnu have no triton dtype mapping; activations whose hidden dim is
not a multiple of the activation quant block fail act_quant's divisibility
assert; and weights whose dims are not multiples of the weight block cannot
be tiled by the triton dequant kernel.
Cast non-float scales to float32 on entry, and when the hidden dim does not
divide into the activation block, dequantize the weight and run a plain
matmul instead of the fp8 block matmul. The dequant goes through a new
shape-safe helper that falls back to a torch-native scale expansion when the
weight does not tile evenly; backward uses the same helper so the gradient
path works for every shape the forward accepts. Full-size checkpoints are
unaffected.
* Add tiny / e8m0 fp8 block-quant regression test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix FP8 block-quant fallback: real block size in dequant and scalar-scale fast path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route rectangular fp8 blocks through torch dequant and keep block_size across e8m0 upcast
The triton weight_dequant kernel uses one BLOCK_SIZE for both axes, so
rectangular blocks (block_size[0] != block_size[1]) mis-index the column
scale and corrupt grad_X. Route those through the torch scale expansion,
which handles each dimension independently, and keep the triton path for
square blocks only.
Also preserve a block_size attribute carried on the scale tensor across the
e8m0 -> float32 upcast so the later lookup no longer falls back to [128, 128].
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* GRPO: optional sequence packing for the no-grad old/ref logp path
Add an opt-in sequence-packing fast path to _get_per_token_logps_and_entropies, enabled with
UNSLOTH_GRPO_SEQ_PACKING=1. When the batch is text-only, the padded [B, Lmax] per-chunk forward is
replaced by a single varlen [1, sum L] forward (BlockDiagonalCausalMask via packed_seq_lengths with
reset position_ids). Per-token logps use the same float32 chunked_hidden_states_selective_log_softmax
as the padded path, so the old and reference logps are bit-for-bit identical.
Safety: the packed path is self-verified once against the padded ground truth on a batch that has at
least two rows with real completion tokens (self._unsloth_seq_packing_nograd_ok), so cross-sample
contamination would actually manifest; a degenerate all-pad / fully tool-masked batch leaves the
verdict unset and re-verifies later. If a backend silently ignores packed_seq_lengths (flat batch run
under a normal causal mask, samples leaking across boundaries), the packed logps will not match and
packing is disabled instead of corrupting logps. It also forces use_cache=False (a populated
past_key_value disables varlen packing), skips packing when a sliding window is shorter than the
packed stream, runs the same GPT-OSS offload device_synchronize the padded loop uses, and falls back
on any exception (UNSLOTH_GRPO_SEQ_PACKING_DEBUG=1 prints the reason).
Default off, so existing behavior is unchanged. Pairs with the matching gradient-path change in
unsloth_zoo so the full GRPO logp + loss + backward can run packed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: address review feedback
- Cache the packed-vs-padded verdict per unwrapped model instead of on the
trainer, so a separately forwarded reference model is verified on its own
forward path rather than inheriting the policy model's verdict.
- Force the padded path when token_type_ids or mm_token_type_ids are present,
matching the extra vision kwargs the padded loop forwards.
- Require the xformers varlen backend before packing. Without it the packed
mask falls back to a dense O(T^2) SDPA mask that can OOM on the flattened
batch, so we keep the padded loop in that case.
- On any packed-forward failure (missing backend, OOM, unsupported forward)
empty the cache on OOM, disable packing for that model, and fall back to the
chunked padded loop instead of retrying every step.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: default-on, verify against per-row reference
Redesign of the optional sequence-packing fast path for the no-grad
old/ref logprob recompute, after establishing that the packed forward is
the exact per-row computation and the padded batch forward is the side
that mis-positions left-padded rows on long completions.
- Default the packing on (UNSLOTH_GRPO_SEQ_PACKING, disable with 0).
- Verify the packed logprobs against the per-row clean forward (each
row's real tokens alone, reset 0-based positions, no padding), not the
padded batch which is itself wrong for left-padding. Cross-sample
contamination (a backend ignoring packed_seq_lengths) shows up as a
large mismatch and falls back to the padded loop.
- Make the trust decision shape and RoPE aware: re-verify whenever the
packed total length or the longest segment grows past what was
verified, so a later batch crossing a LongRoPE short/long cache
boundary is re-checked instead of trusted blindly.
- Run lm_head only on completion-prediction positions instead of every
packed prompt token, so long-prompt/short-completion batches do not
pay for projecting the whole packed prompt.
- Drop the hard xformers import so the path also runs in
FlashAttention-only environments; the per-row verification guards
correctness regardless of backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: disable entirely on cross-sample mismatch
When the per-row verification fails, distinguish the two failure modes by
magnitude instead of by sequence length:
- A large mismatch (>= 1.5) is the cross-sample contamination signature:
the model's attention does not honor the block-diagonal packed mask
(seen on some MoE / custom-attention models, e.g. qwen2_moe). Disable
packing entirely for the model so later batches do not pay the
verification cost again.
- A moderate mismatch is more likely a length-boundary effect (a LongRoPE
short/long cache switch): keep marking just that length region unsafe so
packing still runs for smaller shapes.
Validated: Qwen1.5-MoE falls back after a single verification (grad and
no-grad ok flags go False, no re-verify on later steps); dense Llama-3.2
and Qwen3 still verify and engage packing.
* GRPO no-grad packing: trim comments to be concise
* GRPO no-grad packing: fix per-row completion boundary for left-padded rows
The completion-target selection used a single global boundary
(col >= L - logits_to_keep). After left-packing, each row's completion
starts at (L - logits_to_keep) - left_pad[row], so for left-padded rows
the first left_pad completion tokens fall below the global boundary and
were dropped, leaving 0 logprobs at real completion positions that the
loss mask keeps. Use the per-row boundary so packed coverage matches
create_completion_attention_mask exactly, and widen the self-verify mask
to the full per-row completion region so it can catch coverage gaps.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: gate verification on real completion rows
Count active rows via create_completion_attention_mask (the same mask the
loss uses) instead of any non-pad token in the packed window. Prompt-only
rows carry prompt-overflow tokens in the window and could otherwise satisfy
the >= 2 verification guard, letting a batch with a single real completion
row cache a trust decision. This matches the gradient path, which already
gates on the completion mask. The same mask is reused for the self-verify
comparison.
* GRPO no-grad packing: gate debug logging on UNSLOTH_ENABLE_LOGGING
Use the shared UNSLOTH_ENABLE_LOGGING global (import_fixes, re-exported by
_utils) instead of a bespoke UNSLOTH_GRPO_SEQ_PACKING_DEBUG env var for the
packing debug prints, matching the rest of the codebase.
* GRPO packing: import UNSLOTH_ENABLE_LOGGING inside the injected logp function
_get_per_token_logps_and_entropies is copied verbatim into the generated GRPO trainer
via inspect.getsource, and that module never imported UNSLOTH_ENABLE_LOGGING, so the
default-on packing verify path raised NameError (and the except handler re-raised it).
Import the flag locally, before the try, so the name is defined in the generated module
too. Drop it from the now-unused module-level import.
* GRPO no-grad packing: harden unsafe-length skip, verify guard, fallback cleanup
Three fixes to the no-grad logp packing path, mirroring the grad path:
- skip the packed forward for known-unsafe lengths by reading unsafe_T and
gating on it before the forward, instead of running the full packed pass and
the result build only to discard them (wastes a pass, can OOM at large T)
- only widen the verified T/seg envelope when >= 2 completion rows actually
exercised cross-sample packing; a < 2 row batch cannot expose leakage, so it
must not extend the trusted shape that later multi-row batches skip verify for
- drop the packed intermediates (hidden/sel/result/ref) before the padded
fallback loop so it does not run with the flattened hidden state still resident
* GRPO no-grad packing: cap the flattened forward at one mini-batch budget
The packed path built a single [1, sum L] forward over every row before any
size check, so a large batch could exceed the memory the padded path bounds
per mini-batch. Gate packing on _pk_T <= _pk_cap (B * seq_len, one padded
mini-batch's token budget); larger batches fall back to the chunked padded
loop.
* GRPO no-grad packing: disable unless unsloth_zoo has the masked-column guard
The packed path leaves masked prompt/pad logprob columns at 0, which only stays
finite if unsloth_zoo grpo_compute_loss zeroes them before exp() (zoo#840). An
older unsloth_zoo without that guard would NaN. Detect the guard once (cached on
the model) via inspect.getsource and gate packing on it, so #6738 is safe with
any unsloth_zoo version and re-enables packing automatically once a guarded zoo
is installed, independent of the pinned lower bound.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: hoist env gates and zoo-guard detection to one-time module checks
Read UNSLOTH_GRPO_SEQ_PACKING and detect the unsloth_zoo masked-column guard once at
import time (module constants plus RL_PRE_ITEMS for the generated trainer cache) instead
of per call, and drop the in-function UNSLOTH_ENABLE_LOGGING import for a module-top one.
The UNSLOTH_GRPO_SEQ_PACKING_VERIFY force-verify debug knob is commented out, kept in
place for hand re-enable; the first-use and envelope-growth self-verify stays active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: cap the flattened forward by the padded chunk rows
B counts chunks at this point, so B * seq_len understated (small runs) or overstated
(large runs) the padded mini-batch token budget; use batch_size * seq_len, the rows the
padded loop actually forwards per chunk.
* Add PrefixGrouper for GRPO: dedup the shared prompt across a group's completions
In GRPO every prompt spawns G=num_generations completions that share the prompt
prefix, so the trunk logprob forward re-encodes that prefix G times. PrefixGrouper
stores the prefix once and concatenates only the G suffixes behind a FlexAttention
shared-prefix mask, cutting the forward from G*(P+R) to P+G*R tokens across both the
no-grad old/ref forwards and the grad logp forward. Default off behind the
UNSLOTH_GRPO_PREFIX_GROUPER env gate, so the gate-unset path is byte-identical to
today. A tok_r auto-gate and a first-use self-verify (fall back and mark the shape
unsafe on mismatch) keep it from ever shipping wrong logprobs silently.
Wired for llama, mistral, qwen3, gemma2, cohere, granite and falcon_h1, plus qwen2
and gemma through the shared LlamaAttention_fast_forward. Stacked on the GRPO
sequence-packing PR (#6738); the grad path lands in a companion unsloth-zoo PR.
Also fixes a latent UNSLOTH_ENABLE_LOGGING NameError in the seq-packing no-grad
verify path by defining the name as a generated-cache pre-item.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: enforce the sliding-window cap, gate softcap models, bound the mask cache
Add a max_segment_cap kwarg to build_group_layout so it falls back when a group's
span (prefix + longest suffix) exceeds the model's local window, and pass the config
sliding_window into the no-grad engage gate the same way the packed _pk guard derives it.
Skip PrefixGrouper entirely for attn_logit_softcapping models, since the FlexAttention
kernel never applies logit softcapping. Bound _BLOCK_MASK_CACHE to a FIFO of 8 so
per-step lengths cannot pin BlockMasks forever, release the PG hidden before the verify
forward, and align the UNSLOTH_ENABLE_LOGGING pre-item truthiness with the canonical form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: vectorize the real-column scan in build_group_layout
Replace the per-row O(B*L) Python scan of the keep mask with a GPU-derived
contiguous-run fast path (first real column + count per row), keeping the
general scan only as a fallback for non-contiguous rows. Works for both call
sites: the no-grad layout (left-padded prompt + right-padded completion, run
does not start at column 0) and the grad layout (left-packed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: hoist the gate and kernel imports to one-time module checks, AGPLv3 headers
Read UNSLOTH_GRPO_PREFIX_GROUPER and resolve the prefix_grouper imports once at module
level (source constants plus an RL_PRE_ITEMS entry for the generated trainer cache)
instead of per call, matching the sequence-packing gates. The prefix_grouper env helpers
become one-time module reads with unchanged signatures, and attention_dispatch resolves
the FlexAttention kernel once behind the same gate (lazy fallback kept). The two new
prefix_grouper files move to AGPLv3 headers.
* PrefixGrouper: length-envelope trust and hybrid SSM exclusion
Verified signatures now record (max T, max segment) and re-verify when either grows,
matching the packed path's envelope. Hybrid SSM models (FalconH1 etc.) are excluded at
the gate since only attention gets the shared-prefix isolation, and the FalconH1 wiring
is removed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: defer the unverified no-grad forward until the packed reference exists
Unverified shapes no longer run the whole-batch shared-prefix forward up front; it now
runs at the verify site, only when the packed path produced a reference. A declined
packed path (budget, window) therefore costs no wasted PG forward per step. Trusted
shapes still run it first to skip the full-row forward, with the same fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: disable under vLLM (fast_inference=True)
With colocated vLLM generation the rollout dominates the GRPO step, so the shared-prefix
training forward saves little end-to-end and its first-use self-verify (which also runs
the full-row path) is net overhead. Gate PG on not use_vllm so it only engages on the raw
transformers path, where the training forward is on the critical path. Packing is unaffected.
* PrefixGrouper: compile the FlexAttention kernel with dynamic shapes
GRPO changes the packed length T almost every batch. With dynamic=False the flex
forward+backward kernel recompiled on every new T (~14s each on a 4B trunk), which
dominated the step and made PG a net loss. dynamic=True compiles once, then reuses the
kernel across all lengths recompile-free (a new shape drops from ~14s to ~1.4ms after a
two-graph warmup). T is still padded to a multiple of 128 for the backward block assertion.
* PrefixGrouper: default on
Enable PrefixGrouper by default (UNSLOTH_GRPO_PREFIX_GROUPER defaults to 1; set 0 to
disable). Still auto-disabled under vLLM (fast_inference=True) and by the arch/softcap/
SSM/tok_r gates, and the first-use self-verify falls back on any mismatch, so this is a
memory-first default on the raw-transformers path with no correctness risk.
* GRPO PrefixGrouper: gate on zoo masked-column guard and exclude MoE
- Require the zoo masked-column guard (zoo#840) before PrefixGrouper can engage.
PG rides the sequence-packing path, so when the first-step self-verify is off the
fast path trusts PG output directly; without the guard those masked columns feed
NaN into the packed loss. Gate PG on the same UNSLOTH_ZOO_HAS_MASKED_COL_GUARD
the packing path already checks.
- Exclude MoE configs (num_experts, num_local_experts, n_routed_experts,
moe_intermediate_size) alongside the hybrid-SSM markers. Only the threaded
attention forwards carry the shared-prefix isolation, so a MoE decoder that does
not forward prefix_seg_info would let suffixes leak across completions.
- Refresh the stale default-off comments now that UNSLOTH_GRPO_PREFIX_GROUPER is
on by default.
* GRPO PrefixGrouper: import chunked_hidden_states_selective_log_softmax
The shared-prefix forward passes chunked_hidden_states_selective_log_softmax
into extract_logps, but the name was only ever provided by the generated
trainer cache (rl.py injects grpo_selective_log_softmax_code), never bound in
this module. Import it from unsloth_zoo.rl_replacements next to its sibling
chunked_selective_log_softmax so the source resolves the name in every scope
(the new _pg_run_forward closure included). No runtime change: the cache still
defines the function via template injection.
* GRPO PrefixGrouper: dropout gate, device-safe layout, Mistral mask skip
Addresses three review findings on the shared-prefix path:
- Skip PrefixGrouper when the model sets a nonzero attention_dropout. The normal
backends apply config.attention_dropout while training (e.g. Granite dense
flash/sdpa/xformers), but the FlexAttention shared-prefix path is deterministic,
so gate PG off for those configs rather than train on mismatched activations.
- Move the shared-prefix mask labels to the consumer (Q) device in get_block_mask
and the target index maps to hidden.device in extract_logps, mirroring the packed
path moving its metadata to the consumer device. Prevents cross-device indexing
when the model is sharded across GPUs.
- Do not synthesize a causal attention_mask in the Mistral forward when
prefix_seg_info is present. On the no-xFormers path that synthetic mask tripped
resolve_prefix_seg_info and forced PG to always fall back to the packed forward.
* GRPO sequence packing: tighten comments
* GRPO PrefixGrouper: tighten comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO PrefixGrouper: persistent disable on runtime failure; build block-mask labels with inference mode disabled
- rl_replacements: on a PG forward exception (FlexAttention/Triton compile failure or OOM), set a model-level _unsloth_prefix_grouper_nograd_disabled flag and consult it in the engage gate, mirroring the seq-packing handler, so a GPU-wide failure is not retried and re-paid every step.
- prefix_grouper_kernel: move the .to(device) label copies inside the inference_mode(False) block so a cross-device (model-parallel shard) first build does not capture inference tensors, which otherwise cannot be saved for backward when the grad training forward reuses the cached BlockMask.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* GRPO: optional sequence packing for the no-grad old/ref logp path
Add an opt-in sequence-packing fast path to _get_per_token_logps_and_entropies, enabled with
UNSLOTH_GRPO_SEQ_PACKING=1. When the batch is text-only, the padded [B, Lmax] per-chunk forward is
replaced by a single varlen [1, sum L] forward (BlockDiagonalCausalMask via packed_seq_lengths with
reset position_ids). Per-token logps use the same float32 chunked_hidden_states_selective_log_softmax
as the padded path, so the old and reference logps are bit-for-bit identical.
Safety: the packed path is self-verified once against the padded ground truth on a batch that has at
least two rows with real completion tokens (self._unsloth_seq_packing_nograd_ok), so cross-sample
contamination would actually manifest; a degenerate all-pad / fully tool-masked batch leaves the
verdict unset and re-verifies later. If a backend silently ignores packed_seq_lengths (flat batch run
under a normal causal mask, samples leaking across boundaries), the packed logps will not match and
packing is disabled instead of corrupting logps. It also forces use_cache=False (a populated
past_key_value disables varlen packing), skips packing when a sliding window is shorter than the
packed stream, runs the same GPT-OSS offload device_synchronize the padded loop uses, and falls back
on any exception (UNSLOTH_GRPO_SEQ_PACKING_DEBUG=1 prints the reason).
Default off, so existing behavior is unchanged. Pairs with the matching gradient-path change in
unsloth_zoo so the full GRPO logp + loss + backward can run packed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: address review feedback
- Cache the packed-vs-padded verdict per unwrapped model instead of on the
trainer, so a separately forwarded reference model is verified on its own
forward path rather than inheriting the policy model's verdict.
- Force the padded path when token_type_ids or mm_token_type_ids are present,
matching the extra vision kwargs the padded loop forwards.
- Require the xformers varlen backend before packing. Without it the packed
mask falls back to a dense O(T^2) SDPA mask that can OOM on the flattened
batch, so we keep the padded loop in that case.
- On any packed-forward failure (missing backend, OOM, unsupported forward)
empty the cache on OOM, disable packing for that model, and fall back to the
chunked padded loop instead of retrying every step.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: default-on, verify against per-row reference
Redesign of the optional sequence-packing fast path for the no-grad
old/ref logprob recompute, after establishing that the packed forward is
the exact per-row computation and the padded batch forward is the side
that mis-positions left-padded rows on long completions.
- Default the packing on (UNSLOTH_GRPO_SEQ_PACKING, disable with 0).
- Verify the packed logprobs against the per-row clean forward (each
row's real tokens alone, reset 0-based positions, no padding), not the
padded batch which is itself wrong for left-padding. Cross-sample
contamination (a backend ignoring packed_seq_lengths) shows up as a
large mismatch and falls back to the padded loop.
- Make the trust decision shape and RoPE aware: re-verify whenever the
packed total length or the longest segment grows past what was
verified, so a later batch crossing a LongRoPE short/long cache
boundary is re-checked instead of trusted blindly.
- Run lm_head only on completion-prediction positions instead of every
packed prompt token, so long-prompt/short-completion batches do not
pay for projecting the whole packed prompt.
- Drop the hard xformers import so the path also runs in
FlashAttention-only environments; the per-row verification guards
correctness regardless of backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: disable entirely on cross-sample mismatch
When the per-row verification fails, distinguish the two failure modes by
magnitude instead of by sequence length:
- A large mismatch (>= 1.5) is the cross-sample contamination signature:
the model's attention does not honor the block-diagonal packed mask
(seen on some MoE / custom-attention models, e.g. qwen2_moe). Disable
packing entirely for the model so later batches do not pay the
verification cost again.
- A moderate mismatch is more likely a length-boundary effect (a LongRoPE
short/long cache switch): keep marking just that length region unsafe so
packing still runs for smaller shapes.
Validated: Qwen1.5-MoE falls back after a single verification (grad and
no-grad ok flags go False, no re-verify on later steps); dense Llama-3.2
and Qwen3 still verify and engage packing.
* GRPO no-grad packing: trim comments to be concise
* GRPO no-grad packing: fix per-row completion boundary for left-padded rows
The completion-target selection used a single global boundary
(col >= L - logits_to_keep). After left-packing, each row's completion
starts at (L - logits_to_keep) - left_pad[row], so for left-padded rows
the first left_pad completion tokens fall below the global boundary and
were dropped, leaving 0 logprobs at real completion positions that the
loss mask keeps. Use the per-row boundary so packed coverage matches
create_completion_attention_mask exactly, and widen the self-verify mask
to the full per-row completion region so it can catch coverage gaps.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: gate verification on real completion rows
Count active rows via create_completion_attention_mask (the same mask the
loss uses) instead of any non-pad token in the packed window. Prompt-only
rows carry prompt-overflow tokens in the window and could otherwise satisfy
the >= 2 verification guard, letting a batch with a single real completion
row cache a trust decision. This matches the gradient path, which already
gates on the completion mask. The same mask is reused for the self-verify
comparison.
* GRPO no-grad packing: gate debug logging on UNSLOTH_ENABLE_LOGGING
Use the shared UNSLOTH_ENABLE_LOGGING global (import_fixes, re-exported by
_utils) instead of a bespoke UNSLOTH_GRPO_SEQ_PACKING_DEBUG env var for the
packing debug prints, matching the rest of the codebase.
* GRPO packing: import UNSLOTH_ENABLE_LOGGING inside the injected logp function
_get_per_token_logps_and_entropies is copied verbatim into the generated GRPO trainer
via inspect.getsource, and that module never imported UNSLOTH_ENABLE_LOGGING, so the
default-on packing verify path raised NameError (and the except handler re-raised it).
Import the flag locally, before the try, so the name is defined in the generated module
too. Drop it from the now-unused module-level import.
* GRPO no-grad packing: harden unsafe-length skip, verify guard, fallback cleanup
Three fixes to the no-grad logp packing path, mirroring the grad path:
- skip the packed forward for known-unsafe lengths by reading unsafe_T and
gating on it before the forward, instead of running the full packed pass and
the result build only to discard them (wastes a pass, can OOM at large T)
- only widen the verified T/seg envelope when >= 2 completion rows actually
exercised cross-sample packing; a < 2 row batch cannot expose leakage, so it
must not extend the trusted shape that later multi-row batches skip verify for
- drop the packed intermediates (hidden/sel/result/ref) before the padded
fallback loop so it does not run with the flattened hidden state still resident
* GRPO no-grad packing: cap the flattened forward at one mini-batch budget
The packed path built a single [1, sum L] forward over every row before any
size check, so a large batch could exceed the memory the padded path bounds
per mini-batch. Gate packing on _pk_T <= _pk_cap (B * seq_len, one padded
mini-batch's token budget); larger batches fall back to the chunked padded
loop.
* GRPO no-grad packing: disable unless unsloth_zoo has the masked-column guard
The packed path leaves masked prompt/pad logprob columns at 0, which only stays
finite if unsloth_zoo grpo_compute_loss zeroes them before exp() (zoo#840). An
older unsloth_zoo without that guard would NaN. Detect the guard once (cached on
the model) via inspect.getsource and gate packing on it, so #6738 is safe with
any unsloth_zoo version and re-enables packing automatically once a guarded zoo
is installed, independent of the pinned lower bound.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: hoist env gates and zoo-guard detection to one-time module checks
Read UNSLOTH_GRPO_SEQ_PACKING and detect the unsloth_zoo masked-column guard once at
import time (module constants plus RL_PRE_ITEMS for the generated trainer cache) instead
of per call, and drop the in-function UNSLOTH_ENABLE_LOGGING import for a module-top one.
The UNSLOTH_GRPO_SEQ_PACKING_VERIFY force-verify debug knob is commented out, kept in
place for hand re-enable; the first-use and envelope-growth self-verify stays active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: cap the flattened forward by the padded chunk rows
B counts chunks at this point, so B * seq_len understated (small runs) or overstated
(large runs) the padded mini-batch token budget; use batch_size * seq_len, the rows the
padded loop actually forwards per chunk.
* GRPO sequence packing: tighten comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
The setup.ps1 unit-tests job intermittently fails on the windows-latest
runner with 'No repository with the name PSGallery was found.' when the
default PowerShell Gallery is not registered, so Set-PSRepository throws
before Pester can be installed. Register the default gallery first when it
is missing, then set its policy and install Pester as before.
* Fix export-time trust_remote_code bypass in FP8/INT8/GGUF-LoRA export
The torchao, compressed-tensors, and LoRA GGUF export paths re-read the merged
checkpoint and used to set trust_remote_code from the checkpoint config's static
auto_map (the torchao path also scanned the staged tokenizer/processor configs).
A model that loads with built-in Transformers classes can carry an auto_map entry,
which skips the load-time remote-code consent scan (that only runs when the load
already requested trust_remote_code) yet flips trust_remote_code on at export,
running unvetted custom code.
Derive the reload trust_remote_code from the approved load decision instead: a new
_loaded_via_remote_code() checks whether the in-memory model / tokenizer was itself
loaded from custom code (its class lives in the transformers_modules package),
walking PEFT / wrapper layers. Built-in-loaded models no longer gain trust from
config metadata; genuine custom-code models (loaded with consent) still reload
correctly. Add CPU-only regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden _loaded_via_remote_code against a None/missing __module__
Read type(node).__module__ via getattr and require a string before startswith,
so a dynamically created or C-extension class with a None module does not raise
during export. Add a regression test.
* Split model and tokenizer trust for the compressed subprocess, walk processor components
The compressed-tensors export collapsed model and tokenizer trust into
one --trust-remote-code flag, so an approved custom tokenizer would have
let an unapproved model's custom code run inside the quantization
subprocess. The subprocess now takes --trust-remote-code-tokenizer for
the processor load and keeps --trust-remote-code for the model loads,
matching the torchao path's separate model_trust / tok_trust.
_loaded_via_remote_code now also walks processor components (tokenizer,
image_processor, feature_extractor, video_processor), so an approved
custom tokenizer held inside a built-in ProcessorMixin keeps its trust
on the export reload instead of failing with trust_remote_code=False.
The walk is a bounded BFS with a seen set so wrapper cycles terminate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix: skip fp16/bf16 validation for full finetuning in RL trainers
When doing full finetuning (FFT) of a bfloat16 model, the fp16/bf16
mismatch validation fires before the corrective logic runs, causing a
misleading error even though the code would properly handle it downstream.
Skip the validation when full_finetuning is active.
Fixes#6731
* Fix: auto-correct fp16/bf16 mismatches for full finetuning before validation
Instead of entirely skipping validation (which could let mismatches
through when mixed_precision_dtype is float32), auto-correct explicit
fp16/bf16 settings that conflict with the model's dtype for FFT. This
way the existing validation still catches real mismatches for non-FFT
cases, and the corrective logic below handles the normalized settings.
Fixes the issue raised in Codex review of PR #6813.
* Guard Windows ROCm torchao override skip
Detect installed ROCm torch directly before applying the torchao override so Windows ROCm environments never install the crashing torchao package even if the earlier ROCm-installed flag is missing.
* Update unsloth/models/rl.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/install_python_stack.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Harden ROCm probe and sync RL precision flags
Tolerate stray stdout noise when probing Windows ROCm torch installs by checking the last non-empty output line, matching the existing torch version probe behavior. Also keep args.fp16 and args.bf16 synchronized with the full-finetuning precision auto-corrections in the RL trainer patch so downstream eval settings see a consistent TrainingArguments state.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add MLX trainer compatibility shims
Patch imported MLXTrainer and MLXTrainingConfig objects to preserve the expected dataclass field ordering and to provide a _train_dataset_for_batches fallback when older trainers or test doubles only expose train_dataset. Also add focused worker tests covering both compatibility paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope PR to Windows ROCm torchao guard
* Restore PR scope to Windows ROCm guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: cover Windows ROCm torchao skip behavior
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Ayushman Paul <ayushman@HP>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Fix external drive custom folder selection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/tests/test_linux_external_media_paths.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep legacy media scan validation strict
* Apply sensitive-dir denylist to legacy folder browser for PR #6799
The legacy /api/models browse endpoint gained the new /run/media mount
roots in its allowlist but not the credential/config guard that scan-folder
registration and the Hub browser already enforce. Filter sensitive names
during enumeration and reject them in _resolve_browse_target so .ssh, .aws,
.config, etc. under allowlisted roots stay unbrowseable, matching the Hub
browser. Add a public contains_sensitive_path_component helper and cover the
legacy resolver with a regression test.
* Trim redundant comments in PR #6799 changes
* Skip sensitive Linux media roots
* Reject sensitive dirs at exact browse roots for PR #6799
Both _resolve_browse_target functions only checked contains_sensitive_path_component
while walking descendant parts, so requesting an allowlisted root itself (empty
relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh,
~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to
the allowlist on upgrade and could then be browsed. Check the resolved target once
before returning in both the legacy and Hub browsers, and cover the root case in
both test suites.
* fix: avoid unused path helper reexports
* fix: import sensitive path helpers directly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Fix TrainingArguments silently disabling unsloth gradient checkpointing
* Cover loaded adapters and preserve explicit None in GC restore
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: flush passthrough stream headers before upstream prefill stalls
* Studio: clean up delayed passthrough send failures
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close passthrough preheader cleanup gaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: retry delayed passthrough overflow truncation
* Studio: close completed passthrough send responses
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: expose full compressed-tensors scheme set in an export formats dropdown
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: multi-select export formats, portable torchao FP8/INT8, GGUF LoRA, source parity
Export page overhaul on top of the formats dropdown:
- Unify merged precision into one sorted multi-select list (16-bit first, then
8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16),
INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live
in a multi-select "More formats" dropdown, so several formats export in one run.
- Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig /
Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM.
FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged
and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and
_unsloth_save_torchao, parallel to the compressed-tensors path.
- Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep
16-bit and portable FP8/INT8. The backend also rejects a compressed request on
non-NVIDIA hardware so it stays authoritative.
- Relax merged export to non-PEFT models so Local Model and Hugging Face sources
get the same 16-bit / compressed / portable options.
- GGUF: send the whole quant list in one call (merge once, quantize many).
- LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype
select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter.
- Thread the new fields through models, routes, orchestrator, and worker; extend
the export tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate export by accelerator with a torch-aware reason; fix export save dir naming
Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel
GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even
with PyTorch installed. Add export_capability() in utils/hardware that reports
export_supported plus a precise reason so the UI stops showing a generic "no GPU":
- pytorch_not_installed: a --no-torch install (even a physical GPU is unusable)
- no_accelerator: PyTorch present but no supported accelerator (bare CPU)
- mlx_unavailable: Apple Silicon where the MLX stack is missing or too old
Expose the fields on /api/system/hardware and /api/system, and guard the mutating
export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the
reason, leaving read-only endpoints usable so the Export page still renders.
Make core/export/export.py import without PyTorch and without a usable accelerator
(the Unsloth import is caught) so the export worker degrades to a clear message
instead of crashing at import.
Frontend: keep /export reachable on chat-only hosts and gray out the method and
format options with the backend reason (Alert plus disabled MethodPicker) instead
of silently redirecting to /chat, so users see why export is unavailable.
Also fix the export save directory producing "model/null" for Local Model and
Hugging Face sources that have no run/checkpoint, naming the folder from the model id.
* CI: validate Studio export capability gating on Linux, Windows and macOS
Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py
on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS,
that hardware.export_capability() reports the right decision and reason
(pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export
backend imports without PyTorch and degrades to a clear message instead of crashing.
Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why"
path a Mac/Windows user without an accelerator sees; a real accelerator export is
validated separately. The job installs only a CPU PyTorch plus the backend import
deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU.
* Studio export: address Codex review (source-aware gating, GGUF LoRA token/MLX/guard)
Frontend (export-page):
- Gate LoRA and quantized-model restrictions on the active source. isAdapter /
isQuantized come from the selected checkpoint; in Local Model / Hugging Face
("model") source mode they were stale, so LoRA stayed wrongly enabled for a
direct base model (backend then rejects "No adapter to export") and a stale
"quantized" flag disabled every method for an unrelated, exportable model. Add
effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use
them in the method-reset effect and the MethodPicker disabled state.
- Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on
MLX), so users no longer pick it, wait through the load, and always fail. Disable
the "GGUF adapter" button on a Mac host and never send loraGguf there.
Backend (core/export/export.py):
- Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a
gated/private base model's config fetch in convert_lora_to_gguf.py is
authenticated; without it the load can succeed but the conversion fails.
- Guard the save_pretrained_gguf capability check with getattr so an older Unsloth
model that lacks the method returns the clean "not supported" message instead of
an AttributeError that surfaces as a generic 500.
* Studio export: address 2nd Codex review (CI index, empty merged, test import)
- studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to
the torch install so torch's transitive deps still resolve; --index-url alone
replaces PyPI with only the CPU wheel index, which does not serve all of them.
- export-page handleStart: reject an empty merged selection (mirrors canExport), so
clicking the panel's Start button with every precision pill deselected no longer
submits mergedSelections: [] and launches an unintended default 16-bit export.
- test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py
as text (like the other ast/string checks) instead of `import unsloth.save`, which
raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth
installed.
* Studio export: make comments succinct across the export changes
* Studio export: use load token for local GGUF LoRA export of gated bases
* Studio export: harden portable torchao path and gate multi-format Hub push
torchao (_unsloth_save_torchao):
- merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted
- narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted
- forward trust_remote_code (from auto_map) to the reload so custom-code models export
Export UI:
- hide portable torchao formats on macOS/MLX (backend rejects quantized export there)
- restrict a Hub merged export to a single format (each writes to the repo root)
* Studio export: torchao tokenizer remote-code + XPU offload, scale GGUF timeout
torchao (_unsloth_save_torchao):
- honor auto_map in the staged tokenizer/processor configs (not just model.config) when
deriving trust_remote_code, so custom-code tokenizers reload after the merge
- offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching
the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy
Export orchestrator:
- scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a
large model does not time out at a flat 3600s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export: show portable torchao formats only on non-NVIDIA (CPU) hosts
Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path.
On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor
auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and
continue hiding them on macOS/MLX.
* Studio export: report all output folders and the exported formats
- Multi-format merged export now collects every sibling output directory (one per selected
precision) instead of only the last; the success banner lists them all.
- Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations),
so the panel says what is being exported rather than just 'Merged Model'.
- Persist the selected formats in the run summary and seed them on mount, so navigating away and
back (or toggling the export method) restores the selection instead of resetting to 16-bit.
* Studio export: list all output formats, add GGUF LoRA target, default Q8_0, auto-select newest checkpoint
- Progress/summary panel now shows a Formats row with the selected merged
formats, and the success banner lists every output folder a multi-format
merged run creates (one line per format) instead of only the last one.
- Merged format selection is seeded from the active run, so navigating away
and back (or switching method cards) no longer resets it to 16-bit.
- GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA
adapter) for adapter checkpoints, reusing the LoRA GGUF export path.
- Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI,
the request model, and the backend defaults; the outtype list is now
Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers.
- When a finetune has no checkpoint selected, auto-select the newest one.
* Studio torchao export: robust reload class + optional VLM import
Two fixes to the portable torchao FP8/INT8 export reload, from review of the
narrowed VLM detection:
- Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs.
With the narrowed is_vlm test they now correctly skip the image-text class,
but fell through to AutoModelForCausalLM and failed to reload after the merge.
Reload them with their own architecture class from the config instead.
- AutoModelForImageTextToText was imported unconditionally at the top of the
torchao path, so on Transformers builds without that class the import aborted
every torchao export (even text-only). Import it lazily only for a VLM, with
the AutoModelForVision2Seq fallback used elsewhere in Unsloth.
* Studio: enable FP8/FP4 compressed export for newer-transformers models
The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed
for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the
quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS.
Run the quantization against a dedicated llm-compressor-main "shadow": a --target
package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered
over the existing torch. It installs --no-deps so torch is never touched (works on any
Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned
off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN.
- transformers_version.py: provision + validate .venv_llmcompressor.
- export.py: route all compressed exports through the shadow when available; else keep
the workspace 0.10.x path and fail fast past its transformers ceiling.
- save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow.
- _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the
RedHatAI and NVIDIA reference quants, and is required by the grouped schemes).
Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and
fp8 on Gemma-4, end to end through Studio.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF LoRA export tests
* Fix export CI expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* inference: add passthrough tool-call healing core (heal_gate, heal_openai_message, StreamToolCallHealer, nudge helpers)
Small GGUF models often emit tool calls as text (<tool_call>{...}</tool_call>,
Gemma <|tool_call>, <function=> XML) instead of structured tool_calls. Studio's
enable-tools loop already heals these, but the client-tool passthrough
(unsloth run --disable-tools, unsloth start agents) relays them verbatim, so
the agent sees prose and the turn dies.
This module is the shared response-side repair layer the passthrough routes
will call: promote parsed text-form calls to structured calls, but only for
function names the client actually declared; coerce arguments through the same
canonical-key healing as the tool loop; never touch the upstream request body
(llama-server KV/slot reuse stays byte-identical). StreamToolCallHealer is the
streaming buffer-and-repair state machine: prose forwards immediately, only a
partial-signal tail or a suspected tool block is held, false alarms flush
verbatim, and a 64 KiB bound caps memory. nudge_should_retry/nudge_messages
support an opt-in single-retry nudge for non-streaming routes (wired later).
Kill-switch: UNSLOTH_DISABLE_TOOL_CALL_HEALING=1. Reuses
core/tool_healing.parse_tool_calls_from_text, strip_tool_call_markup, and
tool_loop_controller.coerce_tool_arguments unchanged.
* inference: heal text-form tool calls on the OpenAI and Responses passthrough
Wire the passthrough healing core into /v1/chat/completions and /v1/responses,
default ON whenever the request declares client tools:
Non-streaming: heal_openai_message runs inside the existing response-mutation
loop; a promoted call flips finish_reason to tool_calls and nulls the content,
and the verbatim-bytes fast path still applies when nothing was healed.
/v1/responses non-streaming inherits this through openai_chat_completions.
Streaming: a StreamToolCallHealer per stream. Ordinary prose relays
byte-for-byte (a fast path keeps upstream bytes when the healer passes a chunk
through whole); once a tool signal appears, content is held, and at the
finish/[DONE] boundary either synthetic delta.tool_calls chunks replace the
markup (finish_reason rewritten to tool_calls, including the synthetic-finish
path) or a false alarm flushes the held text verbatim. Structured upstream
deltas put the healer to sleep after flushing anything held, so grammar-mode
responses stay byte-identical. The Responses stream feeds healed calls through
the same per-call state machinery as structured deltas (indexes live in a
disjoint range so a healed call can never merge into a structured call's
state), and the visible/reasoning split runs first so reasoning text is never
promoted. parallel_tool_calls=false caps healed calls on every path.
The upstream request body is never touched and healing issues no extra
generation, so llama-server slot/KV-cache reuse is unchanged. Opt-out per
request with auto_heal_tool_calls=false (Responses reads it from the
extra-body); requests without tools relay verbatim.
* inference: heal text-form tool calls on the Anthropic /v1/messages passthrough
Streaming: AnthropicPassthroughEmitter.enable_healing(allowed_tools) routes
content deltas through the shared StreamToolCallHealer. A promoted call closes
any open text block (only the safe prose prefix ever streamed into it), opens a
synthetic tool_use block with a fresh toolu_* id, carries one input_json_delta,
and closes; finish() then forces stop_reason to tool_use unless a truncation
(max_tokens) wins. Structured upstream deltas flush anything held and put the
healer to sleep, so grammar-mode responses are untouched, as is every stream
where enable_healing is never called (Studio's own loop, no-tools requests).
disable_parallel_tool_use caps healed calls too.
Non-streaming: the OpenAI message dict is healed BEFORE block building, so the
existing tool_use promotion loop and stop_reason line treat promoted calls
exactly like native ones (finish_reason length still maps to max_tokens). The
legacy tool-XML strip still runs on remaining text, so opted-out requests keep
today's cleanup behavior byte-for-byte.
auto_heal_tool_calls is now a typed field on AnthropicMessagesRequest
(default True, mirroring Chat Completions) and threads into both passthrough
calls. Healing never touches the upstream request body.
* inference: opt-in single-retry tool-call nudge on the non-streaming passthrough
When the model clearly tried to call a tool (a tool signal in the text) but
healing produced nothing usable, re-ask once: the retry body is the original
body plus an assistant turn (the model's own failed text) and a short user
nudge naming the declared tools. The prompt prefix stays byte-identical, so
llama-server reuses the slot's KV cache and only the two-message suffix is
prefilled. The retry replaces the original response only when it actually
yields a promotable or structured call; on any error or still-garbage output
the original response is returned unchanged. Exactly one retry, non-streaming
OpenAI and Anthropic passthroughs only (a stream has already emitted bytes).
OPT-IN per user decision: nudge_tool_calls=true per request (typed on both
ChatCompletionRequest and AnthropicMessagesRequest, lifted from the Responses
extra-body), or UNSLOTH_TOOL_CALL_NUDGE=1 to flip the process default.
auto_heal_tool_calls=false disables healing AND the nudge.
Also align the non-streaming heal on allow_incomplete=True: the response is
final, so a trailing unclosed tool block is a model failure worth repairing,
matching the enable-tools loop's drain semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: never assume the upstream response shape in the nudge helpers
llama-server error bodies can carry message: null (or no choices at all), and
_last_assistant_text / response_has_promotable_calls / nudge_should_retry
called .get() on the message without a dict check, so a malformed upstream
response raised an AttributeError the surrounding except tuples did not catch,
failing the request instead of degrading to 'nothing to heal'. Route the shape
probing through one _first_choice_message helper that returns None for any
non-dict message, and add a parametrized test over the malformed shapes.
* inference: constrain healing by tool_choice, preserve length finish_reason, keep healed event order in Responses streams
Three review findings on the passthrough healer:
- heal_gate now honors the request's tool_choice: "none" disables healing
outright and a forced function narrows the promotion allowlist to that
one function, so healing can never contradict the request's tool-choice
constraint. Wired through the OpenAI chat (stream and non-stream),
Responses, and Anthropic (converted shape) passthroughs.
- The OpenAI non-streaming heal only upgrades finish_reason "stop" to
"tool_calls"; a truncated generation keeps "length" (the healed call
stays attached) matching the streaming and Anthropic paths.
- The Responses stream emits healer events in order instead of collapsing
all text ahead of the healed calls, so text after a healed call no longer
jumps ahead of the function_call item and output indexes are claimed in
the order the model produced them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: all-or-nothing promotion when a response mixes declared and undeclared text-form calls
Promoting a subset used to strip ALL tool markup from the content, which
silently deleted the text of any call naming an undeclared tool. The heal
now declines entirely when any parsed call is unpromotable, so the whole
message relays verbatim (pre-PR behavior) and no bytes are ever lost. In
streaming, a declared call that completed before an undeclared one arrived
is already emitted; the late undeclared markup still flushes as raw text.
The nudge helpers mirror the same contract via a shared predicate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: wrap long lines in the Responses healing tests to the project style
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: span-exact healing, disjoint healed stream indexes, per-call Responses message items, allowlisted nudge acceptance
Four review findings on the passthrough healer:
- parse_tool_calls_from_text gains an optional with_spans return so healing
removes EXACTLY the promoted calls' markup. This supersedes the previous
all-or-nothing rule: declared calls promote and every unpromoted byte
(undeclared calls, unparseable closed blocks, suppressed alternate
formats such as a <function=...> block after a JSON call) relays as text.
The stream healer also processes one block per pass, so text between two
healed calls keeps its document position instead of trailing them.
- The OpenAI chat stream shifts native tool-call delta indexes past any
already-emitted healed calls; clients merge deltas by index, so a healed
call and a later native call can no longer merge into one.
- A healed call in the Responses stream closes the open message item and
trailing text opens a fresh one with a later output index, matching the
native stream shape; response.completed snapshots every message item
with its own text.
- The nudge retry only replaces the original response when the retry's
structured call names a DECLARED tool; a hallucinated undeclared call is
not an improvement.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the heal path folding trailing prose into a closed function call
parse_tool_calls_from_text(allow_incomplete=True) cut a <function=...> body only
at an end-anchored </function>, so a fully closed call followed by trailing prose
(<function=..>..</parameter></function> words) folded </parameter></function> and
the prose into the tool argument and deleted the prose from visible content. The
strict path (allow_incomplete=False) already cut at the real </function> via rfind.
Do the same in both modes: trim the body at the real </function> when present and
end the removal span there, falling back to the end-anchored strip and body_end
only when the call is genuinely truncated. Add a regression test.
* inference: one shared single-call budget for healed and native calls
Codex round 5: the parallel-call caps counted healed and native calls
separately, so a healed text-form call followed by a native structured
delta double-emitted on all three streaming surfaces when the client
disabled parallel calls.
- OpenAI SSE: once a healed call went out with parallel_tool_calls
false, native tool_call deltas are dropped instead of index-shifted.
- Anthropic emitter: native deltas skip block allocation when the
healed-plus-native count already filled the single slot, and healed
emission counts open native states too.
- Responses stream: native deltas that survived the chunk-level cap are
skipped once a healed call claimed the slot.
Also adds a span assertion for the closed-</function> trailing-prose
parse fixed in the previous commit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: relay undeclared text-form calls as text on Anthropic non-streaming
heal_openai_message promotes only declared text-form tool calls and
span-trims just their markup, deliberately leaving every unpromoted byte
(undeclared text-form calls included) in the content to relay as text.
The Anthropic non-streaming builder then ran a blanket _TOOL_XML_RE strip
over that content unconditionally, deleting the undeclared block before
building the text part, so Anthropic clients silently lost a call the
OpenAI non-streaming path preserves. The strip was harmless when healing
was all-or-nothing but became data loss once healing turned span-exact.
Gate the legacy strip on whether healing promoted a call, matching the
OpenAI passthrough and the intent already stated in the comment above.
Add a route-level regression test for the mixed declared+undeclared case.
* inference: require fully declared nudge retries; keep unpromoted Anthropic text
Codex round 6, two findings:
- response_has_promotable_calls accepted a nudge retry when any one
structured call named a declared tool, so a mixed retry (hallucinated
undeclared call plus a declared one) replaced the original and the
caller forwarded the undeclared call, or with parallel_tool_calls
false could keep only it. All structured retry calls must be declared.
- The Anthropic non-streaming builder still ran the legacy _TOOL_XML_RE
strip after span-exact healing, deleting undeclared or malformed call
text that healing deliberately preserved. The legacy strip now runs
only when healing is off (no declared tools, or opted out), matching
the OpenAI passthrough.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: keep unpromoted Anthropic text whenever healing is active
The previous commit skipped the legacy strip only when a call was
actually promoted, so an undeclared-only (or malformed-only) response
was still silently emptied: exactly the dead-turn shape this path
exists to fix, and inconsistent with the OpenAI passthrough, which
relays those bytes verbatim. Gate the strip on healing being active
instead; opt-out and no-tools requests keep the legacy strip.
* Fix schema-aware tool healing for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix passthrough healing ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stream finish ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
* replaced connect with start
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix
* Studio: build the coding-agent command from the selected server
The API keys panel showed a hardcoded `unsloth start claude`. `unsloth start`
defaults to 127.0.0.1:8888 and only mints a key for a loopback server, so a
non-default port or a tunnel/remote base would target the wrong server or fail
to mint. Build the command from the panel base/key (and emit a key for
non-loopback), matching the other snippets in the panel.
* CLI: keep `unsloth connect` as a hidden alias for `unsloth start`
Avoids breaking existing scripts and docs that still call `unsloth connect`.
* Tests: stub _unstarted_cleanup in same-task disconnect test
The test builds _SameTaskStreamingResponse via __new__, so set the attribute
that __call__ now reads.
* Match coding-agent command loopback check to the CLI 127.0.0.0/8 rule (#6613)
* Keep unsloth_cli.commands.connect importable as a deprecated shim (#6613)
* Format the new coding-agents panel strings and import per biome (#6613)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the unsloth connect alias and shim; unsloth start is the only command (#6613)
* Route unsloth connect to unsloth start as a hidden backward-compatible alias (#6613)
* Forward unsloth run model-load flags to unsloth start (gguf-variant, context-length, load-in-4bit, tensor-parallel) (#6613)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Session-scope coding agent config in unsloth start
Configure each agent for the current session instead of writing the Studio endpoint, key, and default model into the user's own config. Codex, OpenCode, OpenClaw, and Hermes get a private config relocated through their config-path env vars (CODEX_HOME, OPENCODE_CONFIG overlay, OPENCLAW_CONFIG_PATH plus OPENCLAW_STATE_DIR, HERMES_HOME). Claude Code suppresses the attribution header for the session via the CLAUDE_CODE_ATTRIBUTION_HEADER env var plus a --settings overlay, with no ~/.claude write. --launch uses an ephemeral temp dir removed after the agent exits; --no-launch uses a stable Unsloth-owned dir and prints the matching export lines.
* Read relocated agent session config in Local Agent Guides CI
The contract crosscheck and the openclaw/hermes patch helpers now read each agent's config from the relocated path printed by unsloth start --no-launch (CODEX_HOME, OPENCODE_CONFIG, OPENCLAW_CONFIG_PATH, HERMES_HOME) instead of fixed home paths. The Claude attribution A/B toggles the header for the session only (shipped-config HIT vs vanilla MISS) instead of editing ~/.claude/settings.json.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip the POSIX-only --no-launch parser test on Windows
test_no_launch_output_is_parseable mirrors the #6547 bash CI parser, which greps export/unset lines and only runs on Linux/macOS runners. On Windows --no-launch prints PowerShell ($env: / Remove-Item), so the export-line assertion does not apply there. Cross-OS staging CI surfaced this.
* Size Claude Code's auto-compact window to the loaded model's context
Claude Code auto-compacts against its native (~600k token) window, so against a smaller local model it overflows the server's context (silent truncation) long before it compacts. Set CLAUDE_CODE_AUTO_COMPACT_WINDOW to the loaded model's real context length (the value codex/openclaw already get via model_context_window / contextWindow). Omitted when the model reports no context length.
* Pin OpenCode/Hermes context window and set 90% compaction across agents
Feed every agent the server-determined sequence length (the value /v1/models reports from runtime_context_length) and a ~90% compaction threshold. OpenCode: a custom-provider model with no limit defaults to context 0, which silently disables auto-compaction, so set limit.context/output and scale the compaction buffer to 10% of the window. Hermes: pin model.context_length (it otherwise falls back to a 256k default when the server's /v1/models omits the field) and set compression.threshold 0.9. Claude: add CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=90 alongside the window. Codex (model_context_window) and OpenClaw (contextWindow) already carried the window and auto-manage off it.
* Add `unsloth start pi` recipe
Pi was the only agent without a built-in recipe, so the agent-guides CI
hand-wrote ~/.pi/agent/models.json. Add a first-class `pi` command mirroring
the others:
- write_pi_config writes the session-scoped OpenAI-compatible provider config
(key in the config, like openclaw/opencode).
- pi() launches `pi --provider unsloth --model <id>` (Pi defaults to the google
provider, so the provider/model are pinned on the command line) with HOME
relocated for the session. Pi has no config-dir env var and resolves ~/.pi off
$HOME, so HOME-scoping keeps the user's ~/.pi untouched.
Migrate the agent-guides CI off the hand-written config onto the
`unsloth start pi --no-launch` path (connection + file-edit), with a crosscheck
for the provider api, so the documented recipe is exercised.
* Harden unsloth start for Windows and WSL agent launches
Address the Codex review on PR 6613:
- write_pi_config now pins the loaded contextWindow and a sane maxTokens so Pi
compacts instead of overflowing a small Studio context (it otherwise assumes
its 128000 default), matching the other agents.
- pi() sets USERPROFILE (and HOMEDRIVE/HOMEPATH when present) alongside HOME on
native Windows, where Node resolves ~/.pi via USERPROFILE rather than HOME, so
the session no longer reads or writes the user's real ~/.pi.
- The WSLENV bridge flags path-valued vars with /p so a Windows npm shim under
/mnt receives translated paths, while scalar vars (the numeric context window)
pass through untranslated. WSLENV is deduped on the bare name.
- _print_env prints the launch command with PowerShell-safe quoting so the inline
--settings JSON survives copy-paste on native Windows --no-launch.
Add tests for the WSLENV path flagging, PowerShell quoting, the Pi context
window, and the Pi USERPROFILE relocation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Set CLAUDE_CODE_NO_FLICKER for the Claude session
A local server streams in bursts, so Claude Code's full-screen TUI redraw
flickers between tokens. Disable it for the session via CLAUDE_CODE_NO_FLICKER,
alongside the other CLAUDE_CODE_* session env knobs.
* Add a normalized --yolo flag routed to each agent's auto-approve mode
It is easy to forget which agent spells "run tools without prompting" which way,
so `unsloth start` now accepts all three spellings as one option (--yolo,
--dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox) and
routes to the agent's own mechanism:
- claude: --dangerously-skip-permissions
- codex: --dangerously-bypass-approvals-and-sandbox
- hermes: --yolo
- pi: --approve (Pi's only approval gate is project trust)
- opencode: a permission allow block in opencode.json (no CLI flag exists)
- openclaw: tools.exec security=full / ask=off / host=gateway (no CLI flag exists)
Because the option is parsed by `unsloth start`, the "wrong" spelling for an
agent still routes correctly instead of leaking through to the agent and erroring.
IS_SANDBOX is deliberately left unset for Claude so its root/sandbox safety gate
still applies. Adds routing, cross-routing, and per-config tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review findings: IPv6 loopback command, pi USERPROFILE under WSL, yolo guard
From a 10-reviewer pass over the PR:
- studio/frontend agent-command.ts: normalize bracketed IPv6 hosts. URL.hostname
returns "[::1]" for http://[::1]:8888, which never matched the "::1" loopback
checks, so the copied command embedded the placeholder API key for a local IPv6
server instead of the bare auto-minting command. Now [::1] is treated as loopback
like the CLI's is_loopback_url, so the command matches the CLI contract.
- pi(): also relocate USERPROFILE (and HOMEDRIVE/HOMEPATH) when running under WSL
against a /mnt Windows shim, not just on native Windows. Windows Node resolves
~/.pi via USERPROFILE, and the WSLENV bridge translates the path, so pi no longer
falls back to the user's real ~/.pi in that case.
- _yolo_command_flags: use .get so a config-based agent (or a typo) yields no flag
instead of a latent KeyError.
Adds tests for the WSL pi USERPROFILE relocation, the yolo unmapped-agent guard,
and that opencode/openclaw --yolo stays config-only (no argv flag).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix round-2 review findings: WSLENV /p upgrade, agent help text
- _merge_wslenv now upgrades a user's pre-existing unflagged WSLENV entry (e.g. a
bare HOME or USERPROFILE) to the path-translated form (HOME/p) instead of leaving
it as-is, so a Windows agent shim under WSL receives the translated session path
rather than the raw Linux path.
- Generalize the `unsloth start` registration help to list all six agents (was only
"Claude Code, Codex").
Adds a test for the WSLENV unflagged-entry upgrade.
* Fix round-3 review findings: complete openclaw --yolo, refresh stale copy
- openclaw --yolo now also writes the host approvals file (exec-approvals.json with
defaults security=full / ask=off / askFallback=full) alongside the tools.exec
config. OpenClaw gates tool execution on both layers (the stricter wins), so the
config alone could still leave it prompting or denying. Mirrors `openclaw
exec-policy preset yolo`. ask=off means nothing is ever prompted, so the runtime
socket block is unnecessary.
- Studio API panel copy: clarify that a local server auto-mints the key while a
remote one embeds it in the command, and add pi to the swap hint.
- Local Agent Guides CI: drop the stale "pi has no start.py recipe" note now that
all six agents are driven via `unsloth start <agent> --no-launch`.
Adds the openclaw approvals-file assertions and a no-yolo openclaw test.
* start: parse claude --version with a regex so a format change does not drop optimization flags
* start: offer to install a missing agent (prompt then run its install command)
* start: auto-start a Studio server for --model when none is running, and stop it on exit
* inference: surface an actionable message when llama-server cannot compile a tool grammar
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review findings: kill the auto-started server tree on Windows; apply the tool-grammar message to the OpenAI passthrough too
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: split --model org/repo:variant so a running session is not evicted
`unsloth start <agent> --model org/repo:QUANT` failed against an already-running
Studio server and, worse, killed whatever model another session had loaded.
/v1/models lists a loaded GGUF under its bare repo id (e.g. unsloth/Qwen3-1.7B-GGUF),
so _resolve_model never matched the `:QUANT`-suffixed request. It then POSTed
/api/inference/load with model_path=org/repo:QUANT, which (a) Hugging Face rejects
("Repo id must use alphanumeric chars, '-', '_' or '.'") and (b) evicts the model the
other session was using, so a second 'unsloth start' in a new tmux/terminal tore down
the first. Re-running the command then attached to the now-empty server, which is why
it 'worked the second time'.
Mirror the org/repo:QUANT -> org/repo + --gguf-variant QUANT shorthand that
'unsloth run' and llama.cpp already accept, splitting it in _connect before we match or
serve. Matching now resolves against the loaded bare repo id (no spurious reload, no
eviction), and any real load uses a valid repo id plus gguf_variant. An explicit
--gguf-variant still wins; local paths and Windows drive letters pass through untouched.
The auto-serve path likewise spawns 'unsloth run --model org/repo --gguf-variant QUANT'.
* start: harden auth-key handling, codex teardown, and CI transcript redaction
Three review findings:
1. CI could leak a live key. agent-guides-drive.sh printed the raw
'unsloth start --no-launch' transcript (which carries export UNSLOTH_API_KEY /
ANTHROPIC_AUTH_TOKEN lines) to the Actions log on both the failure path and the
success path before redact() ran. Add cat_redacted() and use it for those two
prints, so the key is scrubbed on the way to the log while the on-disk file stays
intact for the env parsing that follows.
2. Outages masqueraded as bad keys. _key_accepted caught a broad Exception and
returned False, so a 5xx or timeout while checking a cached key looked like a
rejection: it discarded a good key and minted extra ones (local) or reported 'no
saved key' (remote). Only treat HTTP 401/403 as a rejection; let other errors
propagate so a real outage surfaces.
3. Codex preflight could leave the auto-started server up. _require_gguf_for_codex
runs after _connect may have auto-started Studio but before _run installs its
teardown finally, so a preflight rejection (e.g. a transformers-backend model) left
the server holding the port/GPU until the atexit backstop. Tear it down explicitly
at the point of failure.
Tests: a 5xx on a saved key surfaces without minting; a non-GGUF codex preflight
tears down the auto-served server.
* start: fix IPv6/portless studio URLs, Pi config-dir isolation, and Pi install recipe
Four review findings:
1. Pi ignored the session config when PI_CODING_AGENT_DIR was already set. Pi's
getAgentDir() reads process.env.PI_CODING_AGENT_DIR before falling back to
$HOME/.pi/agent, so a value inherited from the user's shell sent Pi to their real
config and skipped our provider/key (the HOME relocation alone was not enough). Pin
PI_CODING_AGENT_DIR at the session's .pi/agent dir; it is path-valued so the WSL
bridge translates it automatically.
2. Pi install hint dropped Pi's documented --ignore-scripts. Pi's README installs with
'npm install -g --ignore-scripts @earendil-works/pi-coding-agent' and notes it needs
no install scripts, so accepting the prompt now follows that safe recipe.
3. Auto-start ignored a portless UNSLOTH_STUDIO_URL. unsloth run binds to
'parsed.port or 8888', so http://127.0.0.1 launched the child on 8888 but the health
poll (and the returned base) still used port 80, stalling until the startup timeout.
Normalize the base to host:8888 (IPv6-safe) before starting and polling.
4. API-panel command mistook IPv6 loopback for the bare default. The bare 'unsloth
start' only probes 127.0.0.1:8888 on the IPv4 stack, so http://[::1]:8888 must carry
an explicit UNSLOTH_STUDIO_URL. Drop ::1 from the bare-default host set while keeping
it a loopback host (URL emitted, no key needed).
Tests: PI_CODING_AGENT_DIR is set to the session dir; _effective_base normalizes
portless/IPv6 bases; a portless UNSLOTH_STUDIO_URL auto-serves on :8888.
* start: apply fresh-review findings across CLI, CI, and the API-panel command
From a fresh multi-reviewer pass over the merged head plus the latest Codex bot review:
1. Load knobs now always consult the server. _resolve_model matched on model id alone,
so --gguf-variant / --context-length / --no-load-in-4bit / --tensor-parallel were
silently ignored whenever the id was already loaded (asking for UD-Q4_K_XL kept a
Q8_0 serving). With any explicit knob the CLI defers to /api/inference/load, whose
already-loaded dedup answers without reloading when variant and settings match, so a
second session running the same command still attaches without evicting the first.
2. OpenCode --yolo and the session model pin now ride in OPENCODE_CONFIG_CONTENT. A
project's own opencode.json outranks OPENCODE_CONFIG, so a repo config could silently
override the session model and the --yolo permission block; OPENCODE_CONFIG_CONTENT
outranks project config. The API key stays in the private file, never in printed env.
3. The --no-launch recipe's last line is a self-contained one-liner (inline VAR=value
assignments before the command, conflicting vars blanked). People copy just the last
line, and a bare codex/claude there ran against the user's real ~/.codex or Anthropic
credentials with zero isolation, e.g. inheriting a pre-existing damaged ~/.codex
state DB and blaming the recipe. The CI drive script scrubs the key from the one
'invoking:' echo this adds.
4. The auto-serve log is 0600 and the parent handle is closed. It sat world-readable in
the shared tempdir under a predictable name while carrying the minted sk-unsloth-
key from the unsloth run banner.
5. _key_accepted fails with a clean message on outages. Non-auth errors (5xx, network,
timeout) surfaced as a raw traceback; 401/403 still mean a rejected key.
6. _effective_base strips URL paths, and https loopback targets never auto-serve.
http://127.0.0.1:8888/studio polled /studio/api/health (404) and https://127.0.0.1
polled the wrong scheme, both spinning until the 15-minute startup timeout.
7. API-panel command: only literal 127.0.0.1:8888 earns the bare command. localhost can
resolve to ::1, which the bare CLI never probes, so it keeps UNSLOTH_STUDIO_URL.
8. CI artifact sweep covers redacted-configs/ and agent-workdir/, not just logs/.
Tests: 125 CLI tests pass (new coverage for each fix), 156 backend tests pass, ruff
clean. Adds an unsloth connect alias regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: hand Pi a clean screen at launch
Pi paints inline from wherever the cursor sits: its first render assumes a
clean screen instead of clearing or entering the alternate screen itself
(current Pi never emits a clear at startup). Launched under unsloth start,
that left the session starting mid-scroll beneath the connection output.
Clear the screen (click.clear, cross-platform, no-op without a TTY) right
before the Studio banner so Pi opens exactly one line down on a clean
viewport. Launch path only: --no-launch recipes and piped output are never
wiped, and alternate-screen agents are left alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: auto-override hermes' 64K context floor for small model windows
Hermes refuses to initialize when the served model's context window is
under 64,000 tokens, and a second copy of the same check rejects the
compression model mid-session. write_hermes_config previously pinned the
real window, so any small local model (e.g. 40,960) failed at startup
with manual config.yaml instructions.
For windows below the floor the recipe now claims 65,536 in
model.context_length, scales compression.threshold so compaction still
fires at 90% of the real window, and sets
auxiliary.compression.context_length to cover the mid-session check.
Windows at or above the floor keep the exact previous behavior.
* ci: install pi with --ignore-scripts, matching the start.py hint
The pi cell predates the pi recipe in start.py and still installed the
package with lifecycle scripts enabled, so CI stopped exercising the
exact command users are prompted to run. npm_retry now passes extra
flags through, the pi branch mirrors the install hint verbatim, and the
stale no-recipe comment is refreshed.
* ci: fail loudly when a relocation var is missing from connect output
The empty-string guards ran after appending /config.toml or /config.yaml,
so they could never fire: crosscheck_contract silently skipped its
contract checks and patch_hermes_tools died on the root path with a bare
traceback. Check the raw variable first and guide_fail with the real
cause.
* staging: 6613 round 6 (https elision, no-launch home reuse, auto-start key fallback)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* fast_generate: clear error for vLLM-style inputs when fast_inference=False
When fast_inference=False, fast_generate falls back to HuggingFace
generate, and the wrapper already rejects vLLM-only usage (a
sampling_params or lora_request kwarg, or a string prompt). A vLLM prompt
dict ({'prompt':..., 'multi_modal_data':...}) or a SamplingParams passed
positionally slipped through and hit transformers.generate, raising a
cryptic 'SamplingParams object has no attribute update'. Detect both and
raise the same clear 'only supported with fast_inference=True' error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate: also reject positional list of SamplingParams and list of vLLM prompt dicts
Address review feedback: the slow-mode guard missed SamplingParams passed inside a
positional list and a list of {"prompt": ...} dicts, both valid vLLM batched shapes
that leaked into transformers.generate. Fold the checks into small predicates and
extend the GPU-free test (now 7 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test_fast_generate_slow_guard: expose assertions via a test_ function so pytest collects them
The assertions lived in run(), only called from __main__, so pytest reported no tests
collected and CI skipped the coverage. Rename to test_fast_generate_slow_guard; the
standalone script entrypoint still works.
* fast_generate: reject vLLM tokenized/embeds prompt dicts in the slow-mode guard
vLLM also accepts prompt dicts keyed by prompt_token_ids or prompt_embeds, not just
prompt/multi_modal_data. Those slipped past the slow-mode guard and fell through to
HuggingFace generate with a cryptic error. Recognize all vLLM prompt-dict keys and
add a TokensPrompt test case (now 8 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: catch vLLM prompts= keyword form
vLLM's generate names its first argument `prompts`, so a slow-mode call
like fast_generate(prompts="hi") or prompts=[{"prompt": ...}] bypassed the
guard and leaked into HuggingFace generate as an unexpected kwarg. Check
kwargs["prompts"] with the same _is_vllm_prompt predicate and add two test
cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: reject vLLM tokenized prompt kwargs
vLLM's legacy call shape passes tokens as prompt_token_ids= (and prompt_embeds=),
which are not HuggingFace generate arguments. In slow mode these bypassed the
guard and leaked into HF generate as unexpected kwargs. Reject their presence
with the same tokenize-first message and add a test case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: treat prompts= as vLLM-only
prompts is a vLLM keyword, not a HuggingFace generate argument, so any value
passed as prompts= (including a bare token-id list, which _is_vllm_prompt
deliberately ignores for positional HF token ids) is a vLLM-style call. Reject
prompts= / prompt_token_ids= / prompt_embeds= on presence, and keep the
conservative _is_vllm_prompt check only for the positional arg.
* fast_generate slow-mode guard: reject vLLM prompt kwargs on presence
prompts / prompt_token_ids / prompt_embeds are vLLM-only keyword names that
HuggingFace generate does not accept, so a defaulted call like prompts=None
should raise the actionable slow-mode error instead of leaking a None kwarg
into HF generate. Check membership in kwargs rather than a non-None value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* report a complete load once llama-server is healthy
load_progress() derived its fraction purely from the llama-server's VmRSS over the GGUF shard total. With layers offloaded to VRAM (-ngl) the process releases the mmap'd weight pages after upload, so VmRSS sinks back well below the shard total: the fraction climbs toward ~1.0 during mmap, then collapses to a small value (~8%) once the weights are on the GPU. A fraction-driven progress bar therefore restarts and sticks there indefinitely even though the model is loaded and serving, which reads as a hang at "Starting model...".
Once the server is healthy the load is complete by definition, so report
fraction 1.0 (and bytes_loaded == bytes_total) in the ready phase regardless of resident set size. The VmRSS read is factored into _read_rss_bytes() with its original semantics preserved (0 on a missing VmRSS line, None when /proc is unavailable) so it can be unit-tested off Linux.
Fixes#5740
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* stub heavy deps in the load-progress test and guard a valueless VmRSS
Two review fixes:
1. The new test imported core.inference.llama_cpp at module top, which pulls in
loggers/structlog/httpx and fails collection with ModuleNotFoundError in the
lightweight backend test env when the file is run on its own. Stub loggers,
structlog and httpx via sys.modules.setdefault before the import, mirroring
test_llama_cpp_load_progress_matrix.py; setdefault keeps the real modules when installed. Verified the file now collects and passes with only pytest present.
2. Catch IndexError in _read_rss_bytes: a "VmRSS:" line with no value column
would make line.split()[1] raise and crash a load-progress poll. Return None
instead, with a test for the valueless line.
* Hold load-progress high-water mark and explain a never-healthy load (#5740)
load_progress() now holds a per-process VmRSS high-water mark, so the bar
no longer regresses to ~8% when -ngl offloads the weights and frees the
mmap pages mid-load.
A live server that never returns 200 on /health now gets a specific error
(context/VRAM too large, or a local proxy/VPN intercepting the loopback
probe) instead of the generic invalid-GGUF/out-of-memory message.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Hakan Baysal <hakan.baysal@trmix.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* feat: add mlx public trainer api
* test: cover mlx public trainer api
* fix: preserve mlx epoch trainer configs
* fix: pass mlx warmup ratio through config
* fix: align mlx trainer dataset order
* fix: keep mlx chat templates import-light
* fix: infer mlx trainer context length
* fix: mirror cuda mlx context defaults
* fix: align mlx notebook trainer defaults
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep mlx public helpers import-light
* refactor: reuse mlx optimizer normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address mlx review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten mlx training argument parity
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align mlx trainer eos default
* Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template
* Trim redundant docstrings on internal MLX helpers
* MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps
* MLX review round 3: keep chat_templates importable without torch on MLX
* fix: preserve MLX trainer notebook shims
* fix: ignore CUDA tokenizer moves on MLX
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden MLX trainer shims
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: unwrap MLX scheduler enum args
* fix: coerce integral MLX epoch counts
* fix: spoof CUDA compatibility APIs on MLX
* fix: harden MLX notebook compatibility shims
* MLX: add torch.cuda.mem_get_info to the compatibility shim
Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by
is_available), so on MLX it raises without a shim. Return (free, total) bytes
from the MLX device stats, consistent with the other torch.cuda compat helpers,
and add a matching assertion to the compat-API test.
* MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device
Address review on the MLX compatibility shim:
- torch.cuda.mem_get_info() now derives free bytes from current active MLX
memory instead of the peak high-water mark, so a capacity check stays
accurate after a transient spike or a prior run.
- BatchEncoding.to(device=...) passed by keyword no longer forwards a positional
None alongside the keyword (which raised "multiple values for 'device'"), so
non-CUDA keyword moves like .to(device="cpu") delegate correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: accept preserve_dataset_order; stub RL trainers with a clear error
Two fixes so unmigrated notebooks behave predictably on MLX (torch present):
- preserve_dataset_order is a real MLXTrainingConfig field but was missing from
the extra-argument allowlist, so passing it (as a config or trainer kwarg)
could be rejected as unknown on a zoo without the field. Add it to
_MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable.
- GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones
the installed trl exposes to a stub that raises a clear 'not supported on MLX'
error instead of importing the real torch/CUDA trainer and crashing deep
inside it. Only existing trainers are retargeted (no invented attributes),
idempotent across re-imports.
* MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory
Address review on the MLX shims:
- The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers
trl's lazy trainer import and pulls torch -- that can crash import unsloth on a
torch-free MLX install just to check existence. Decide what to stub from
trl.__all__ + already-materialized attrs (vars) instead; never resolve the real
trainer. All trl trainer names are in __all__, so they are still stubbed (even
torch-free), and the probe no longer imports torch.
- torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were
aliased to peak max_memory_reserved. Back them with current active MLX memory so
cleanup / capacity checks see live usage; max_* keep the peak high-water mark.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias
Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to
the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3
(max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig
built without an explicit length silently ran 60 MLX steps instead of TRL's 3
epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the
TRL epoch default only when neither max_steps nor num_train_epochs is given;
explicit lengths pass through untouched, and the native public args class keeps
its MLX default. Epoch mode is supported by the MLX trainer.
* MLX CI: keep the GGUF reload smoke under the job timeout
The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is
CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed
right on the 300s cliff and killed the process. This step is a save/reload
integrity smoke (it only needs a few chars of output), so the token count is
incidental: generate 8 tokens with explicit threads and a small headroom on the
subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS /
_TIMEOUT). Cuts the reload well under the 25 minute job budget.
* MLX: broaden trainer stubs, real peak-memory reset, fix shim tests
Address review on the MLX public API:
- The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments,
but the alias now points at the _MLXSFTConfig subclass that preserves TRL's
epoch default, so the MLX suite failed before testing the shim. Assert
issubclass instead.
- torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept
earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory
with the same core/metal fallback used for the reads.
- The unsupported-trainer stubs were a fixed list, so trainers outside it (a
newer RLOOTrainer) still routed to the real torch trainer. Derive the set from
trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear
MLX message; names come from __all__ so trl is never resolved.
- The non-MLX export smoke skipped only on missing bitsandbytes/triton; other
absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError)
made it fail on CPU hosts. Skip on any ImportError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep MLX notebook compatibility minimal
* MLX CI: force CPU + small context for the GGUF reload smoke
The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a
fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's
Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli
would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context
(-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX /
_N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so
a future hang is diagnosable instead of an opaque TimeoutExpired.
* MLX CI: export the reload-smoke GGUF as q8_0, not bf16
The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny
context and 8 tokens. Root cause is the format, not the flags: the smoke exported
quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's
bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0
(fast_quantized, the exporter default and what users deploy) instead -- llama.cpp
has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in
seconds. The reload stays CPU-only (-ngl 0) with a small context.
* test: clear TRL shim before availability check
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Add --with-llama-cpp-dir flag to install.ps1 and install.sh
Users can now pass --with-llama-cpp-dir /path/to/llama.cpp to the
installer to skip downloading or building llama.cpp and use a local
directory instead. A junction (Windows) or symlink (Linux/macOS) is
created at the canonical install location, bypassing both the prebuilt
download (Phase 3) and source build (Phase 4) steps in setup.ps1/setup.sh.
The path is passed via UNSLOTH_LOCAL_LLAMA_CPP_DIR env var which
setup.ps1 and setup.sh read directly.
Ported from the idea in unslothai/unsloth#4384, reimplemented against
current Studio architecture.
* test: add static wiring test for --with-llama-cpp-dir flag
Cross-checks install.sh, install.ps1, studio/setup.sh and studio/setup.ps1
so the flag's contract (parse -> UNSLOTH_LOCAL_LLAMA_CPP_DIR env var -> link
local dir, skip prebuilt download and source build) can't silently regress.
Wired into studio-backend-ci.yml alongside the other tests/sh installer tests.
* Address review feedback on --with-llama-cpp-dir flag
- setup.ps1: delete an existing junction/symlink via DirectoryInfo.Delete()
instead of a recursive remove, which can traverse the link and wipe the
user's real llama.cpp directory on PowerShell 5.1.
- setup.ps1: short-circuit the build chain when a local dir is linked so CMake
never runs inside the user's checkout when it lacks a Windows-layout binary.
- install.sh / setup.sh: resolve paths with CDPATH= cd -P so a set CDPATH
cannot corrupt the resolved path.
- install.sh: seed _WITH_LLAMA_CPP_DIR from UNSLOTH_LOCAL_LLAMA_CPP_DIR so an
exported env var (piped-install style) is honored instead of being clobbered.
- setup.sh: create the root llama-quantize shim when linking a local source
build so GGUF export's check_llama_cpp() still finds it.
- setup.sh / setup.ps1: drop a stale link before the custom-home ownership
assert so re-runs with the flag stay idempotent.
- test: pin the new linked-dir build short-circuit.
* Harden --with-llama-cpp-dir against Codex/Gemini review findings
- install.sh: error when --with-llama-cpp-dir is the final arg with no path,
matching the existing --package/--python post-loop guards (was a silent
fallback to the normal prebuilt/source install).
- studio/setup.sh: canonicalize LLAMA_CPP_DIR before the self-link no-op
compare. _RESOLVED_LOCAL is fully resolved while LLAMA_CPP_DIR was textual,
so a symlinked $HOME made the guard miss and the rm -rf could wipe the
user's real llama.cpp tree.
- studio/setup.sh: make the llama-quantize shim non-fatal; it writes through
the link into the user's tree, which may be read-only (shared/CI cache),
and under set -e a failed ln aborted an otherwise-good reuse.
- studio/setup.ps1: detect a broken junction via Get-Item -Force instead of
Test-Path so a dangling link from a prior run is removed and mklink can
relink to a new valid directory.
- studio/setup.ps1: use Copy-Item -LiteralPath so a source path containing
[ ] isn't treated as a wildcard in the junction copy fallback.
- tests: update the wiring assertions for the LiteralPath copy and the
canonicalized compare.
* Validate/reuse local llama.cpp tree and guard the in-use case
Addresses the second Codex pass on the --with-llama-cpp-dir flag:
- Validate the linked tree before disabling installs (setup.sh + setup.ps1):
reusing a local dir skips BOTH the prebuilt download and the source build,
so the dir must already contain a runnable llama-server (build/bin on
Linux/macOS, build\bin\Release\llama-server.exe on Windows). Bail out with a
clear message instead of linking an unbuilt/wrong-platform checkout and
leaving Studio with no usable binary.
- Treat a canonical-path target as already linked when it holds a build
(setup.sh + setup.ps1): point the flag at ~/.unsloth/llama.cpp itself and an
existing build is reused (skip prebuilt + source) rather than clobbered by
the staged prebuilt installer (which uses os.replace()/replace). An empty
canonical dir still falls through to the normal in-place install.
- Abort when an in-use llama.cpp can't be removed on Windows (setup.ps1):
Remove-Item -ErrorAction SilentlyContinue can silently leave a locked tree
in place; detect that and stop with the same active-process message + exit 3
the prebuilt path uses, instead of junctioning over a half-present dir.
Left as follow-up (already tracked by the PR author as a non-blocker): the
in-app "Update llama.cpp" updater does not yet recognize a local-link install
as externally managed; that fix belongs in studio/backend/utils/llama_cpp_update.py.
* Accept all backend llama-server layouts in --with-llama-cpp-dir validation
The linked-tree validation only accepted build/bin[/Release]/llama-server, but
LlamaCppBackend._layout_candidates() resolves a root-level llama-server first,
then build/bin, then build/bin/Release on Windows. A `make` build or a flat
release extract (binary at the dir root) was therefore rejected with a hard
installer failure even though Studio would have run it.
Validate the same candidate set the backend uses in both setup scripts, and add
wiring-test assertions so the check can't silently narrow again.
* Treat --with-llama-cpp-dir local links as externally managed
A --with-llama-cpp-dir install junctions/symlinks the canonical llama.cpp dir to
the user's own checkout, but two backend paths still treated it as a Studio-owned
tree:
- The in-app updater (llama_cpp_update) offered and could apply an official
prebuilt over the link, writing through it into the user's checkout (or
failing) and silently dropping the link the flag created.
- Orphan cleanup (LlamaCppBackend._kill_orphaned_servers) resolved the linked
root into its kill allowlist, so a llama-server the user launched from the same
checkout was classified as ours and killed on startup.
Detect the canonical dir being a symlink/junction (reparse point) and treat the
install as unmanaged: get_update_status reports unsupported, start_update refuses
with reason "local_link", and the linked root is left out of the orphan
allowlist. Adds behavioral tests (link vs plain dir, updater refusal, and the
spared-vs-killed orphan control).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add behavioral shell test for --with-llama-cpp-dir linking
The existing tests/sh/test_with_llama_cpp_dir_flag.sh is a static grep of the
scripts. This adds a behavioral test that extracts the real link block from
studio/setup.sh (by content anchors, with a self-validating extraction) and runs
it against hermetic fake dirs, asserting the outcomes that matter:
- an external CMake build links and arms neither the prebuilt download nor the
source build
- a flat / make tree (root-level llama-server, no build/bin) is accepted too
- an unbuilt tree is rejected with a non-zero exit and no link left behind
- relinking over a stale link preserves the target's contents (no data loss)
- pointing at the canonical path is a no-op reuse, not a self-referential link
Symlink-identity checks run only where real symlinks exist (skipped on Windows
git-bash copy-mode); the link/skip/no-data-loss checks run everywhere. Wired into
studio-backend-ci.yml next to the static test.
* Install psutil in backend CI so orphan-cleanup tests run
The new orphan-cleanup tests import psutil for the process scan, but the Backend
CI deps step installed studio.txt plus a fixed extras list that omits it, so the
two tests failed with ModuleNotFoundError. Add psutil to both backend pytest dep
steps (kept in shared shape), and guard the import with pytest.importorskip so a
minimal env without psutil skips these tests instead of erroring.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: Implementation of the Portuguese (Brazil) language and VRAM/RAM monitor.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/hooks/use-gpu-utilization.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/features/settings/components/usage-examples.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/studio/sections/progress-section.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix: resolve automated review feedback on API shape
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review issues for PR #6509: Cpu icon, VRAM percent, system polling
- model-inspector: use the exported CpuIcon (Cpu is not a Hugeicons export)
- app-sidebar: guard the VRAM percent on totalVram to avoid Infinity, and
reset the system poll cache only after each request settles so a slow probe
is reused instead of stacking overlapping requests
- use-gpu-info: populate CPU/RAM on hosts without a GPU
- progress-section: label GPUs by visible_ordinal instead of array index
- hub-page: base the RAM label on systemRamTotalGb
- usage-examples: emit JS sampling and tool options at the top level instead
of nesting them under extra_body (the JS SDK does not unwrap extra_body)
- main: read torch and transformers versions from package metadata instead of
importing the libraries on every system poll, and guard the VRAM math
against null values
- hardware: translate a leftover comment to English
* Harden /api/system: guard psutil.boot_time for PR #6509
Simulating restricted containers and some VMs (where psutil.boot_time can raise)
showed the /api/system endpoint would 500 on the unguarded boot_time call, the
same failure class already handled for cpu_freq, disk_usage, and Process. Wrap
boot_time and return uptime_seconds as null when it is unavailable so the sidebar
monitor degrades gracefully instead of breaking. Widen the uptime_seconds type to
number | null to match.
* Studio: make the sidebar hardware monitor a toggle (default on) for PR #6509
Adds a "Show hardware monitor" switch under Settings > Appearance > Layout,
backed by a localStorage preference (default on), mirroring the existing
useSidebarPin pattern. When turned off, the sidebar hides the VRAM/RAM meters
and useSystemInfo stops the 3s /api/system poll entirely, so no nvidia-smi /
SMI probes run while the monitor is disabled. Adds the en and pt-BR strings.
* Studio: default the sidebar hardware monitor to off (opt-in) for PR #6509
* Studio pt-BR: fix three small translation defects for PR #6509
- learningRateDescription: "5e-5 for CPT" -> "5e-5 para CPT" (leftover English)
- exportScopeRecents: "Recents" -> "Recentes" (untranslated)
- relativeMonthsAgo/relativeYearsAgo: add the missing space ("há {count} meses"/
"há {count} anos") so they no longer render as "há 3meses"
* Studio pt-BR: translate the last 10 fallback keys for PR #6509
Adds the settings.general.storage block (Armazenamento) and the
settings.chat.modelDisclaimer pair, so pt-BR now covers all en keys
(679/679) with no English fallbacks.
* Studio: hide sidebar VRAM row on CPU-only hosts for PR #6509
* Studio: tighten and trim code comments for PR #6509
* fix: UI issue in the stop button dialog box (fine-tuning)
* Studio pt-BR: translate 18 new keys from main merge (password dialog, GGUF export, dataset streaming) for PR #6509
* Rounding to GB
* Fix/adjust System resources tab for PR #6509
* Fix/adjust GPU monitor review items for PR #6509
* Fix/adjust remaining GPU monitor review items for PR #6509
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust MLX resource fallback for PR #6509
* floating window implementation
* resize for floating window
* Fix resource monitor review items
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore frontend optional dependency lock entries
* Make GPU selection tests hermetic
* Fix GPU monitor CI test failures
* Bound MLX GGUF reload smoke
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix MLX GGUF reload smoke exit
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: self-heal a pre-#6483-fix anyio>=4.14 stuck in existing installs
The <4.14 cap in constraints.txt/no-torch-runtime.txt only constrains new
anyio resolutions. An install made before that cap existed can already be
sitting on anyio 4.14+, and since it already satisfies mcp/fastmcp's
anyio>=4.5 floor, every later constrained install skips it as
already-satisfied -- so affected installs never recover and keep hitting
the cancel-scope RuntimeError on every request (#6797, a recurrence of
#6483). Force-reinstall anyio<4.14 whenever a stuck 4.14+ is detected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: also repair anyio on the update fast path
setup.sh's _SKIP_PYTHON_DEPS and setup.ps1's $SkipPythonDeps skip
install_python_stack.py entirely once the installed package version already
matches PyPI latest, so an install stuck on anyio>=4.14 with an otherwise
up-to-date package never reaches the repair added in install_python_stack.py.
Probe anyio on that fast path too and fall through to the full dependency
pass when it's still >=4.14, mirroring the existing ROCm/CPU-torch override
right below it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add customizable RAG embedding model setting and reorganize settings tabs
Chat with files, project sources, and knowledge bases previously always
embedded with unsloth/bge-small-en-v1.5. This adds a Settings option to
pick any Hugging Face embedding model (or local path), with HF search
autocomplete, server-side verification that the repo is actually an
embedding model, and a save anyway escape hatch for offline or local
models. The setting persists in app_settings and applies at runtime to
both the sentence-transformers and llama-server GGUF embedder backends
without a restart.
Also reorganizes the General settings tab: Documents & RAG sits above
Uploads, Helper LLM moved above the danger zone, and Model auto-switch
(OpenAI API) moved to the bottom of the API tab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Support local model paths on the GGUF embedder and normalize default saves
Found by simulation testing of the embedding model setting:
Local paths saved as the embedding model now work on the llama-server
GGUF backend (the default backend on macOS and CPU). A path to a .gguf
file is used directly and a directory is scanned for a variant-matching
non-mmproj .gguf, with a clear error when none exists. Previously a
local path was sent to the HF hub API and failed with a repo lookup
error.
Saving the default model explicitly no longer stores an override, so
is_custom stays false and the UI does not show a reset button for the
default value.
* Address review: stale-vector handling, GGUF derivation, save-time guards
Review follow-ups, each verified by new tests:
Re-uploading a document after an embedding model change now re-indexes
instead of deduping by content hash. Documents record the embedder that
produced their vectors (lazy embedding_model column, NULL legacy rows
keep deduping) and a mismatch replaces the old document.
A vector width change no longer bricks the dense index. ensure_vec
drops and recreates chunks_vec when the dim changes (old vectors are in
a foreign space and only block inserts) and search_dense returns empty
on a width mismatch instead of surfacing a vec0 error, so lexical
search keeps working until documents are re-uploaded.
Saving a local sentence-transformers folder with no .gguf now returns
409 with a clear message when the install embeds via llama-server,
instead of failing at first index. force still saves.
A custom RAG_EMBEDDING_MODEL env without RAG_EMBED_GGUF_REPO now
derives the -GGUF companion repo instead of silently keeping the bge
GGUF on CPU and macOS installs.
The resolved GGUF path is tagged with the repo captured at entry, so a
setting change during a download cannot mark the old model as current.
GGUF repo detection matches gguf as a whole name segment rather than a
substring, hf_token is trimmed before verification, and the settings
combobox drops a redundant state mirror of its controlled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shrink embedding model font to 11px in the input and dropdown
The combobox wrapper applies className to the outer input group, so the
size utility must target the inner input element; the previous text-xs
never reached it and the field rendered at the browser default.
* Show curated unsloth embedding models when the search field is empty
The empty-query listing was the global top-downloads page, which holds
no unsloth mirrors for the unsloth-first float to reorder, so the
dropdown opened on third-party models. Match the model picker: curated
unsloth listing when empty, whole-Hub search once a query is typed.
* Address review: settings resilience and index consistency
Keep the last known embedding model on settings store errors, remove the
re-entrant dim lock in the llama-server backend, accept local GGUF saves
and verify GGUF availability for HF repos on that backend, match local
path embedders exactly in model list filters, drop same-width stale
vectors from dense search, pin the embedder per ingestion job, and only
replace completed documents after the re-index succeeds.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate the GGUF repo derivation tests
* Trim to a single core embedding-model test
* Address review: GGUF repo saves and cache race
Accept a GGUF-named HF repo on the llama-server backend by verifying GGUF
availability instead of the sentence-transformers metadata gate, and guard
the settings cache with a generation counter so a read overlapping a save
cannot repopulate it with the pre-save value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio RAG: fix RTL/Indic PDF corruption and dropped DOCX tables
The RAG parser prefers pymupdf4llm.to_markdown for PDFs, but that rebuilds text from
positioned glyphs and mangles complex-shaping scripts (RTL Arabic/Hebrew come back as
shaped Presentation Forms, Indic matras drop to U+FFFD) and can silently drop most of a
heavy-RTL page. _pdf now compares the Markdown against PyMuPDF's logical-order
get_text() per page and falls back to it when the Markdown looks corrupted (shaped
Presentation Forms or U+FFFD above a small floor/ratio) or holds far fewer letters than
the raw layer. Latin PDFs are unaffected and keep their Markdown tables/headings.
_docx walked document.paragraphs, which excludes table cells, so DOCX tables were
dropped entirely. It now walks body content in document order via iter_inner_content,
emitting each table row as pipe-joined cells (deduped across merged cells); the preview
locator already anchors on pipes.
Adds parser tests for the corruption and incompleteness fallbacks and for DOCX table
extraction. These mirror the chat document-extractor guard raised in the unslothai/
unsloth#5351 review; the RAG parser is a separate module and needed its own fix.
* RAG DOCX: keep empty table cells and collapse in-cell newlines
Skipping empty cells shifted later cells left and broke column alignment across rows;
a cell with internal paragraphs (newlines) also broke the pipe-joined row. Keep every
cell (dropping the row only when all are empty) and normalize each cell with
" ".join(split()) so multi-paragraph cells stay on one row. Adds a test for both.
* RAG DOCX: dedup merged table cells on the <w:tc> element directly
Store the shared <w:tc> lxml element in the seen set instead of its id(); it is
hashable and compares by the underlying node, so it dedups spanned/merged cells the
same way without relying on id(). Adds a merged-cell test.
* RAG DOCX: align merged cells, pad skipped grid columns, flatten nested tables
* RAG DOCX: walk cells in document order so nested tables keep in-cell position
* RAG DOCX: dedup vertically merged cells so a spanning label is indexed once
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add a shared fits-on-device filter to the model selects
The chat model selector gains an Only show models that fit on this
device tick under its filter row, and the Hub page gains a matching
Fits device pill next to the sort menu. Both read one persisted
preference (unsloth_models_fit_on_device_only), so toggling either
applies to both.
The filter reuses the Recommended sort's existing fit math, extracted
into hfModelFitsDevice: size from safetensors metadata, GGUF param
count, or the repo name, against the 0.7 GPU + 0.7 RAM budget, with
unsizable models hidden. In the chat selector it extends the fit
filtering to the Trending and Recent sorts and to search results;
downloaded models stay visible regardless. An unknown device budget
keeps everything. The preference is cleared by Reset all local
preferences like the other picker toggles.
* Move the device-fit toggle into the sort dropdowns
* Tighten sort menu footer spacing and shorten the label
* Align the footer checkbox with the option text
* Make the footer checkbox circular with a smaller tick
* Clear menu highlight when the pointer leaves the options
* Address review: fit filter coverage and sizing
Exempt on-disk models from the Hub fit filter, apply it to the feed
trending rows and curated search results, size safetensors and MLX rows
by the quantized load estimate instead of checkpoint bytes, and replace
the native title hint with the app Tooltip.
* Make the whole device-fit row toggle the filter
* Fix gpt-oss offload_embedding and generate() logits_to_keep on fused models
offload_embedding=True moved embed_tokens to CPU but left the input/output device-shuffling forward hooks commented out ('[TODO] Doesn't seem to work!'), so an eager forward/generate with CUDA input_ids hit the CPU embedding and raised a device-mismatch RuntimeError. Re-implement them in a testable helper _install_offload_embedding_hooks that saves the origin device on the module (the pre-hook returns a new tensor, so a device stashed on the original input is lost) and runs the lookup on the embedding weight's CURRENT device. Reading the weight device at call time (not a hard-coded cpu) also handles a non-quantized (bf16) embedding that a later model.to(...) pulls back onto the GPU, which the hard-coded version broke in the opposite direction.
unsloth_base_fast_generate injected logits_to_keep/num_logits_to_keep whenever an inner submodule forward accepted it, but transformers validates generate kwargs against the top-level prepare_inputs_for_generation (plus forward when it takes kwargs). On fused/PEFT-wrapped gpt-oss this raised 'model_kwargs are not used by the model: [logits_to_keep]'. Only inject when the top level would accept it, mirroring transformers _validate_model_kwargs. Behavior is unchanged for every model that works today.
Adds tests/test_offload_embedding_hooks.py and tests/test_generate_kwarg_gate.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload hooks: store origin device on the tensor, not the shared module
The pre-hook stashed the input device on embed_tokens itself, which races when
concurrent forwards share the module (serving). Ride it on the moved tensor and
read it from the post-hook args instead: stateless and thread-safe.
* Also strip mm_token_type_ids that generate() rejects (Qwen3-VL vision GRPO)
The vision processor (Transformers 5.x path) emits mm_token_type_ids, which
Qwen3-VL's generate() then rejects in _validate_model_kwargs on transformers
4.x, so vision GRPO fails at the first rollout:
ValueError: The following `model_kwargs` are not used by the model:
['mm_token_type_ids']
Unlike logits_to_keep this is an incoming kwarg rather than one we inject, so
drop it in unsloth_base_fast_generate when the top level generate does not
accept it, reusing the same _unsloth_generate_accepts_kwarg gate. Extends the
GPU-free gate test with the accept/reject mm_token_type_ids cases (7/7 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim mm_token_type_ids comment
* Trim comments in gpt-oss offload/logits fix (comment-only)
* gpt-oss offload: return embedding output to the decoder device, not the input's
When offload_embedding moves the embedding to CPU, model.device can become CPU and
inputs then arrive on CPU, so returning the output to the input device left it on CPU
and the CUDA decoder hit a device mismatch. Capture the decoder device before offload
and always return there. This also drops the per-request tensor state (stateless, so
concurrent forwards stay correct).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: refuse offload_embedding for tied word embeddings
Tied models share embed_tokens.weight with lm_head, so offloading the weight
to CPU strands the output projection there (device mismatch at generate) and
saves no VRAM since lm_head still needs it on GPU. Detect the shared weight via
get_output_embeddings and raise NotImplementedError instead of loading into a
crash. Untied models (gpt-oss, Llama-3.1-8B) offload unchanged.
Adds tests/test_offload_tied_guard.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip embedding offload on fast_inference (vLLM)
vLLM manages its own weights, so offload_embedding cannot apply on the
fast_inference path (previously it was silently ignored). Disable it with a
notice, mirroring the WSL and Windows skips.
* Trim offload embedding comments (comment-only)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: track decoder device live so it survives model.to()
The post-hook returned the embedding output to a device captured at load time.
If a model is loaded on CPU then moved with model.to(cuda), that device is
stale and the output lands on the wrong device. Read the decoder device live
from the (untied) output embeddings, keeping the captured device as a fallback.
Adds a stale-fallback regression test.
* Make generate-kwarg-gate cases pytest-collectable
Cases lived in run(), which pytest does not collect, so CI never exercised the
gate. Expose them as test_generate_kwarg_gate; still runnable via __main__.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip a meta (disk-offloaded) lm_head as the return device
A device_map that disk-offloads an untied lm_head leaves its weight on the meta
device until that module's own hook runs, so reading it as the decoder device
would move real hidden states to meta. Skip meta (and a missing weight) and fall
back to the captured device. Adds a regression test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm
The OpenAI-compatible endpoints serve whichever GGUF is loaded and ignore the
request model field, so an OpenAI client that changes model never reloads. Add
an opt-in setting that, when a /v1 request names a downloaded local GGUF
different from the loaded one, loads it before serving by reusing the existing
/load path (its dedup, tensor fallback, and threading apply). Unknown names
still serve the loaded model, so drop-in compatibility is preserved and no
remote download is triggered.
Also add an optional idle auto-unload (TTL keep-warm): a pure-ASGI middleware
tracks in-flight inference requests so a stream is never unloaded mid-response,
and a lifespan loop unloads the model after the configured idle seconds. Both
settings default off and live in the app_settings store, exposed via
GET/PUT /api/settings/openai-auto-switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: variant-aware auto-switch, /v1/responses coverage, keep-warm load stamp
Follow-ups from review of the opt-in OpenAI auto-switch path:
1. Variant-aware dedup. _maybe_auto_switch_model compared only the repo id, so
requesting another quant of the loaded repo (e.g. Q4_K_M loaded, Q8_0 asked)
was served by the old quant. Compare hf_variant too, matching /load dedup.
2. Streaming /v1/responses now calls the auto-switch hook. It went straight into
_responses_stream and only checked is_loaded, so stream=True could serve the
old model or 400. Non-streaming already routed through chat completions; the
hook is idempotent once loaded.
3. resolve_local_gguf tries an exact id match before splitting a trailing
:VARIANT, so local ids that contain a colon (e.g. a Windows path) resolve
instead of being cut at the drive letter.
4. Idle keep-warm stamps activity on a load/swap transition. _last_active was
only refreshed by inference requests, so a model loaded after the server sat
idle past the TTL could be unloaded before its first request.
Tests cover each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the /v1/responses auto-switch test order-independent
The new streaming-responses test passed in isolation but failed under the CI's
randomized collection order with "object has no attribute 'state'": it passed a
bare object() as the request and stubbed only one dispatcher, so an ordering
where the real dispatcher ran hit request.state. Give the request a state and
stub both dispatchers; the test still asserts the hook fires before dispatch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: assert /v1/responses auto-switch wiring on source, not at runtime
The behavioral version executed openai_responses and relied on stubbing its
callees, which a randomized collection order in CI could defeat (the real
dispatcher ran and hit request attributes). Assert on the function source that
the hook precedes both dispatchers instead; the hook's runtime behavior is
already covered by the direct _maybe_auto_switch_model tests.
* Studio: auto-switch on /v1/embeddings, GGUF-only targets, idle-unload race gate
Second-pass review follow-ups on the opt-in auto-switch path:
1. /v1/embeddings now calls the auto-switch hook before the loaded-state check,
matching the other model-bearing OpenAI endpoints (the keep-warm middleware
already treats embeddings as inference).
2. The resolver index is now GGUF-only. The local-model scanners also surface
Transformers/safetensors repos; without a filter, auto-switch could unload
the GGUF and route a request into the non-GGUF loader. _has_local_gguf checks
a direct .gguf, a models-dir folder, and the HF-cache snapshots layout.
3. Idle keep-warm now holds an asyncio gate across the idle check and the
unload, and a request bumps inflight under the same gate, so the loop can no
longer unload in the window between "looks idle" and the kill.
Tests cover each. Broader local-model source parity (LM Studio, Ollama, legacy
caches, custom scan folders) is a follow-up; missing one of those today just
falls through to the loaded model.
* Studio: variant-aware local resolver, count_tokens + audio auto-switch coverage
Third-pass review follow-ups on the opt-in auto-switch path:
1. The resolver is now variant-aware via list_local_gguf_variants. It indexes
only the quants actually on disk, recursing snapshots and quant subdirs such
as the nested per-quant folders, so a requested repo:VARIANT resolves only
when that quant is local and a bare repo resolves to a concrete local quant.
This fixes two gaps: the previous shallow glob rejected nested-variant GGUF
repos, and a request for an uncached quant could send /load down the remote
download path, breaking the local-only contract.
2. /v1/messages/count_tokens now auto-switches like its sibling /v1/messages, so
a count uses the requested model's tokenizer.
3. /api/inference/audio/generate (direct GGUF TTS) is now tracked as in-flight
inference, so the idle loop cannot unload the model mid-generation.
Tests cover each. Two reviewer items are left as follow-ups: indexing the
remaining local sources (LM Studio, Ollama, legacy/default caches, custom scan
folders), which fails safe today by falling through to the loaded model; and
fully serializing concurrent different-model requests, an inherent limit of the
single-slot llama backend that the opt-in feature is not designed around.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make local GGUF resolver fail-safe so a bad model name cannot 500
The auto-switch hook calls resolve_local_gguf without its own guard, and
/v1/completions and /v1/embeddings pass body.get("model") through unchanged.
A non-string model (e.g. {"model": 123}) or any internal scan failure would
then raise out of the resolver and turn a request that would otherwise be
served by the loaded model into a 500, breaking the drop-in compatibility the
feature is built on.
Guard the resolver at its boundary: reject non-string input up front and wrap
the lookup so any failure returns None (fall through to the loaded model).
Add regression tests for both paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-model launch flags for auto-switched GGUF models
* Studio: list switch-eligible GGUFs in /v1/models when auto-switch is on
* Studio: settings UI for OpenAI model auto-switch and idle auto-unload
* Studio: show save error over the disabled-idle hint in auto-switch settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address gemini review (case-insensitive /v1/models retrieve, idle-input empty guard)
* Studio: address codex review (deterministic override args, exclude probe/embedding models from discovery)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep-warm count_tokens, gate idle on auto-switch, drop hidden models
Three hardening fixes to the opt-in auto-switch path surfaced while reviewing
the work that builds on it:
1. count_tokens keep-warm. /v1/messages/count_tokens counts via the loaded
tokenizer and already auto-switches, but the keep-warm middleware did not
track it, so idle auto-unload could free the model mid-count. It is now a
tracked in-flight path.
2. "Off means unchanged" for idle unload. get_auto_unload_idle_seconds now
reports 0 while auto-switch is disabled. Idle unload only makes sense with
auto-switch on (an unloaded model returns only via the next request's swap),
so a stray TTL can no longer trigger a destructive unload while the feature
is off, keeping the disabled state identical to pre-feature behavior.
3. Hidden models are not switch targets. The resolver index now skips what
Studio hides from its own pickers (the llama.cpp validation probe, RAG
embedding weights) via _is_hidden_model, so they can never be auto-switched
to by name.
Tests added for each.
* Studio: bare-id reuse, responses validation order, in-flight tracking
Review follow-ups after folding in the per-model overrides and discovery work:
1. A bare model id (no :VARIANT) is now satisfied by any loaded quant of that
repo. Previously a bare name resolved to the largest local quant, so it could
force a slow reload when a different quant of the same repo was already
serving. An explicit repo:VARIANT request still honors the quant.
2. /v1/responses now runs the auto-switch hook after the empty-input validation
so a request that 400s can no longer trigger a multi-minute model load before
being rejected. The hook still precedes both dispatchers, so streaming
requests switch.
3. The keep-warm middleware now tracks in-flight requests whenever auto-switch
is enabled rather than only when the idle TTL is already positive, so a stream
that starts with the TTL at 0 is still protected if idle-unload is enabled
mid-stream. Off still passes straight through.
Tests added for each.
* Studio: tighten auto-switch code comments
Comment/docstring-only pass over the OpenAI auto-switch feature: collapse
multi-line blocks, drop a comment that restated the gate it sits next to, and
trim verbose docstrings on internal helpers while keeping the load-bearing
rationale (concurrency, API behavior, drop-in compat, gotchas). No logic
change: verified comment-only with the AST/printer signature check.
* Studio: bind auto-switch locks per running loop
Review follow-up. The auto-switch swap lock and the keep-warm unload gate were
module-level asyncio.Lock objects. That is safe under the single uvicorn loop
and on Python 3.10+ (the Lock resolves the running loop lazily on acquire), but
a module-level Lock binds to one loop on pre-3.10, which can raise a loop
mismatch in multi-loop runners. Resolve each lock through a per-loop accessor
backed by a WeakKeyDictionary so every running loop gets its own Lock and stale
loops are collected. No behavior change under the server's single loop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch re-review fixes (body codes, coverage, swap, alias, tracking)
Follow-ups from a second review pass over the opt-in OpenAI auto-switch feature:
1. OFF-state status codes: /v1/completions and /v1/embeddings moved the body
read ahead of the loaded-state check, so a malformed/empty body with no model
loaded returned 500 instead of the prior 503. A shared helper reads the body
defensively (an unparseable/non-dict body yields no model), and the handler
re-reads after the 503 gate to surface the original parse error exactly as
before. OFF behavior is unchanged.
2. Local-model coverage: the resolver index only scanned ./models and the active
HF cache, while the model picker also lists the legacy/default HF caches, LM
Studio dirs, and user scan folders. A request for one of those named models
silently served the loaded model instead. _build_index now scans the same
roots (Ollama's symlink-creating scanner is skipped on the request path), and
resolution is offloaded with asyncio.to_thread so the wider scan never blocks
the event loop.
3. Swap vs in-flight stream: a cross-model swap killed the llama-server while
another client was still streaming from it. The hook now tracks how many
requests are streaming on the loaded model (in-flight minus those still inside
the hook) and returns 409 instead of swapping while one is active. Concurrent
same-model requests never reach this path, so they are unaffected.
4. Idle-unload + alias: after idle-unload freed the model, an unknown/alias name
resolved to nothing and 503'd, though it served the active model before the
TTL. Idle-unload now remembers the freed id and an alias request reloads it
(only an already-local model, so no remote download), cleared once a model is
loaded again.
5. In-flight tracking: the keep-warm middleware tracked in-flight only while the
feature was on, so a stream started while off could be unloaded if idle-unload
was enabled mid-stream. It now tracks on every inference path; counting is
cheap and invisible to clients.
Tests added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove stray async_task_outputs files committed by mistake
* Studio: auto-switch review round 3 (revert swap guard, hardening)
Addressing a third review pass:
- Revert the cross-model swap guard. It counted keep-warm in-flight (which
includes external-provider calls that never touch the local model) and so
could 409 a local swap spuriously, and it still left a same-model request able
to start streaming on the model a concurrent swap was unloading. A correct fix
needs a request-lifetime reader/writer barrier; a partial guard was worse than
the honest single-slot behavior, so concurrent different-model use is back to
being serialized (documented), like llama-swap's single slot.
- Non-string request model (e.g. {"model": 123} on a raw-body endpoint) is now
treated as absent, so it falls through instead of raising in the membership
checks once an idle-unload stash exists.
- Idle-unload now stashes and replays the freed quant: an alias reload restores
the exact (id, variant) that was freed rather than the largest local quant.
- Anthropic /v1/messages validates max_tokens before the auto-switch hook, so a
request that 400s never triggers a model load.
- Keep-warm tracks a pending count for requests waiting on the unload gate, so
the idle loop cannot unload the model out from under a request that is blocked
on the gate but not yet counted as in-flight.
- The idle-unload task is awaited after cancel on shutdown to avoid pending-task
warnings.
- The resolver's HF cache scan is None-safe and logs at debug instead of letting
a bad root abort the whole index build.
- upsert_app_setting_map_entry rolls back explicitly on error.
Tests updated/added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep saved idle-unload seconds when auto-switch is toggled off
* Studio: auto-switch hardening (thread-safe lock maps, body validation)
Defensive fixes from review:
- Guard the per-loop WeakKeyDictionary get-or-create for both the unload gate and
the auto-switch lock with a threading lock, since WeakKeyDictionary mutation is
not thread-safe when two event loops run on different threads.
- Build the resolver index under the cache lock so concurrent callers with an
expired cache don't all run the multi-dir scan at once.
- /v1/completions and /v1/embeddings return a clean 400 for a valid JSON body
that is not an object (e.g. a list), instead of a 500 from body.get(...).
- The keep-warm middleware only tracks POST requests (inference is always POST),
so CORS preflight (OPTIONS) is not counted, and tolerates a None path.
Tests added for the list-body 400 and the non-POST skip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 4 (local-path load, swap guard, idle fixes)
From a 10-reviewer pass:
- HF-cache entries now load by a concrete local path, not the bare repo id. The
resolver records a load_path (the snapshot dir for a models--* cache repo, the
file/dir otherwise) so /load takes the local branch and can never trigger a
download to satisfy a partial cache. The advertised loader_id (repo id) is kept
as the launch-override key. resolve_local_gguf now returns
(load_path, variant, loader_id).
- Re-add a single-slot swap guard: a cross-model swap returns 409 model_switch_busy
while another inference request is active rather than killing its stream (the
caller is excluded from the count), and holds the keep-warm gate across the load
so no new inference starts mid-swap. Concurrent same-model requests never reach
this path. A residual spurious 409 is possible while a concurrent or external-
provider request is active; that is the documented single-slot tradeoff.
- Idle keep-warm tracks (model_identifier, hf_variant): reloading the same repo at
a different quant counts as a fresh model, so it is not unloaded before one TTL.
- Track Studio's own /api/inference/generate/stream so the idle loop can't unload
the model mid-stream on that route.
- A successful manual /load clears the idle-unload reload stash synchronously, not
only on the next idle poll.
Also merged origin/main (the branch had fallen behind, which would have reverted
unrelated files on merge). Tests added/updated for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 5 (concurrency, identity, load gate)
From a 10-reviewer pass (9 request-changes, 1 approve):
- Concurrent same-target requests load once instead of each returning 409. The
count-based busy guard could not tell "another request wants the same model"
(safe, load once) from "another request is using the loaded model" (refuse).
Track in-flight auto-switch requests per (target, variant) and subtract
same-target waiters from the busy count; a cross-model swap still 409s while a
genuinely different request is active.
- Fix the identity confusion introduced when round 4 began loading by concrete
local path: the backend identifier became a filesystem path. Record the
advertised repo id on the backend after an auto-switch load and use it so
(a) a model loaded manually by repo id is recognized as already serving
(no spurious reswap/409), (b) /v1/models reports the repo id, never a host
path or a duplicate, and (c) the idle-unload stash keeps the override keyed by
the repo id, so an alias reload after TTL keeps the user's saved launch flags.
- Gate the manual /load route with the keep-warm lifecycle gate so idle
auto-unload can't unload a model mid-load. load_model now wraps _load_model_impl
in the gate; auto-switch calls _load_model_impl directly since it already holds
the gate.
- Restore default-off parity on Anthropic /v1/messages: an unloaded backend with
auto-switch disabled 503s before the max_tokens 400 check, as it did pre-feature.
When the feature is on, request-shape validation still runs before any load.
Tests added for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 6 (concurrency ordering, leaks, unload gate)
From a second 10-reviewer pass (8 request-changes, 2 approve):
- Same-target concurrency: register a waiter by the raw requested model before
the (slow) resolve, and exclude pending requests from the swap busy count. The
middleware counts a concurrent same-model request as in-flight before it
resolves and joins the resolved-target waiter map, so the prior fix could still
409 it. The guard now subtracts max(same resolved-target, same raw-request)
waiters and ignores pending (a pending request is blocked in the middleware,
not generating, so a swap can't interrupt it).
- External-provider requests no longer block a local swap. The keep-warm
middleware counts every inference-path POST, but external-provider chat returns
before the auto-switch hook and never touches the local GGUF. The chat handler
now untracks itself before proxying, so its in-flight stream can't trip
model_switch_busy on a concurrent local auto-switch. The middleware skips its
own end-decrement for an untracked request.
- Manual /unload is gated like load and idle-unload: it holds the lifecycle gate
and returns 409 rather than tearing down llama-server while an inference request
is in flight.
- Response model id no longer leaks the load path. /v1/models already advertised
the repo id; chat, completions, embeddings, Anthropic messages, and audio
response bodies now use the same _llama_public_model_id helper instead of the
concrete on-disk model_identifier.
- Chat completions validates the non-system-message requirement before the
auto-switch hook (as /responses and /messages already do), so an invalid
request can't swap the resident model before returning 400.
Tests added for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: auto-switch review round 7 (teardown policy, Unsloth-active swap, training)
From a third 10-reviewer pass (9 request-changes, 1 approve), all on the same
asymmetric-teardown theme. Resolved per the intended policy that only automatic
paths defer to an active stream; deliberate user actions stay interrupting:
- Revert the manual /unload in-flight guard added last round. A manual /load or
/unload is a deliberate action and tears down immediately, as before; only the
automatic idle-unload loop and auto-switch defer to an active request. This
removes the asymmetry the reviewers flagged (manual /load, the /unload Unsloth
branch, and the opposite-backend swaps inside _load_model_impl) by not
extending the guard to deliberate paths, rather than spreading it.
- Auto-switch now refuses a swap whenever another inference request is in flight,
not only when a GGUF is already loaded. _load_model_impl also unloads an active
Unsloth/transformers backend before loading a GGUF, so the busy guard must cover
that case too; otherwise an Unsloth stream could be killed by an auto-switch.
- Refuse API-initiated training while inference is active. When Studio is driven
as an inference API (sk-unsloth key auth), POST /api/training/start returns 409
if a request is in flight, since training frees VRAM by unloading the chat
model and would kill the stream. The Studio UI (session auth) still starts
training and coexists/frees VRAM as before. A mixed UI+API session is not yet
special-cased. Adds auth.authentication.authenticated_via_api_key.
Tests added/updated for each; full backend suite diff vs baseline is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add UNSLOTH_MODEL_IDLE_TTL env override for idle-unload
Borrowed from PR 6517: a startup env var that sets the idle-unload TTL without
the settings UI. Unlike the stored setting (gated on auto-switch), the env value
is a standalone default that enables idle-unload even with auto-switch off, for
headless/container deploys. An explicit UI/API value still overrides it and stays
gated. The settings GET reflects the env default when nothing is stored.
* Studio: auto-switch fixes from review (paths, embeddings input, env idle reload)
- /v1/models advertises a client-facing alias instead of a filesystem path:
the ./models and LM Studio scanners report the on-disk path as the model id,
so the index now prefers model_id/display_name as the advertised/override id
and keeps the concrete path internal as load_path, still resolvable by path.
- /v1/embeddings validates input before auto-switch: a request with a model but
no input now 400s before the hook (like chat/responses/messages), so an
invalid embeddings request cannot unload or swap the resident model.
- Standalone UNSLOTH_MODEL_IDLE_TTL reloads the freed model: the hook now runs
when auto-switch or idle-unload is active, and with auto-switch off it skips
the resolver and only restores the idle-unloaded model, so the first idle
timeout no longer leaves later /v1 requests with nothing loaded.
- Do not resurrect a stale GGUF over an active Unsloth model: the reload-stash
path bails when a non-GGUF backend is loaded, so an unknown /v1 name cannot
tear down a live Transformers/Unsloth model.
- Defensive HF cache scan: each cache root's resolve/dedup is wrapped so a
missing or malformed root skips that root rather than aborting the index.
- Single-model retrieve checks the id is a string before lowercasing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix automatic-load asymmetry, audio reload, preview, idle timer
The standalone UNSLOTH_MODEL_IDLE_TTL reload is a second automatic-load
trigger, but several validate-before-switch guards and reload hooks only
checked the auto-switch toggle. Add a shared _automatic_model_load_may_run()
(auto-switch on, or idle TTL > 0) and route every guard through it.
- /v1/completions validates prompt before any automatic load (it was the one
model-bearing route with no pre-check).
- /v1/chat/completions and /v1/embeddings pre-checks gate on the shared
predicate so a standalone idle TTL cannot reload then reject.
- /v1/messages no longer 503s before the reload hook can restore an idle-freed
model when auto-switch is off.
- Raw completions/embeddings with no model field pass a non-empty sentinel so
the idle-stash reload runs, restoring the legacy "omit model, use loaded" path.
- /api/inference/audio/generate gains the reload hook (after message validation)
so an idle-freed audio GGUF is restored.
- Public preview opts out of auto-switch via a request-scope flag, so a caller's
model field cannot swap away from the pinned checkpoint; preview chat streams
are now matched by _is_inference_path so idle-unload cannot kill them.
- Keep-warm no longer stamps activity on request start, and external-provider
untracking decrements without restamping, so periodic external traffic can no
longer keep the local GGUF warm forever.
Merges origin/main (the branch had fallen behind, which also brought in the
preview route the review flagged).
* Studio: surface model auto-switch in the API tab and demo it in examples
The OpenAI auto-switch toggle previously lived only in Settings -> General.
Add the same toggle to the API tab's usage-examples panel (it shares the
settings cache), and make the examples reflect it: when on, the Python
examples append a second call naming a different downloaded GGUF (so the
model field visibly selects which model serves), and the curl examples gain
a one-line note. Reuses the existing settings API client and i18n keys.
* Studio: harden OpenAI auto-switch reload-only path and Anthropic tool validation
- Omitted-model raw-body requests pass a reload-only sentinel so the idle-stash
reload still restores an idle-freed model, but the resolver never matches a
downloaded GGUF literally named "default".
- Reject malformed Anthropic client tools before _maybe_auto_switch_model so an
invalid request can no longer evict the loaded model.
* Studio: extend auto-switch reload-only and tool validation to schema endpoints
- Schema-backed endpoints (chat completions, responses, count_tokens, messages,
audio) defaulted an omitted model to "default" and passed it to the switch
hook, so a downloaded GGUF named "default" could be swapped to. Route the hook
through a helper that switches only on an explicitly set model, else reload-only.
- Propagate the explicit-set status when building the chat request from a
Responses request, so the non-streaming chat re-check stays reload-only too.
- Validate Responses function tools before the switch hook so a malformed tool
returns 400 without evicting the loaded model.
* Studio: serialize auto-switch swaps across event loops with a process-wide gate
The auto-switch lock is a per-event-loop asyncio.Lock, so two /v1 swaps on
different loops in one process could both pass it and race the single model slot
(the backend and _load_model_impl are process-wide). Add a process-wide
threading gate around the swap, acquired off the loop so a cross-loop wait never
blocks it, layered with the existing per-loop lock. Add a cross-loop test that
fails without the gate (two slow loads overlap) and passes with it.
* Studio: make the auto-switch swap gate wait cancellation-safe
_acquire_swap_gate awaited asyncio.to_thread(lock.acquire) when another loop held
the process-wide gate. to_thread cancellation doesn't stop the worker thread, so a
/v1 request cancelled mid-wait (client disconnect during a cross-loop swap) would
have its thread acquire the gate after the fact, while the finally that releases it
never runs -- permanently deadlocking later auto-switch swaps.
Poll a non-blocking acquire off a short asyncio.sleep instead: it still keeps the
wait off the loop and serializes across loops, but a cancel now lands during the
sleep, when the gate is not held, so nothing leaks. Add a test that deadlocks the
to_thread variant (it times out) and passes with the poll.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: validate modality and tool-confirmation before auto-switch
Two more request shapes could load a named GGUF and only then 400, evicting the
resident model:
- An image request naming a different text-only GGUF. The switch hook now takes
require_vision and rejects a swap to a non-vision target before loading it; a
GGUF's vision capability is its companion mmproj, knowable without a load, and
matches the post-load guard. Only the resolver branch is checked, never the
reload-stash restore.
- confirm_tool_calls=true with stream=false and local tools. /v1/chat/completions
now rejects that shape before the hook, mirroring the local tool path's
bypass_permissions exemption and intent signal.
The vision probe threads the ambient HF token to keep the capability-probe
invariant. Reload-only and idle-reload paths are unaffected.
* Studio: extend validate-before-switch and make the lifecycle gate process-wide
- /v1/messages/count_tokens now rejects malformed client tools before the switch
hook, like /messages (shared _validate_anthropic_client_tools helper), so a
count request can't evict the loaded model.
- /v1/chat/completions rejects a malformed tool_choice forcing object (a
{"type":"function","function":{}} with no name) before the switch hook.
- The inference lifecycle gate that blocks new inference during a swap is now
process-wide (a poll-acquired threading lock, cancellation-safe), not a
per-loop asyncio lock, so a request on another event loop can't start inference
while a swap tears the single backend down.
- Usage examples no longer hard-code a switch-demo repo most users lack; the
model is an explicit placeholder the user replaces.
* Studio: extend the auto-switch modality guard to /v1/responses and /v1/messages
The pre-load vision check that guards /v1/chat/completions now also runs on
/v1/responses and /v1/messages, so an image request naming a text-only GGUF is
rejected before the swap and never evicts the resident vision model. Run the
vision capability probe off the event loop. Make the /v1/models retrieve
loaded fast-path case-insensitive, and never advertise a host path from the
resolver. Remove the dead list_switch_eligible_ids helper, superseded by the
/v1/models catalog.
* Studio: filter /v1/models to GGUF, per-loop catalog lock, reject system-only Responses
Address review findings on the auto-switch path:
- /v1/models advertises only GGUF models the API can actually switch to; a
safetensors/LoRA entry would be selectable but never loadable via llama.cpp.
- The /v1/models catalog cache uses a per-loop lock (like the auto-switch path)
so a second event loop awaiting it can't hang in a multi-loop process.
- /v1/responses rejects system/developer-only input before the switch, mirroring
chat, so an invalid request can't evict the resident model.
- _build_index guards each scan source on its own so one bad root drops only
that source; the vision probe logs a real detection failure instead of
swallowing it.
* Studio: list cached GGUFs in /v1/models by inspecting files, not model_format
The HF-cache scanner leaves model_format unset for GGUF snapshots, so the
previous model_format == "gguf" filter dropped every downloaded HF-cache GGUF
from /v1/models and the retrieve fallback. Decide GGUF-ness from the on-disk
files via the resolver (info_has_local_gguf) instead, run off the event loop, so
the catalog advertises exactly what /v1 can serve.
* Studio: fix /v1/messages/count_tokens route binding plus auto-switch review fixes
The @router.post decorator for /messages/count_tokens had been separated from
anthropic_count_tokens by the _validate_anthropic_client_tools helper, so the
route bound to the validator and dropped its auth dependency. Move the decorator
back onto the handler. Add route-binding tests asserting each /v1 endpoint maps
to its handler with the auth dependency, so a decorator/handler split is caught
at the route level (the direct-call tests missed it).
Also from review:
- update_openai_auto_switch writes both settings keys in one transaction so a PUT
can't leave one updated and the other stale (drop the now-unused single setters).
- max_seq_length override rejects 0 at the boundary (ge=1) instead of accepting
then silently dropping it.
- Document that embeddings auto-switch is best-effort: GGUF pooling has no cheap
pre-load probe like vision's mmproj, so a guard would false-reject GGUF embedders.
- Add a positive idle-unload test (loop frees the model and stashes it for reload).
* Studio: validate Responses tool_choice + Anthropic mixed tools before switch, filter Ollama from catalog
More auto-switch review findings:
- /v1/responses rejects a forcing-function tool_choice with no name before the
switch, mirroring chat, so a malformed request can't evict the resident model.
- /v1/messages rejects mixing Anthropic server tools with custom client tools
before the switch (the check depends only on the payload, so it moves up cleanly).
- /v1/models no longer advertises Ollama-link models: info_has_local_gguf excludes
.studio_links / ollama_links entries, which the resolver skips and can't switch
to, so an advertised id never silently falls through.
* Studio: guard chat audio input before switch; surface env-backed idle unload in settings UI
A chat request carrying audio_base64 rides the same companion mmproj
projector as a vision request, so a text-only target cannot serve it
either. Flag require_vision for audio input as well so the multimodal
probe runs before the switch and a rejected request never evicts the
working model. Generalize the reject message to cover image and audio.
The settings response now reports idle_unload_active (effective TTL > 0)
so the UI can distinguish idle-unload that is active via the
UNSLOTH_MODEL_IDLE_TTL env var from the case where it needs the toggle
enabled.
* Studio: harden auto-switch eviction guards (count_tokens vision, TTS reload-only, mmproj/stash)
Four eviction/correctness fixes on the opt-in /v1 auto-switch path:
- /v1/messages/count_tokens now carries the same require_vision guard as
/messages, so an image count naming a text-only GGUF can't evict a loaded
vision model for a swap that can't serve the request.
- /audio/generate is now reload-only. A local GGUF's audio-input capability
is not a cheap pre-load probe (the companion mmproj signal can't tell an
audio projector from a vision one, and codec TTS ships no projector), so
resolving the client model could load a text/vision-only target and evict
the working audio model before the audio check fails. Only the idle-stash
restore runs here; switching TTS models is an explicit /load.
- The resolver no longer treats a standalone mmproj .gguf as a servable
model. _scan_models_dir's standalone-file pass does not filter mmproj the
way its directory scan does, so /v1/models could advertise a projector and
a switch could load it over the real weights.
- A non-GGUF (Transformers/Unsloth) load and a deliberate /unload now clear
the idle reload stash, so a manual load/unload is never superseded by a
stale idle-freed GGUF that the next /v1 request resurrects.
* Studio: report advertised repo id consistently after an auto-switch
Two model-id reporting fixes so an auto-switched cached HF GGUF is named by
its repo id everywhere, not its snapshot path:
- Streamed /v1/responses envelopes now derive the model id from
_llama_public_model_id (which prefers _openai_advertised_id) instead of the
raw model_identifier. After an auto-switch the identifier is the snapshot
path while the repo id lives in _openai_advertised_id, so the stream used to
report a snapshot basename while /v1/models, chat completions, and
non-streaming Responses all reported the repo id.
- When an advertised alias already resolves to the loaded model (a model
loaded by local path, requested by its repo or LM Studio id), the
already-serving early return now records the alias as the advertised id, so
/v1/models and responses report the alias and mark it loaded instead of the
path-derived basename. Resolver branch only; safe lock-free because an
in-flight request blocks any concurrent swap via the single-slot busy guard.
* Studio: validate request shapes before auto-switch (prompt/input/audio/mcp confirm)
Four more validate-before-switch guards so a deterministic client error never
evicts the resident model on the opt-in /v1 auto-switch path:
- /v1/completions rejects an object/number prompt (only a string or array is
valid) before the switch, instead of loading the named GGUF and letting
llama-server reject the shape afterward.
- /v1/embeddings rejects an object/number input the same way.
- Chat rejects an oversized audio_base64 upload (413) before the switch. The
size cap is a cheap, target-independent length check; the decode itself
stays post-switch to avoid decoding a valid upload twice.
- The chat confirm-without-stream pre-switch guard now mirrors the tool loop's
actual enablement: _effective_enable_tools (honoring a CLI --enable-tools
policy) and mcp_enabled (which opens the tool loop on its own but defers to a
CLI --disable-tools policy). Previously a confirm+no-stream request with only
mcp_enabled slipped past and 400'd after the swap.
* Studio: fix model-id retrieval, streaming n>1, resolver cache TTL, keep-warm auth
Four fixes from review:
- GET /v1/models/{id} legacy raw-path fallback now maps the raw identifier to
the same public id its /v1/models entry uses. After an auto-switch load the
identifier is the snapshot path while the entry is keyed by the advertised
repo id, so a client that cached the old absolute path no longer 404s on a
model that is in fact loaded.
- stream=true with n>1 is now rejected before the switch. Only the
non-streaming GGUF path returns multiple choices, so streaming n>1 is invalid
on every local serving path; both fields are known pre-switch, so it must not
load model B only to 400 and evict model A. Non-streaming n>1 stays
post-switch where the serving path decides.
- The resolver index cache is stamped after _build_index, not with the pre-scan
timestamp. On installs with enough local models for the multi-root scan to
exceed the 5s TTL, the cache was stored already expired and every request
rebuilt it.
- The keep-warm middleware no longer stamps model activity for 401/403
responses. It runs before FastAPI auth, so unauthenticated probes used to
refresh the idle timer without touching llama.cpp; they now decrement the
in-flight count without keeping the model warm.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Extracted and narrowed from unslothai/unsloth#6543 by @TheJagStudio.
This keeps the startup/banner and text file encoding hardening separate from the already-merged Python code-exec UTF-8 fix in #6548.
Co-authored-by: Jagrat Patel <81472856+TheJagStudio@users.noreply.github.com>
* Pin llm-compressor auto-install to a vetted version range
install_llm_compressor() auto-installs llm-compressor on first use of an FP8/FP4
compressed export when it is not already present. The install command used the bare
package name, so pip resolved to whatever the configured index served; a compromised,
dependency-confused, or inflated-version ("999.0.0") release could then run under the
Unsloth process at install and import time.
Bound the automatic install to a vetted range
(_LLM_COMPRESSOR_SPEC = "llmcompressor>=0.8.0,<0.13"), which the oneshot /
QuantizationModifier API this uses supports, so pip can no longer jump to an arbitrary
future or inflated version. An already-installed newer llm-compressor is still used
as-is (the import short-circuits), so this only constrains the auto-install, never a
user's own install.
Add UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL=1 to forbid the automatic install
entirely and require a manual, vetted install, for locked-down or air-gapped
environments. Update the manual-install hints to the pinned spec.
Add tests/saving/test_llm_compressor_install_pin.py: static (ast) guards that the spec
stays a bounded pin, that the install command never passes an unpinned llmcompressor
literal, and that the opt-out env gate is evaluated before any install runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Loosen llm-compressor auto-install ceiling to <1.0 so new models still export
The earlier <0.13 ceiling was too tight: brand-new architectures (for example
Qwen3_5ForConditionalGeneration / qwen3_5, gemma-4 MoE) can require a newer
llm-compressor, and _unsloth_save_compressed_tensors already fails with
"requires a newer llm-compressor" when a scheme is unavailable. Capping the
auto-install at 0.12 would block getting that newer release and break compressed
export for new models.
Widen to llmcompressor>=0.8.0,<1.0. pip still auto-installs the latest 0.x
(where new-architecture support lands), while the <1.0 ceiling continues to block
a jump to an inflated-version ("999.0.0") or 1.0+ dependency-confusion release.
An already-installed newer llm-compressor is still used as-is (the import
short-circuits), and UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL still forbids the
automatic install entirely for locked-down environments.
* Lower llm-compressor floor to 0.6.0 so supported old torch still resolves
The >=0.8.0 floor conflicts with the torch this install pins in its constraints
file. Unsloth supports torch>=2.4, but llm-compressor 0.7.0+ require torch>=2.7
(0.10+ need >=2.9, 0.12+ need >=2.10). On a supported torch 2.4-2.6 box pip then
has no candidate in [0.8.0, 1.0) and FP8/FP4 export fails before quantization.
Lower the floor to 0.6.0 (its metadata only needs torch>=1.7), which never
conflicts with any supported torch. pip still prefers the newest compatible
release, so modern torch continues to get the latest 0.x (0.12.0). The <1.0
ceiling that blocks an inflated-version supply-chain jump is unchanged.
Add a regression test asserting the floor stays <= 0.6.0.
* Cap llm-compressor auto-install ceiling to a vetted minor (<0.13)
A bare <1.0 ceiling still admits any 0.x, so an inflated "0.999.0" served by a
compromised or misconfigured index would win pip's highest-version selection --
the same dependency-confusion this pin is meant to block. Cap the ceiling to the
current vetted minor (<0.13) so that jump is blocked; bump it deliberately, after
vetting, when a newer llm-compressor is needed (e.g. for a brand-new architecture
scheme). Current new models are unaffected: 0.12.0 is < 0.13 and supports them.
The 0.6.0 floor (torch>=1.7 compatible) is unchanged, so resolution still works
across Unsloth's whole supported torch range (2.4 -> 0.6.0 ... 2.12 -> 0.12.0).
Add a regression test asserting the ceiling admits the current vetted release but
blocks an inflated 0.x and the next major.
* Trim comments in the llm-compressor pin (comment-only, no code change)
* Cap llm-compressor auto-install to the exact vetted patch (<=0.12.0)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* scan_packages: key baseline on matched-code hash
The baseline matched on (package, package-relative file, check), which
excluded the matched code, so a future finding of the same check in the
same file was suppressed regardless of what the code did. A malicious
future version of an already-baselined package could place a payload in
the same file under the same check and pass the enforcing gate.
Key the baseline on a hash of the matched code too. The hash is over the
deduped, sorted set of matched spans with L<NN>: line markers stripped, so
version bumps, line shifts and match reordering stay stable while new or
changed flagged code reopens the finding. Version is left out of the key so
routine dependency bumps do not reopen every entry. The hash is capped and
recomputable from the stored evidence.
Regenerate scan_packages_baseline.json against the current dependency set;
the hf-stack, studio and extras scan shards pass enforcing (no active
CRITICAL or HIGH).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: refresh baseline for newer unsloth-zoo release
A newer unsloth-zoo published after the first regenerate added
tests/test_mlx_save_export_regressions.py, a benign test fixture
(temporary_location="/tmp/ignored") that trips the /tmp dropper check.
Regenerate the hf-stack shard against the current set so the entry is
allowlisted; studio and extras are unchanged.
* scan_packages: harden baseline loading against malformed JSON
Guard against a non-dict top-level baseline and non-dict entries so a
corrupt or hand-edited allowlist warns and fails closed instead of
crashing with AttributeError, and treat an explicit evidence: null as
empty.
* scan_packages: hash the full match set, keep indentation, strip only the marker
Address the evidence-hash review feedback:
- Capture every matching line, not the first three, so a payload appended
after existing matches in a baselined file and check reopens the finding
instead of riding the sample.
- Preserve leading indentation so a flagged line moved out of a guarded block
reads as changed.
- Strip only each span's prefix up to the first L<NN>: marker, so an L<NN>:
inside the matched code is kept and a change to it reopens the finding.
Evidence and its hash are stored in full and stay recomputable from the stored
field. Regenerate the baseline; hf-stack, studio and extras pass enforcing with
no active CRITICAL or HIGH.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind baseline evidence to full matched code
Address review feedback on the evidence-hash baseline key:
- Split evidence only on real span delimiters (" | " before an L<NN>:
marker, or a newline), so a bitwise-or or union type in matched code
is no longer split apart into separate spans.
- Record matched lines in full (drop the 160-char per-line cap) and
record every distinct multiline match, so code appended past the cap
or a second cross-line match reopens the finding instead of riding the
first one.
- Give the large-JS-bundle and .pth base64-blob findings a content
digest instead of empty or prefix-only evidence, and record all .pth
import lines, so a changed bundle, blob or import no longer inherits a
baselined empty or truncated key.
- Warn when a loaded baseline has entries without evidence_hash so a
legacy baseline is regenerated rather than silently degraded.
Regenerate scripts/scan_packages_baseline.json against the current dep
set and add regression tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: harden multiline and duplicate evidence handling
Follow-up hardening so the evidence hash tracks the full matched code:
- For DOTALL patterns that match across lines, record every line the match
spans (not just the start line), so a change on a continuation line (the
URL inside a baselined C2 loop, a swapped credential path) reopens the
finding. A pathological greedy span is bounded to its head line plus a
digest of the rest.
- Keep duplicate spans in the canonical evidence so a second identical
matched line in a new code path changes the key instead of deduping away.
- Anchor the evidence prefix to strip only a genuine leading label or
line-number marker, leaving a marker-like "L<NN>:" inside raw .pth code
intact.
- Make the legacy-baseline warning explicit that entries without an
evidence_hash reopen rather than suppress under a coarse key.
Regenerate scripts/scan_packages_baseline.json (same finding set; entries
for same-file repeated checks are now tracked separately) and add tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: bind every combo and large finding to its full content
Close the remaining asymmetric-evidence gaps so a changed payload cannot
ride a reviewed baseline entry:
- Digest a capped multiline span from the code without line markers, so a
pure line shift stays stable while a continuation-line change reopens.
- Give the "Unusually large executable .pth" finding a content digest
instead of keying on byte size and import-line count alone.
- Record both contributing signals for the JS credential+network stealer,
the shell credential+network and persistence-hook combos, and the hidden
network+exec docstring payload, so changing the network/exec side reopens.
- Allow punctuation in an evidence label prefix so a "network+exec:" label
is stripped and line shifts do not change the key.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* scan_packages: bind remaining Python combos; key npm baseline on evidence
Python scanner: the openssl+key, anti-analysis, DNS-exfil and base64+exec+blob
combos recorded only one contributing signal, so a changed payload on the other
side could ride a reviewed baseline entry. Each now binds every co-occurring
signal (and the blob is digested, since it can sit on a separate line from the
decode call).
npm scanner: scan_npm_packages.py keyed its allowlist on (package, path,
pattern) only, the same coarse-key bypass the Python scanner just closed. Add an
evidence hash to the key (schema v3, fail-closed on older baselines) and store
full evidence. The committed baseline stays empty by design.
Regenerate scripts/scan_packages_baseline.json and add tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_npm_packages: bind full blob evidence and harden baseline loader
Follow-up on the npm evidence-hash key:
- _evidence now records every match and, when a snippet is truncated for
display, appends a digest of the full match. The obfuscated-blob key was
hashing only the truncated first-match snippet, so a changed payload tail or
an appended blob in the same package/file/pattern could ride a reviewed entry.
- _load_baseline guards that the root is an object, entries is a list, and each
entry is a dict before reading it, so a malformed baseline warns and fails
closed instead of raising AttributeError.
Add tests for a changed blob tail reopening the key and for malformed entries.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan_packages: symmetric baseline-loader guards; bind npm outbound host context
- Python _load_baseline now rejects a non-list "entries" with a warning instead
of raising TypeError, matching the npm loader.
- npm cred-surface-host (outbound) records the host with its URL path / fetch
call / host config, so a changed outbound path, headers or body reopens the
key rather than riding the bare host literal.
Add tests for both.
* scan_npm_packages: migrate v2 baselines and bind host-config outbound context
- _load_baseline now migrates schema v2 entries by recomputing the evidence
hash from stored evidence (with a legacy warning), matching the Python
loader, instead of discarding them; only pre-v2 basename schemas are rejected.
- The cred-surface-host (outbound) host-config branch now captures the whole
line (path, headers, body), so a changed outbound payload on the same
hostname line reopens the key instead of riding the bare host snippet.
Add tests for v2 migration and the host-config context binding.
* scan packages: bind PEM key bodies and npm windowed evidence to baseline keys
scan_packages: embedded-key findings now pin the full PEM block (BEGIN..END)
via a content digest, so a key body swapped under the same marker reopens the
finding instead of riding the unchanged BEGIN line. Single-line and DER keys
were already bound by their full matched line; marker-only references with no
END block (validation header lists) are unaffected, so the committed baseline
is unchanged.
scan_npm_packages: _evidence now digests the full containing line whenever the
shown snippet is only a window into it (short match on a long line, or a
truncated payload), so a changed payload tail outside the display window
reopens the key. The npm baseline is empty, so this changes no suppressions.
Adds regression tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan packages: bind multi-line evidence and every blob to baseline keys
_extract_evidence now extends each single-line match over its bracket
continuations, so a multi-line call binds its argument lines and a changed
URL or body on a continuation line reopens the key. After the per-line pass it
also records cross-line matches the scan cannot otherwise see (a DOTALL regex,
or a multi-line construct appended under a check that already had a one-line
match), so an appended multiline payload reopens instead of riding the key.
_blob_digest hashes every large base64 blob (not just the first) for the
base64+exec finding and the .pth large-blob finding, so an appended or swapped
second encoded payload reopens; single-blob files keep the same digest.
scan_npm_packages _evidence digests the full logical line (the matched line
plus its bracket-continuation lines), so a multi-line fetch's option and header
lines bind and a changed payload on a following line reopens the outbound key.
Regenerated the Python baseline: same package/file/check set, 24 entries pick
up the wider multi-line evidence. Adds regression tests for each case.
* scan packages: stop giant greedy spans from binding a whole-file digest
When a greedy DOTALL pattern (reverse shell socket...subprocess, C2 loop) has
its anchor tokens far apart, the match span covers the whole file. Digesting
that span bound thousands of unrelated lines, so the evidence hash drifted on
any edit between the anchors (a dependency bump reshuffling the file), which
made a baselined finding reopen on an upstream release. The multiline pass now
skips an oversized span when the per-line pass already bound the signal lines,
so the evidence is the stable matched lines; a genuinely appended multi-line
construct stays under the cap and is still recorded.
Regenerated the Python baseline against Python 3.12 (the version the scan CI
shards run) so the resolved dependency set matches CI. Same package/file/check
set. Adds a regression test.
* scan packages: tighten evidence binding (order, string brackets, span size)
Address review follow-ups on the evidence extraction:
- _canon_evidence keeps discovery (line) order instead of sorting. Line-shift
stability already comes from stripping the L<NN>: markers, so order stays
significant and reordering matched lines (a multi-line call's arguments)
reopens the finding.
- _logical_line_end (Python) and _logical_line_text (npm) blank string literals
before counting brackets, so a ) inside a string argument does not close the
logical line early and drop later argument lines.
- The oversized-span skip now only drops a giant whole-file bridge (over 60
lines); a genuinely appended multi-line construct is recorded so its payload
reopens, rather than riding an existing one-line match.
- npm _logical_line_text binds the enclosing bracket group, so a host-config
object whose { is on a prior line binds its path/headers/body lines.
Regenerated the Python baseline (Python 3.12, matching the scan CI shards):
same package/file/check set. Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan npm packages: normalize and bound the logical-line digest
- _evidence whitespace-normalizes the logical line before digesting (matching
_evidence_hash), so a formatter-only reindent of the bound continuation lines
does not change the sha256 suffix and reopen an unchanged finding.
- _logical_line_text follows a bracket group to its close up to a hard 200-line
cap (digest input only), so a config object longer than the backward window
still binds its whole tail instead of silently truncating.
Adds regression tests. npm baseline is empty, so no regeneration is needed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: cap single-line evidence and widen npm opener window
Cap each rendered evidence line at 200 chars in scan_packages.py: a long
or minified one-line file is shown as a bounded prefix plus a sha256 of the
full line, so a packed payload cannot dump unbounded content into the CI
logs or baseline while a change past the cutoff still changes the digest
and reopens the finding. Mirrors how the npm scanner bounds its snippets.
Widen the npm backward opener window (_MAX_CONT_LINES 12 to 200, symmetric
with the forward cap) so a host deep inside a large options object binds
the whole object, not just its own line; a changed path, header, or body on
any property reopens.
Regenerate the Python baseline with Python 3.12: only the protobuf
nspkg.pth and unsloth-zoo compiler.py evidence change, both from the new
line cap; the package/file/check key set is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* scan: bind all host contexts, deep call continuations, far-back npm openers
Three fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: measure the forward bracket-group cap from the matched
line (idx + _MAX_GROUP_LINES) instead of the opener, so an opener found near
the widened backward limit no longer consumes the forward budget and drops
the path, headers, or body that follow the host.
scan_npm_packages.py: _outbound_host_evidence now records every outbound
context form for a host (URL, fetch-context, host-config), claiming each
non-overlapping match in form order, so a separate host-config request added
beside an already-baselined URL changes the evidence and reopens the key.
The common single-context case keeps its existing snippet.
scan_packages.py: follow a matched Python call over its continuations up to a
separate _MAX_CALL_LINES (40), decoupled from the 12-line display threshold,
so a multi-line requests.post( binds its whole argument list in the digest
and a changed body deep in the call reopens; bounded so a miscounted bracket
cannot swallow unrelated code. No baseline change: the current dependency set
has no matched call that closes between 13 and 40 lines, confirmed by a
Python 3.12 regenerate that produced a byte-identical baseline.
* scan: clamp npm depth, pin large bundles, follow backslash and bound .pth dump
Four fail-closed evidence gaps surfaced by review of the previous round.
scan_npm_packages.py: clamp the backward opener scan at depth 0 so a leading
unmatched closer (a preceding block whose opener is outside the backward
window) no longer drives depth negative and masks the real enclosing opener
that follows; a host-config object after such a block now binds and a changed
path reopens.
scan_packages.py: a large JS bundle now pins its whole content even when
another JS heuristic already fired. The bundle digest was only added when no
other finding existed; it is now appended to every finding's evidence on a
large bundle, so an unchanged obfuscation signature no longer lets changed
payload elsewhere ride the matched-line key.
scan_packages.py: _logical_line_end follows explicit backslash line
continuations, so a call split with a backslash before its parenthesis binds
the continuation line (URL/body) instead of returning at the zero-depth API
line.
scan_packages.py: the catch-all .pth import evidence is bounded through
_cap_line (prefix plus a digest of every line) so a large .pth of benign
imports cannot dump the whole member into the logs or baseline while an
appended or swapped import still reopens.
Baseline regenerated with Python 3.12: key set unchanged; one entry
(unsloth-zoo compiler.py) gains the backslash-continued banner lines now
bound by the continuation fix.
* scan: handle multi-line strings, lifecycle bodies, and de-quadratic evidence
Addresses a review round plus a performance audit of the evidence extractor.
Correctness (fail-closed):
- Bind the UNION of the single-line-blanked and multi-line-blanked bracket spans
in both scanners. The multi-line view blanks a triple-quoted Python string or a
backtick template literal that spans lines, so a `)` inside such a string no
longer closes the enclosing call early and drop later arguments. The single-line
view still counts a payload embedded INSIDE a string, so a dropper that hides a
call in a string keeps its argument lines bound. Taking the larger span never
shrinks the binding below either view, avoiding a fail-open regression.
- cred-env-in-lifecycle now pins the whole lifecycle script body via a digest, so
a changed non-token line (e.g. adding a curl exfil beside the token reference)
reopens, not just a change on the token line.
Performance / DoS (the scanner runs on attacker-controlled package files up to the
64 MiB / 16 MiB member caps, with no per-file time budget):
- _extract_evidence precomputes newline offsets once and maps match offsets with
bisect, removing the O(matches) whole-file content.count per match that made the
finditer fallback quadratic (a crafted minified file went from ~13 s/MiB and
hours at the cap to linear).
- npm _index_text splits and string-blanks the file once per evidence call instead
of per match (was O(matches x file) time and allocation).
- Bound evidence output: _MAX_EVIDENCE_SPANS (Python) and _MAX_EVIDENCE_MATCHES
(npm) fold the remainder into a digest so a file with thousands of matches cannot
build a multi-megabyte evidence/baseline blob while an added/removed match past
the cap still changes the key.
- _outbound_host_evidence caps matches per form and bounds the overlap claim so a
host repeated many times cannot make it quadratic.
No baseline change: a Python 3.12 regenerate is byte-identical (the union equals the
legacy single-line span for every current dependency file; the cap thresholds sit
above the largest real entry), so these are forward-looking hardening with no drift.
* scan: count all overflow matches, bind their context, blank JS regex literals
Follow-ups on the evidence output caps from the previous commit.
- _outbound_host_evidence no longer truncates each pattern's match iterator with
islice; it iterates every match and runs the overlap dedup only while the
display list is below the cap (so claimed stays bounded and the check is O(cap)
per match, not quadratic), folding every match past the cap into the overflow
digest. A host context beyond the 64th is counted again, so it reopens.
- The overflow digest (both scanners, via a shared _overflow_digest) binds each
overflow match's logical-line context, not just the regex match text, so a
changed payload on an over-cap line reopens even with the matched token
unchanged.
- The multi-line JS blanked view now blanks regex-literal bodies (tracking the
previous significant char for regex-vs-division and char classes for a literal
`/` inside `[...]`), so a `)` inside `/)/` no longer closes an outbound call
early. The bound span is the union of the single-line and multi-line views, so
an imperfect regex decision only ever grows the span, never shrinks it.
- The Python overflow digest canonicalizes spans (strips L<NN>: markers via
_canon_evidence) before hashing, restoring line-shift stability for the
over-cap region.
No baseline change: the overflow branches only trigger above the per-finding caps
(above the largest real entry), and the npm baseline is empty, so a Python 3.12
regenerate is byte-identical.
* scan: refresh baseline for ipython interactiveshell.py span drift
A newer ipython release changed the filesystem-enumeration span in
IPython/core/interactiveshell.py, so its content digest no longer matched the
baselined evidence and the studio scan shard flagged it as a non-baselined
CRITICAL. Regenerated with Python 3.12: only the ipython entry's evidence_hash
changes; the package/file/check key set is unchanged, and a studio enforcing
spot-check exits 0.
* Bound scanner evidence memory: stream overflow spans and cap lifecycle baseline size
scan_packages.py: _extract_evidence no longer materializes a rendered span
per match before slicing at the display cap. Once out holds _MAX_EVIDENCE_SPANS
spans, further spans fold straight into a running digest, so a minified or
padded file with hundreds of thousands of matching lines keeps memory bounded
to the display cap instead of the match count. The fold reproduces
_canon_evidence(" | ".join(overflow)) byte for byte, so the overflow digest and
every baseline key are unchanged.
scan_npm_packages.py: lifecycle-fetch-exec and cred-path-in-lifecycle stored the
entire install script body as evidence, so --write-baseline on a package with a
multi-MiB lifecycle script bloated the baseline JSON. Both now store a bounded
matched snippet plus a body-sha256 digest, matching cred-env-in-lifecycle. The
digest still binds the whole body, so a change to any line reopens the finding.
Adds tests for the streamed overflow bound and the bounded-but-reopens lifecycle
evidence. Baseline unchanged (byte-identical Python evidence; npm baseline empty).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make npm bracket-group scan order-aware so a same-line close-then-open binds
_scan_group counted brackets with a per-line net (opens minus closes), which
collapses intra-line order: a line that closes a prior block and then opens the
host-config object on the same line, e.g. `}); const opts = {`, nets to <= 0, so
the trailing `{` was dropped and the group started at the hostname line. A
changed path/headers on the following lines then hashed to the same evidence and
could ride an existing baseline key.
Replace the net count with an order-aware (L, R) reduction per line (L closers
needing an opener to the left, R openers needing a closer to the right) and apply
it in order in both the backward and forward scans, clamping stray closers at 0.
The trailing opener now stays visible so the whole object binds and a changed
payload reopens. Per-line cost is unchanged (one C-level bracket findall), so the
existing outbound-host evidence is byte-identical on all prior shapes; only the
previously-dropped same-line case changes. Adds a regression test for it.
* Harden scanner evidence: bound memory and bind Python call tails fail-closed
Five fixes across both scanners, none of which change the committed baseline (a
full regen of all three pip shards produced a byte-identical 185-key set).
scan_npm_packages.py: _evidence and _outbound_host_evidence collected every regex
match into a list before applying the 64-match display cap, so a text file under
the size cap that repeats a cheap signal (such as NPM_TOKEN) millions of times
could allocate a huge list of re.Match objects and stall or OOM before the
overflow digest ran. They now stream from finditer and fold overflow as matches
arrive via a shared _fold_overflow_match helper, byte-identical to the prior
digest.
scan_packages.py:
- _extract_evidence kept inserting every unique over-cap span into the seen set
even after it stopped appending to the display list, so a generated file with
millions of one-line matches still grew that set unbounded. It now tracks spans
only while filling the display list (per-line spans are unique by line number,
so dropping them past the cap cannot miss a dedup).
- _scan_line_end counted brackets with a per-line net, so a continued statement
that closes on the same line it opens a flagged call (a leading "]" before
"requests.post(") had the call's open paren cancelled and bound only the opener
line. It now applies brackets in order via _bracket_lr (leading closers clamp at
0), matching the npm bracket fix.
- a single-quoted string continued by a trailing backslash was not tracked across
lines, so a close paren inside the continued string on the next line closed the
call early; _blank_code_strings now carries the continuation.
- a call with more argument lines than the soft cap was hashed only through the
cap, so a changed data=/headers tail past it stayed suppressed; a closing call
is now followed to its real close under a 200-line hard limit (a never-closing
opener still stops at the 40-line soft cap so it cannot swallow the file).
Adds regression tests for each. npm baseline is empty; the Python baseline is
unchanged (verified byte-identical by regenerating all three shards).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bind giant DOTALL span anchors and add context to constant IOC evidence
Two fail-closed gaps where a changed payload could keep the same evidence hash
and stay suppressed by the baseline.
scan_packages.py: a giant greedy DOTALL span (a cross-line IOC match bridging
more than 60 lines, e.g. RE_TEMP_EXEC matching a /tmp line and a much-later
subprocess line) was dropped entirely once the per-line pass had any match, so an
appended cross-line payload -- a new /tmp line plus a later subprocess line that
share no single line, so the per-line pass never binds them -- produced the same
evidence and rode the key. The span is no longer dropped: it is bound by its head
and tail anchor lines plus a digest over just those (no line numbers, so a pure
line shift is stable). An added or moved anchor reopens the finding, while churn
in the bridged interior stays stable, so this does not reintroduce whole-file
drift. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry
such a span and are refreshed; a full three-shard regen confirmed only those two
keys change.
scan_npm_packages.py: known-ioc-string and cred-surface-host (always-bad) recorded
only the bare needle/host as evidence, so a reviewed tarball that kept the IOC
string while altering the adjacent fetch/exfil body produced an identical key.
They now bind matched-line context: known-ioc-string via the matched line and its
bracket-group continuation, cred-surface-host (always-bad) via the outbound call
context (path/headers/body, falling back to the bare host when not in an outbound
call). A changed payload on the same call now reopens.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
only the two giant-span entries.
* Hash giant-span interiors, bind exec/eval trigger, JS content, intra-literal whitespace
Four fail-closed gaps where a changed payload could keep the same evidence hash.
scan_packages.py:
- A giant bridged DOTALL span was bound only by its head and tail anchors, so a
cross-line payload inserted into the bridged interior between unchanged outer
anchors kept the same key. The whole span content is now digested (via _render),
so any interior change reopens; a pure line shift stays stable because the digest
is over the markerless code. Two baseline entries (multiprocess test, unsloth-zoo
scanner file) carry such a span; with full-interior binding, multiprocess
resolved at two versions across shards now yields two distinct entries where the
anchor digest had collapsed them into one.
- The exec/eval-with-hidden-payload findings omitted the visible exec/eval line
that makes the hidden string executable, so flipping a harmless eval("1+1") to
exec(__doc__) kept the same key while arming the payload. The trigger line from
the real-code view is now bound into the evidence.
- check_js_file extracted evidence with the Python-string-aware extractor, which
does not blank JS backtick template literals, so a template containing a close
paren closed a call's bracket span early and omitted later option/body lines. The
full file content digest is now pinned to every JS finding (not just large
bundles), binding the whole call.
scan_npm_packages.py: the evidence canon collapsed all whitespace via split(),
erasing whitespace inside JS string literals along with harmless indentation, so a
changed request body 'a b' -> 'a b' kept the same key. A new _canon_preserve_strings
collapses whitespace only OUTSIDE string literals (reindent-stable) while preserving
it INSIDE single/double/backtick literals (intra-payload edits reopen). Used for the
evidence hash and the logical-line digests.
Adds regression tests for each. npm baseline is empty; the Python baseline updates
the two giant-span entries and adds the second multiprocess version's entry.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han