ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Revert tool-calling trim on Linux + Windows; keep Mac

Browse source 
Per follow-up: only Mac needs the trim. Linux/Windows runners are
fast enough that the original max_tokens (120/600/600/400/300 on
linux, 600/600/600/400/300 on windows) and the dedicated terminal-
tool SSE round are kept.

Restores on linux + windows:
- Section 3 "Server-side bash (terminal) tool" axis with the hard
  `assert "hello-bash-tool" in content` check (linux) or non-empty
  SSE assertion (windows).
- max_tokens: function calling 96 -> 120 (linux) / 128 -> 600 (windows),
  python tool 320 -> 600, web_search 192 -> 400, thinking 160 -> 300.

Mac job keeps the trim from 7878c655: dropped terminal axis +
halved max_tokens. Macos-14 free runner is ~10 tok/s and the trim
takes the step from 338 s to ~170 s.
This commit is contained in:
Daniel Han 2026-05-08 09:39:12 +00:00
commit e3f9727912
2 changed files with 48 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

39
.github/workflows/studio-inference-smoke.yml vendored
View file

@ -478,10 +478,7 @@ jobs:
"stream": False,
"temperature": 0.0,
"seed": SEED,
# tool_choice='required' constrains the grammar so the
# model emits the JSON tool_call envelope directly; 96 is
# plenty for `{"city":"Paris"}` plus the wrapping fields.
"max_tokens": 96,
"max_tokens": 120,
})
assert status == 200, f"tool call status {status}: {data}"
choice = data["choices"][0]
@ -496,8 +493,6 @@ jobs:
# 123 * 456 = 56088. The agentic loop streams SSE; we
# accumulate the assistant text and look for the answer. We
# accept "56088" or "56,088" since the model may format it.
# 320 tokens covers the tool_call + tool result + brief
# natural-language answer; 600 was 2x what the model needs.
content = post_sse("/v1/chat/completions", {
"messages": [{"role": "user", "content": "What is 123 * 456? Use the python tool to compute it and tell me the number."}],
"enable_tools": True,
@ -505,20 +500,29 @@ jobs:
"session_id": "ci-tool-calling-py",
"temperature": 0.0,
"seed": SEED,
"max_tokens": 320,
"max_tokens": 600,
})
assert "56088" in content or "56,088" in content, (
f"expected 56088 in python-tool answer, got: {content!r}"
)
print(f"[tools] PASS python tool ({len(content)} chars)")
# NOTE: the dedicated "Server-side bash (terminal) tool" axis
# was dropped in favour of the python axis above. Both share
# the same server-side agentic-loop wiring (only the registry
# entry differs); the python axis is the canonical proof.
# Saves one SSE round (~30 s on macos, ~12 s on linux/windows).
# ── 3. Server-side bash (terminal) tool ──────────────────────
content = post_sse("/v1/chat/completions", {
"messages": [{"role": "user", "content": "Use the terminal tool to run `echo hello-bash-tool` and tell me the exact output."}],
"enable_tools": True,
"enabled_tools": ["terminal"],
"session_id": "ci-tool-calling-bash",
"temperature": 0.0,
"seed": SEED,
"max_tokens": 600,
})
assert "hello-bash-tool" in content, (
f"expected 'hello-bash-tool' in terminal-tool answer, got: {content!r}"
)
print(f"[tools] PASS bash/terminal tool ({len(content)} chars)")
# ── 3. Server-side web_search tool ───────────────────────────
# ── 4. Server-side web_search tool ───────────────────────────
# DuckDuckGo is flaky from CI runners and small Qwen3.5-2B
# may not actually search. Only assert that the SSE stream
# opens and yields any data; HTTP / parser failures already
@ -531,13 +535,13 @@ jobs:
"session_id": "ci-tool-calling-web",
"temperature": 0.0,
"seed": SEED,
"max_tokens": 192,
"max_tokens": 400,
})
print(f"[tools] PASS web_search stream ({len(content)} chars)")
except Exception as exc:
print(f"[tools] WARN web_search probe failed (non-blocking): {exc}")
# ── 4. Thinking on / off ─────────────────────────────────────
# ── 5. Thinking on / off ─────────────────────────────────────
# Studio strips think blocks from message.content for tools-mode
# responses, so we toggle plain chat (no enable_tools) and look
# at the surfaced reasoning_content / message.thinking field.
@ -548,10 +552,7 @@ jobs:
"enable_thinking": enable,
"temperature": 0.0,
"seed": SEED,
# 17 is small; 160 tokens is plenty of room for either
# "Yes, 17 is prime" + brief reasoning or a short
# <think>...</think>+answer. 300 was overkill.
"max_tokens": 160,
"max_tokens": 300,
})
assert status == 200
msg = data["choices"][0]["message"]

47
.github/workflows/studio-windows-inference-smoke.yml vendored
View file

@ -609,10 +609,7 @@ jobs:
"stream": False,
"temperature": TEMP,
"seed": SEED,
# tool_choice='required' constrains the grammar so the
# model emits the JSON tool_call envelope directly; 128
# is plenty for `{"city":"Paris"}` plus the wrapping.
"max_tokens": 128,
"max_tokens": 600,
})
assert status == 200, f"tool call status {status}: {data}"
choice = data["choices"][0]
@ -632,8 +629,6 @@ jobs:
)
# ── 2. Server-side python tool ───────────────────────────────
# 320 tokens covers tool_call + result + brief answer; 600
# was 2x what the model needs.
content = post_sse("/v1/chat/completions", {
"messages": [{"role": "user", "content": "What is 123 * 456? Use the python tool to compute it and tell me the number."}],
"enable_tools": True,
@ -641,7 +636,7 @@ jobs:
"session_id": "ci-tool-calling-py",
"temperature": TEMP,
"seed": SEED,
"max_tokens": 320,
"max_tokens": 600,
})
if "56088" in content or "56,088" in content:
print(f"[tools] PASS python tool ({len(content)} chars, found 56088)")
@ -652,13 +647,30 @@ jobs:
f"model didn't return 56088 -- model output drift"
)
# NOTE: the dedicated "Server-side bash (terminal) tool" axis
# was dropped in favour of the python axis above. Both share
# the same server-side agentic-loop wiring (only the registry
# entry differs); the python axis is the canonical proof.
# Saves one SSE round (~12 s on windows-latest).
# ── 3. Server-side bash (terminal) tool ──────────────────────
# On Windows the terminal tool resolves to the system shell
# (cmd.exe wrapper) and `echo hello-bash-tool` works the same
# way it does on POSIX. The model still has to choose to
# invoke the tool; assert non-empty SSE if it doesn't.
content = post_sse("/v1/chat/completions", {
"messages": [{"role": "user", "content": "Use the terminal tool to run `echo hello-bash-tool` and tell me the exact output."}],
"enable_tools": True,
"enabled_tools": ["terminal"],
"session_id": "ci-tool-calling-bash",
"temperature": TEMP,
"seed": SEED,
"max_tokens": 600,
})
if "hello-bash-tool" in content:
print(f"[tools] PASS terminal tool ({len(content)} chars)")
else:
assert content, "terminal tool: SSE stream empty"
print(
f"[tools] WARN terminal tool: SSE OK ({len(content)} chars) but "
f"model didn't echo 'hello-bash-tool' -- model output drift"
)
# ── 3. Server-side web_search tool ───────────────────────────
# ── 4. Server-side web_search tool ───────────────────────────
# DuckDuckGo can be flaky from CI runners; only assert that
# the SSE stream opens and yields any data.
try:
@ -669,13 +681,13 @@ jobs:
"session_id": "ci-tool-calling-web",
"temperature": TEMP,
"seed": SEED,
"max_tokens": 192,
"max_tokens": 400,
})
print(f"[tools] PASS web_search stream ({len(content)} chars)")
except Exception as exc:
print(f"[tools] WARN web_search probe failed (non-blocking): {exc}")
# ── 4. Thinking on / off ─────────────────────────────────────
# ── 5. Thinking on / off ─────────────────────────────────────
def thinking_call(enable):
status, data = post("/v1/chat/completions", {
"messages": [{"role": "user", "content": "Briefly: is 17 prime?"}],
@ -683,10 +695,7 @@ jobs:
"enable_thinking": enable,
"temperature": TEMP,
"seed": SEED,
# 17 is small; 160 tokens is plenty for either "Yes"
# + brief reasoning or a short <think>...</think> +
# answer. 300 was overkill.
"max_tokens": 160,
"max_tokens": 300,
})
assert status == 200
msg = data["choices"][0]["message"]