unsloth/studio/backend
danielhanchen 047d9d4d70 Studio: strip orphaned tool-call markup and byte-fallback chars at the tool-call chokepoints
On an MTP GGUF model, speculative decoding on a quantized target surfaces
byte-fallback garbage as U+FFFD plus an orphaned tool-call close tag whose opener
was drained or mangled, which leaked into chat (issue #7084).

Centralize the scrub at the tool-call chokepoints instead of scattering it
per delta:

- strip_tool_markup now sanitizes control chars / U+FFFD and removes a trailing
  orphan-close run, gated by a </tool_call> style sentinel so code/XML literals
  survive. Kimi and DeepSeek end-of-turn closers join the sentinel set.
- The two streaming-strip entries (safetensors strip_tool_markup_streaming and
  the GGUF _strip_tool_markup_streaming) sanitize at the top and drop trailing
  orphan closes on the final segment, so live display matches the finalized
  answer.
- ToolLoopController.record_result scrubs a tool result before it reaches the
  model or the tool card.

Drops the scattered per-delta content, tool-args, and reasoning-channel scrubs
and the secondary live-stream scrub, which are now covered by the chokepoints
above.
2026-07-20 12:58:21 +00:00
..
assets Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
auth Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
core Studio: strip orphaned tool-call markup and byte-fallback chars at the tool-call chokepoints 2026-07-20 12:58:21 +00:00
hub Studio: hide the RAG embedder and llama.cpp probe from the hub cached inventory (#7018) 2026-07-19 03:20:56 -07:00
loggers Studio: quiet noisy logs, log real progress, and speed up Windows/macOS dataset prep (#7087) 2026-07-15 06:49:52 -07:00
models Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
plugins Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
requirements Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
routes Studio: hide the RAG embedder and llama.cpp probe from the hub cached inventory (#7018) 2026-07-19 03:20:56 -07:00
state Studio: shareable per-checkpoint preview links (#6486) 2026-06-24 06:31:53 -07:00
storage Studio: expose Windows drive roots in the folder browser (#7082) 2026-07-15 00:24:11 -07:00
tests Studio: strip orphaned tool-call markup and byte-fallback chars at the tool-call chokepoints 2026-07-20 12:58:21 +00:00
utils Studio: hide the RAG embedder and llama.cpp probe from the hub cached inventory (#7018) 2026-07-19 03:20:56 -07:00
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
_platform_compat.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
cloudflare_tunnel.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
colab.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
main.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
mcp_server.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
run.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
startup_banner.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00