ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

studio: handle reasoning_content in GGUF streaming

Browse source 
llama-server sends thinking/reasoning tokens as "reasoning_content"
in the SSE delta (separate from "content"). The studio was only
reading delta.content, so all reasoning tokens from models like
Qwen3.5, Qwen3-Thinking, DeepSeek-R1, etc. were silently dropped.

This caused "replies with nothing" for thinking models: the model
would spend its entire token budget on reasoning, produce zero
content tokens, and the user would see an empty response.

Fix: read reasoning_content from the delta and wrap it in
<think>...</think> tags. The frontend already has full support
for these tags (parse-assistant-content.ts splits them into
reasoning parts, reasoning.tsx renders a collapsible "Thinking..."
indicator).

Verified with Qwen3.5-27B-GGUF (UD-Q4_K_XL):
  - Before: "What is 2+2?" -> empty response (all tokens in reasoning)
  - After: shows collapsible thinking + answer "4"
This commit is contained in:
Daniel Han 2026-03-14 08:08:34 +00:00
commit 961720c1b1
1 changed files with 19 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

19
studio/backend/core/inference/llama_cpp.py
View file

@ -592,6 +592,7 @@ class LlamaCppBackend:
url = f"{self.base_url}/v1/chat/completions"
cumulative = ""
in_thinking = False
try:
with httpx.Client(timeout = None) as client:
@ -615,6 +616,9 @@ class LlamaCppBackend:
if not line:
continue
if line == "data: [DONE]":
if in_thinking:
cumulative += "</think>"
yield cumulative
return
if not line.startswith("data: "):
continue
@ -624,8 +628,23 @@ class LlamaCppBackend:
choices = data.get("choices", [])
if choices:
delta = choices[0].get("delta", {})
# Handle reasoning/thinking tokens
# llama-server sends these as "reasoning_content"
# Wrap in <think> tags for the frontend parser
reasoning = delta.get("reasoning_content", "")
if reasoning:
if not in_thinking:
cumulative += "<think>"
in_thinking = True
cumulative += reasoning
yield cumulative
token = delta.get("content", "")
if token:
if in_thinking:
cumulative += "</think>"
in_thinking = False
cumulative += token
yield cumulative
except json.JSONDecodeError: