ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving (#6164)

Browse source 
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving

A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.

Changes:
- After llama-server health, read default_generation_settings.n_ctx from
  /props and adopt it whenever it is below Studio's computed context, with
  a warning. The load response, status route, UI value, and the passthrough
  max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
  budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
  agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
  UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
  the passthrough drops whole middle turn-groups (system prompt, first turn,
  and recent turns kept; tool calls stay paired with their results), clips
  oversized contents middle-out when group-dropping is not enough, clamps
  max_tokens to the generation headroom, and retries. Default stays 'error'
  with code=context_length_exceeded so clients running their own compaction
  keep full control.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)

Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
  always passes --parallel N, which silently splits -c into per-slot windows
  of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
  advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
  --fit-ctx <requested> for explicit requests so fit offloads or fails into
  the existing --fit off retry instead of silently shrinking the window.

Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-06-11 07:49:55 -07:00 • committed by GitHub
commit bc85ecd145
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 924 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

78
studio/backend/core/inference/llama_cpp.py
View file

@ -1153,6 +1153,8 @@ class LlamaCppBackend:
"ngram_mod_flavor": None,
"supports_ngram_mod": False,
"spec_draft_n_max_flag": None,
"supports_kv_unified": False,
"supports_fit_ctx": False,
}
try:
mtime = int(Path(bin_path).stat().st_mtime)
@ -1166,6 +1168,8 @@ class LlamaCppBackend:
mtp_token: Optional[str] = None
ngram_mod_flavor: Optional[str] = None
spec_draft_n_max_flag: Optional[str] = None
supports_kv_unified = False
supports_fit_ctx = False
try:
result = subprocess.run(
[bin_path, "--help"],
@ -1253,6 +1257,9 @@ class LlamaCppBackend:
spec_draft_n_max_flag = "--spec-draft-n-max"
elif _is_real("--draft-max"):
spec_draft_n_max_flag = "--draft-max"
supports_kv_unified = _is_real("--kv-unified")
supports_fit_ctx = _is_real("--fit-ctx")
except (OSError, subprocess.SubprocessError) as exc:
logger.debug(f"llama-server --help probe failed: {exc}")
@ -1263,6 +1270,8 @@ class LlamaCppBackend:
"ngram_mod_flavor": ngram_mod_flavor,
"supports_ngram_mod": ngram_mod_flavor is not None,
"spec_draft_n_max_flag": spec_draft_n_max_flag,
"supports_kv_unified": supports_kv_unified,
"supports_fit_ctx": supports_fit_ctx,
}
cls._capability_cache[cache_key] = info
return info
@ -3232,6 +3241,16 @@ class LlamaCppBackend:
# Fits on selected GPU(s) -- offload all layers
cmd.extend(["-ngl", "-1"])
cmd.extend(
self._ctx_integrity_flags(
n_parallel,
use_fit,
requested_ctx,
effective_ctx,
self.probe_server_capabilities(binary),
)
)
# -1 = llama.cpp auto-detect (physical cores). Pass explicitly
# so we don't inherit llama-server's internal default, which
# has varied (hardware concurrency incl. hyperthreads on some
@ -3585,6 +3604,7 @@ class LlamaCppBackend:
self._effective_context_length = (
effective_ctx if effective_ctx > 0 else self._context_length
)
self._reconcile_effective_ctx_with_server()
self._max_context_length = (
max_available_ctx if max_available_ctx > 0 else self._effective_context_length
)
@ -4468,6 +4488,64 @@ class LlamaCppBackend:
logger.error(f"llama-server health check timed out after {timeout}s")
return False
@staticmethod
def _ctx_integrity_flags(
n_parallel: int, use_fit: bool, requested_ctx: int, effective_ctx: int, caps: dict
) -> list[str]:
"""Flags that keep the per-request window equal to the advertised ctx.
Explicit ``--parallel`` disables llama-server's auto-slots
``--kv-unified`` default, silently splitting ``-c`` into per-slot
windows of ``-c / N``; restore the shared pool so one request can use
the full context. With ``--fit on``, ``--fit-ctx`` floors the fit step
at an explicitly requested ctx (default floor is 4096) so it offloads
or fails instead of silently shrinking the window.
"""
flags: list[str] = []
if n_parallel > 1 and caps.get("supports_kv_unified"):
flags.append("--kv-unified")
if use_fit and requested_ctx > 0 and effective_ctx > 0 and caps.get("supports_fit_ctx"):
flags.extend(["--fit-ctx", str(effective_ctx)])
return flags
def _query_server_n_ctx(self) -> Optional[int]:
"""Per-slot context llama-server actually allocated, from ``/props``.
The memory-fit step or ``--parallel`` slot split can leave this below
the requested ``-c``; requests are validated against this value.
"""
url = f"http://127.0.0.1:{self._port}/props"
try:
resp = httpx.get(url, timeout = 5.0)
if resp.status_code != 200:
return None
settings = resp.json().get("default_generation_settings") or {}
n_ctx = settings.get("n_ctx")
return int(n_ctx) if n_ctx else None
except Exception:
return None
def _reconcile_effective_ctx_with_server(self) -> None:
"""Adopt the server's real ``n_ctx`` when it is below Studio's value.
Keeps ``context_length`` (load response, status route, passthrough
``max_tokens`` ceiling) honest; clients sized to the requested value
would otherwise hit ``exceed_context_size_error`` 400s early.
"""
actual_n_ctx = self._query_server_n_ctx()
if not actual_n_ctx or actual_n_ctx <= 0:
return
if self._effective_context_length and actual_n_ctx < self._effective_context_length:
logger.warning(
"llama-server allocated a smaller per-request context than "
f"requested ({self._effective_context_length} -> {actual_n_ctx}; "
"memory fit or --parallel slot split); clients must treat "
f"{actual_n_ctx} as the real context window."
)
self._effective_context_length = actual_n_ctx
elif not self._effective_context_length:
self._effective_context_length = actual_n_ctx
# ── Message building (OpenAI format) ──────────────────────────
@staticmethod