ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
unsloth/studio/backend/tests
History
Daniel Han bc85ecd145
 Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving (#6164) 
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving

A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.

Changes:
- After llama-server health, read default_generation_settings.n_ctx from
  /props and adopt it whenever it is below Studio's computed context, with
  a warning. The load response, status route, UI value, and the passthrough
  max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
  budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
  agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
  UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
  the passthrough drops whole middle turn-groups (system prompt, first turn,
  and recent turns kept; tool calls stay paired with their results), clips
  oversized contents middle-out when group-dropping is not enough, clamps
  max_tokens to the generation headroom, and retries. Default stays 'error'
  with code=context_length_exceeded so clients running their own compaction
  keep full control.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)

Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
  always passes --parallel N, which silently splits -c into per-slot windows
  of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
  advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
  --fit-ctx <requested> for explicit requests so fit offloads or fails into
  the existing --fit off retry instead of silently shrinking the window.

Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-11 07:49:55 -07:00
..
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
conftest.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_amd_apu_unified_memory.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_anthropic_cache_ttl.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_anthropic_citations.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_anthropic_citations_edge.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_anthropic_code_execution.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_anthropic_compaction.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_anthropic_fast_mode_and_refusal.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_anthropic_fast_mode_edge.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_anthropic_messages.py Studio: improve OpenAI- and Anthropic-compatible API spec compliance (#6010) 2026-06-09 17:13:25 +02:00
test_anthropic_thinking_translation.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_anthropic_tool_versions.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_anthropic_web_fetch.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_audio_token_detection.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_browse_folders_route.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_cache_case_resolution.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_cached_gguf_routes.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_chat_history_routes.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_chat_history_storage.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_cleanup_cancelled_checkpoints.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_cloudflare_tunnel.py Studio: auto Cloudflare tunnel for 0.0.0.0 launches (#6204) 2026-06-11 07:10:08 -07:00
test_context_overflow_truncation.py Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving (#6164) 2026-06-11 07:49:55 -07:00
test_cpu_threads.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_data_recipe_github_progress.py Studio: add github_repo seed reader and GitHub Support Bot recipe (#5169) 2026-04-24 12:02:03 -07:00
test_data_recipe_seed.py fix(seed): disable remote code execution in seed inspect dataset loads (#4275) 2026-03-13 19:37:43 +04:00
test_dataset_upload_limits.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_desktop_auth.py Studio: in-app Update llama.cpp button to install the latest prebuilt (#6097) 2026-06-10 10:04:26 -07:00
test_detect_mmproj_file.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_export_log_cursor.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_external_provider_proxy_env.py Studio: ignore unsupported env proxy during Studio startup (#6102) 2026-06-11 05:13:27 -07:00
test_external_provider_usage_chunk.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_frontend_resolution.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_gemini_provider.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_gguf_completion_usage.py Studio: improve OpenAI- and Anthropic-compatible API spec compliance (#6010) 2026-06-09 17:13:25 +02:00
test_gguf_metadata.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_gguf_reload_inheritance.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_gguf_route_cursor_reset.py Improve local chat tool call flow (#5962) 2026-06-09 07:28:44 -07:00
test_gguf_routing.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_gpu_selection.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_gpu_selection_sandbox.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_host_defaults.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_index_bootstrap_origin.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_index_bootstrap_origin_extra.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_inference_model_validation.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_inference_orchestrator_crash_message.py Studio: fix Gemma-4-12B-it not loading (#6054) 2026-06-10 08:39:07 -07:00
test_install_resolve_prebuilt.py Studio: offer the in-app llama.cpp update for source-build (markerless) installs (#6188) 2026-06-11 02:45:12 -07:00
test_kv_cache_estimation.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_lemonade_llamacpp_rocm_bins_mock.py Source llama.cpp prebuilts from unslothai/llama.cpp (CUDA, ROCm, macOS) (#5963) 2026-06-10 08:49:57 -07:00
test_llama_cpp_cache_aware_disk_check.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_context_fit.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_freshness.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_load_progress.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_load_progress_live.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_load_progress_matrix.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_max_context_threshold.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_mmproj_fallback.py Studio: fall back to text-only when llama.cpp is too old for a model's vision projector (#6075) 2026-06-10 08:58:39 -07:00
test_llama_cpp_mtp_detection.py Studio: surface the llama.cpp update affordance when MTP is disabled (#6192) 2026-06-11 06:10:17 -07:00
test_llama_cpp_no_context_shift.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_props_readback.py Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving (#6164) 2026-06-11 07:49:55 -07:00
test_llama_cpp_start_failure_classification.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_llama_cpp_tool_loop.py Improve local chat tool call flow (#5962) 2026-06-09 07:28:44 -07:00
test_llama_cpp_update.py Studio: offer the in-app llama.cpp update for source-build (markerless) installs (#6188) 2026-06-11 02:45:12 -07:00
test_llama_cpp_wait_for_health.py Windows/WSL installer: fix winget msstore cert failure, amd-smi DiskPart prompt, and enable AMD GPU (Strix Halo gfx1151) (#5940) 2026-06-10 04:24:49 -07:00
test_llama_cpp_wait_for_vram_settle.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_cpp_windows_nvidia_path.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_llama_route.py Studio: offer the in-app llama.cpp update for source-build (markerless) installs (#6188) 2026-06-11 02:45:12 -07:00
test_llama_server_args.py Studio: support separate-file MTP GGUF drafters (Gemma 4) (#6125) 2026-06-10 08:45:12 -07:00
test_llm_assist_startup_opt_in.py Studio: make Helper LLM startup pre-cache opt in (#6113) 2026-06-09 15:28:34 +02:00
test_log_filter_no_truncation.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_login_rate_limit.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_mcp_servers.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_mcp_stdio_improvements.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_mcp_stdio_pr5863.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_middleware.py feat(studio): Hub + Download Manager (#5916) 2026-06-09 04:11:24 -07:00
test_mlx_inference_backend.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_mlx_training_worker_config.py Fix Studio MLX VLM resized image layout (#6019) 2026-06-10 07:55:37 -07:00
test_models_get_model_config_case_resolution.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_mtp_drafter_companion.py Studio: support separate-file MTP GGUF drafters (Gemma 4) (#6125) 2026-06-10 08:45:12 -07:00
test_multimodal_document.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_native_context_length.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_offline_gguf_cache_fallback.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_offline_inference_parent.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_openai_citation_markers.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_openai_citation_markers_edge.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_openai_code_execution.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_openai_compaction.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_openai_container_crud.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_openai_image_generation.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_openai_responses_translation.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_openai_tool_passthrough.py Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving (#6164) 2026-06-11 07:49:55 -07:00
test_openai_tool_result_fallbacks.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_pricing.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_pricing_edge.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_providers_api.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_pytorch_mirror.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_rag_captioning.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_rag_chunking.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_rag_embed_llama_server.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_rag_embeddings.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_rag_ingestion.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_rag_preview.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_rag_retrieval.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_rag_store.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_recommended_folders_permission.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_responses_api.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_responses_tool_passthrough.py Studio: forward preserve_thinking + reasoning_effort on the OpenAI passthrough (#6171) 2026-06-10 22:55:43 -07:00
test_rocm_oom_guard.py Windows/WSL installer: fix winget msstore cert failure, amd-smi DiskPart prompt, and enable AMD GPU (Strix Halo gfx1151) (#5940) 2026-06-10 04:24:49 -07:00
test_safetensors_capability_advertise.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_safetensors_tool_loop.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
test_sandbox_tools.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_server_disk_logging.py Windows/WSL installer: fix winget msstore cert failure, amd-smi DiskPart prompt, and enable AMD GPU (Strix Halo gfx1151) (#5940) 2026-06-10 04:24:49 -07:00
test_studio_api.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_studio_train_validation.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_tool_call_parser_strict.py Improve local chat tool call flow (#5962) 2026-06-09 07:28:44 -07:00
test_tool_loop_controller.py Improve local chat tool call flow (#5962) 2026-06-09 07:28:44 -07:00
test_tool_message_empty_content.py Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving (#6164) 2026-06-11 07:49:55 -07:00
test_tool_policy_gates.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_tool_policy_state.py unsloth run: add --enable-tools/--disable-tools server-side tool policy (#5277) 2026-05-05 12:45:15 +04:00
test_tool_xml_strip.py Improve local chat tool call flow (#5962) 2026-06-09 07:28:44 -07:00
test_trained_model_scan.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_training_history_update.py Studio: Dark theme refactor, right sidebar redesign, and chat UI polish (#5150) 2026-05-07 14:33:31 +04:00
test_training_nan_loss_handling.py fix(studio): surface NaN loss honestly instead of laundering to last finite value (#6016) 2026-06-11 07:25:24 -07:00
test_training_raw_support.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_training_worker_flash_attn.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_transformers_version.py Studio: fix Gemma-4-12B-it not loading (#6054) 2026-06-10 08:39:07 -07:00
test_utils.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_vision_cache.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
test_vram_estimation.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
test_windows_gpu_detection_mock.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00