unsloth/studio/backend
Daniel Han 796f8497e7
Studio (Windows): keep prompt caching on full GPU offload (#7260)
* Studio (Windows): keep prompt caching on full GPU offload (#5692 follow-up)

The #5692 full-offload tuning also added --no-cache-prompt, which disables
in-VRAM prompt-prefix reuse. That is unrelated to the host-RAM KV checkpoints
#5692 fixed (--cache-ram 0 / --ctx-checkpoints 0): a fully offloaded model keeps
its KV cache in VRAM, so reusing a common prefix does not copy to system RAM and
does not cause the PCI-E overhead. --no-cache-prompt only forces every request to
re-prefill the whole prompt, which is small for short chats but severe for large
stable system prompts reused across calls (coding agents, long multi-turn chats).

Remove --no-cache-prompt; keep the checkpoint disables and the thread/OMP tuning.
_prompt_cache_disabled stays False (its default), so slot save/restore is intact.
Verified on a fully offloaded gemma GGUF: an identical repeated prompt reprefills
1 token instead of 2220.

* Guard against re-adding --no-cache-prompt to any llama-server command

Add a backend-wide test that AST-scans studio/backend and fails if
--no-cache-prompt is appended/extended/+= into a command. This locks in
the #7260 fix across every code path, not just load_model. Detecting the
flag or honouring a user-supplied one stays allowed.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-07-20 06:47:08 -07:00
..
assets Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
auth Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
core Studio (Windows): keep prompt caching on full GPU offload (#7260) 2026-07-20 06:47:08 -07:00
hub Studio: validate Hugging Face tokens before use (#7261) 2026-07-20 14:40:14 +01:00
loggers Studio: quiet noisy logs, log real progress, and speed up Windows/macOS dataset prep (#7087) 2026-07-15 06:49:52 -07:00
models Studio: GPU memory configuration for GGUF models (#6414) 2026-07-19 05:46:22 -07:00
plugins Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
requirements Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
routes Studio: validate Hugging Face tokens before use (#7261) 2026-07-20 14:40:14 +01:00
state Studio: shareable per-checkpoint preview links (#6486) 2026-06-24 06:31:53 -07:00
storage Studio: Data settings tab, uploaded files manager, quant pinning, and chat image preview fix (#7029) 2026-07-20 04:57:44 -07:00
tests Studio (Windows): keep prompt caching on full GPU offload (#7260) 2026-07-20 06:47:08 -07:00
utils Studio: validate Hugging Face tokens before use (#7261) 2026-07-20 14:40:14 +01:00
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
_platform_compat.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
cloudflare_tunnel.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
colab.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
main.py Studio: validate Hugging Face tokens before use (#7261) 2026-07-20 14:40:14 +01:00
mcp_server.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
run.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00
startup_banner.py Replace standalone Studio wording with Unsloth (#7221) 2026-07-19 00:47:04 -07:00