unsloth/studio/backend/core/inference
Daniel Han bac2cfac7c feat(studio): architecture-aware KV cache VRAM estimation
Replace the single legacy formula (2 * n_kv_heads * head_dim * n_layers
* n_ctx * bpe) with 5-path estimation that reads 8 additional GGUF
metadata fields:

  1. MLA (DeepSeek-V2/V3, GLM-4.7, GLM-5, Kimi-K2.5) -- K-only cache
     using compressed KV latent + RoPE; no separate V allocation
  2. Hybrid Mamba (Qwen3.5-27B, Qwen3.5-35B-A3B) -- only attention
     layers (1 in N) carry KV; Mamba layers have none
  3. Sliding Window (Gemma-3, gpt-oss) -- SWA layers cache
     min(ctx, window) tokens instead of the full context
  4. Standard GQA -- uses explicit key_length/value_length from GGUF
     instead of embed // n_heads (which is wrong for many models)
  5. Legacy fallback -- identical to old formula for old GGUFs

New GGUF fields parsed: attention.key_length, attention.value_length,
attention.sliding_window, full_attention_interval,
attention.kv_lora_rank, attention.key_length_mla, ssm.inner_size,
ssm.state_size.

Validated against 9 real GGUF files (72/72 field checks pass).
The legacy formula was off by +682% for Gemma-3 and -81% for
DeepSeek-V3.1.
2026-04-01 12:29:20 +00:00
..
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
_html_to_md.py Studio: simplify tool-call dedup and replace html2text with builtin converter (#4722) 2026-03-31 06:15:18 -07:00
audio_codecs.py studio: per-model inference defaults, GGUF slider fix, reasoning toggle (#4325) 2026-03-16 06:37:55 -07:00
defaults.py studio: web search, KV cache dtype, training progress, inference fixes 2026-03-17 00:30:01 -07:00
inference.py [studio] multi gpu: revert to balanced for inference. (#4698) 2026-03-31 01:24:41 -07:00
llama_cpp.py feat(studio): architecture-aware KV cache VRAM estimation 2026-04-01 12:29:20 +00:00
orchestrator.py fix: auto-retry stalled HF downloads with HF_HUB_DISABLE_XET=1 (#4712) 2026-03-31 03:00:46 -07:00
tools.py Studio: simplify tool-call dedup and replace html2text with builtin converter (#4722) 2026-03-31 06:15:18 -07:00
worker.py fix: auto-retry stalled HF downloads with HF_HUB_DISABLE_XET=1 (#4712) 2026-03-31 03:00:46 -07:00