7b048168c8* Studio: match llama.cpp SWA cache sizing * Studio: account for batch-capped SWA ubatch * Studio: match llama.cpp KV stream padding * Match llama.cpp batch and FA-off cache sizing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip unusable compact SWA slot saves * Align KV planning with launched server * Match cache type casing and narrow the compact SWA slot-save skip The launcher tested the requested cache type case-sensitively while the budget lowercases it via _planned_main_cache_types, so a Q8_0 request emitted no --cache-type flag and llama.cpp ran f16 while the estimate priced q8_0 (1.01 GiB under-reserved on a 27B SWA model at ctx 32768 with 4 slots). The compact SWA slot-save skip keyed on the sliding window alone, but the estimator's SWA path also requires key/value length. phi3 GGUFs report a window without those dimensions and llama.cpp runs them non-SWA, so their slots restore fine and were being skipped. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> |
||
|---|---|---|
| .. | ||
| .gitkeep | ||
| __init__.py | ||
| auth.py | ||
| data_recipe.py | ||
| datasets.py | ||
| export.py | ||
| inference.py | ||
| mcp_servers.py | ||
| models.py | ||
| providers.py | ||
| responses.py | ||
| training.py | ||
| users.py | ||