7b048168c8* Studio: match llama.cpp SWA cache sizing * Studio: account for batch-capped SWA ubatch * Studio: match llama.cpp KV stream padding * Match llama.cpp batch and FA-off cache sizing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip unusable compact SWA slot saves * Align KV planning with launched server * Match cache type casing and narrow the compact SWA slot-save skip The launcher tested the requested cache type case-sensitively while the budget lowercases it via _planned_main_cache_types, so a Q8_0 request emitted no --cache-type flag and llama.cpp ran f16 while the estimate priced q8_0 (1.01 GiB under-reserved on a 27B SWA model at ctx 32768 with 4 slots). The compact SWA slot-save skip keyed on the sliding window alone, but the estimator's SWA path also requires key/value length. phi3 GGUFs report a window without those dimensions and llama.cpp runs them non-SWA, so their slots restore fine and were being skipped. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> |
||
|---|---|---|
| .. | ||
| data_recipe | ||
| export | ||
| inference | ||
| rag | ||
| training | ||
| __init__.py | ||
| _torchao_stub.py | ||
| import_guards.py | ||
| research_runs.py | ||
| tool_healing.py | ||