80d84a5b5f
Refactor command building (deduplicate HF/local paths) and add flags for better performance: - --parallel 1: studio is single-user, so only 1 inference slot is needed. The previous auto-detect picked 4 slots, wasting VRAM on 3 unused KV caches. - --flash-attn on: force flash attention for faster inference. Default is "auto" which may not always enable it. - --fit on: auto-adjust parameters to fit in available device memory. Already the default but now explicit. Also cleaned up the duplicated command building for HF vs local mode into a single block. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| audio_codecs.py | ||
| inference.py | ||
| llama_cpp.py | ||
| orchestrator.py | ||
| worker.py | ||