d87c21aebfWhen _select_gpus determines that a GGUF model fits on the selected GPU(s), the code sets CUDA_VISIBLE_DEVICES but never passes -ngl (number of GPU layers) to llama-server. Without -ngl or --fit, llama-server defaults to 0 GPU layers and runs entirely on CPU. This adds -ngl -1 (offload all layers) in the elif branch where gpu_indices is set and use_fit is False, so models that fit in VRAM actually use the GPU for inference. Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| backend | ||
| frontend | ||
| __init__.py | ||
| install_llama_prebuilt.py | ||
| install_python_stack.py | ||
| LICENSE.AGPL-3.0 | ||
| setup.bat | ||
| setup.ps1 | ||
| setup.sh | ||
| Unsloth_Studio_Colab.ipynb | ||