ae2b1b97baThe prebuilt llama.cpp binary (cuda13-newer) links against libcudart.so.13 and libcublas.so.13. When torch is installed via pip, these libraries live in the venv's site-packages under nvidia/cu13/lib/, not in /usr/local/cuda/. The existing LD_LIBRARY_PATH logic only searched /usr/local/cuda* paths (which have CUDA 12.x), so the CUDA backend failed to load silently and llama-server fell back to CPU -- even with -ngl -1. This adds a glob scan of the venv's nvidia package directories (cu*, cudnn, nvjitlink) to LD_LIBRARY_PATH before launching llama-server, matching where pip puts the CUDA runtime. Tested on Colab with RTX PRO 6000 Blackwell (CUDA 13.0, pip torch): before -- 3 MiB GPU, 0% util, CPU inference after -- 13317 MiB GPU, 77% util, full GPU inference Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| backend | ||
| frontend | ||
| __init__.py | ||
| install_llama_prebuilt.py | ||
| install_python_stack.py | ||
| LICENSE.AGPL-3.0 | ||
| setup.bat | ||
| setup.ps1 | ||
| setup.sh | ||
| Unsloth_Studio_Colab.ipynb | ||