ae2b1b97baThe prebuilt llama.cpp binary (cuda13-newer) links against libcudart.so.13 and libcublas.so.13. When torch is installed via pip, these libraries live in the venv's site-packages under nvidia/cu13/lib/, not in /usr/local/cuda/. The existing LD_LIBRARY_PATH logic only searched /usr/local/cuda* paths (which have CUDA 12.x), so the CUDA backend failed to load silently and llama-server fell back to CPU -- even with -ngl -1. This adds a glob scan of the venv's nvidia package directories (cu*, cudnn, nvjitlink) to LD_LIBRARY_PATH before launching llama-server, matching where pip puts the CUDA runtime. Tested on Colab with RTX PRO 6000 Blackwell (CUDA 13.0, pip torch): before -- 3 MiB GPU, 0% util, CPU inference after -- 13317 MiB GPU, 77% util, full GPU inference Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| audio_codecs.py | ||
| defaults.py | ||
| inference.py | ||
| llama_cpp.py | ||
| orchestrator.py | ||
| tools.py | ||
| worker.py | ||