ae2b1b97baThe prebuilt llama.cpp binary (cuda13-newer) links against libcudart.so.13 and libcublas.so.13. When torch is installed via pip, these libraries live in the venv's site-packages under nvidia/cu13/lib/, not in /usr/local/cuda/. The existing LD_LIBRARY_PATH logic only searched /usr/local/cuda* paths (which have CUDA 12.x), so the CUDA backend failed to load silently and llama-server fell back to CPU -- even with -ngl -1. This adds a glob scan of the venv's nvidia package directories (cu*, cudnn, nvjitlink) to LD_LIBRARY_PATH before launching llama-server, matching where pip puts the CUDA runtime. Tested on Colab with RTX PRO 6000 Blackwell (CUDA 13.0, pip torch): before -- 3 MiB GPU, 0% util, CPU inference after -- 13317 MiB GPU, 77% util, full GPU inference Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| assets | ||
| auth | ||
| core | ||
| loggers | ||
| models | ||
| plugins | ||
| requirements | ||
| routes | ||
| state | ||
| storage | ||
| tests | ||
| utils | ||
| __init__.py | ||
| _platform_compat.py | ||
| colab.py | ||
| main.py | ||
| run.py | ||