ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

studio: add CUDA lib paths to LD_LIBRARY_PATH for llama-server

Browse source 
When llama-server is built with shared libs (setup.sh default),
it needs libcudart.so.12 and other CUDA runtime libs. Add
/usr/local/cuda/lib64 and targets path to LD_LIBRARY_PATH so
the server starts correctly even when CUDA isn't on the system path.
This commit is contained in:
Daniel Han 2026-03-17 01:12:55 +00:00
commit 67f0ab592e
1 changed files with 10 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

11
studio/backend/core/inference/llama_cpp.py
View file

@ -888,9 +888,18 @@ class LlamaCppBackend:
env["PATH"] = ";".join(path_dirs) + ";" + existing_path
else:
# Linux: set LD_LIBRARY_PATH for shared libs next to the binary
# and CUDA runtime libs (libcudart, libcublas, etc.)
lib_dirs = [binary_dir]
for cuda_lib in [
"/usr/local/cuda/lib64",
"/usr/local/cuda/targets/x86_64-linux/lib",
]:
if os.path.isdir(cuda_lib):
lib_dirs.append(cuda_lib)
existing_ld = env.get("LD_LIBRARY_PATH", "")
new_ld = ":".join(lib_dirs)
env["LD_LIBRARY_PATH"] = (
f"{binary_dir}:{existing_ld}" if existing_ld else binary_dir
f"{new_ld}:{existing_ld}" if existing_ld else new_ld
)
# Pin to selected GPU(s) via CUDA_VISIBLE_DEVICES