When llama-server is built with shared libs (setup.sh default), it needs libcudart.so.12 and other CUDA runtime libs. Add /usr/local/cuda/lib64 and targets path to LD_LIBRARY_PATH so the server starts correctly even when CUDA isn't on the system path. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| audio_codecs.py | ||
| defaults.py | ||
| inference.py | ||
| llama_cpp.py | ||
| llama_cpp_builder.py | ||
| orchestrator.py | ||
| worker.py | ||