ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

llama.cpp CUDA detection: handle dlopen-ed backend (split build layout)

Browse source 
Current llama.cpp ships the CUDA backend as a dynamically-loaded plugin
(libggml-cuda.so* next to the binary), NOT a load-time dependency, so
ldd llama-server | grep libggml-cuda is a false negative: it reports no
CUDA on a perfectly good CUDA build. That made both is_cuda_server()
(provision_llama_cuda.sh) and _have_cuda_llama_server() (setup.sh) force a
needless full rebuild every run.

Fix both: keep the ldd check (old monolithic builds) and additionally treat
the presence of libggml-cuda.so* beside the binary as the CUDA signal. A
CPU-only build has no such backend, so this stays correct for the CPU case.

Verified on an N1X/sm_121 WSL build: llama-server --list-devices shows
CUDA0 JMJWOA-Generic-GPU and serves on the GPU, while ldd lists no
libggml-cuda; the new check correctly returns CUDA-present.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Daniel Han 2026-06-03 03:16:06 -07:00
commit bb3676d2c8
2 changed files with 22 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

14
studio/scripts/provision_llama_cuda.sh
View file

@ -21,7 +21,19 @@ LLAMA_DIR="${UNSLOTH_LLAMA_CPP_PATH:-$HOME/.unsloth/llama.cpp}"
SERVER="$LLAMA_DIR/build/bin/llama-server"
log() { printf ' - %s\n' "$*"; }
is_cuda_server() { [ -x "$1" ] && ldd "$1" 2>/dev/null | grep -qi 'libggml-cuda'; }
# A llama-server is CUDA-capable in either of two build layouts:
# * old monolithic build -> libggml-cuda is a direct load-time dependency (ldd shows it)
# * current split build -> CUDA ships as a dlopen-ed backend plugin, libggml-cuda.so*,
# sitting next to the binary; ldd will NOT list it
# Checking only ldd (the old behaviour) is a false negative on current llama.cpp and would
# force a pointless full rebuild every run. A CPU-only build has no libggml-cuda.so at all,
# so the presence of that backend beside the binary is the reliable CUDA signal.
is_cuda_server() {
[ -x "$1" ] || return 1
ldd "$1" 2>/dev/null | grep -qi 'libggml-cuda' && return 0
for _so in "$(dirname "$1")"/libggml-cuda.so*; do [ -e "$_so" ] && return 0; done
return 1
}
# 0. Already provisioned?
if is_cuda_server "$SERVER"; then

10
studio/setup.sh
View file

@ -1410,8 +1410,16 @@ fi # end _SKIP_GGUF_BUILD check
# already built a CUDA server are byte-for-byte unaffected. Opt out with
# UNSLOTH_NO_LLAMA_CUDA=1. Best-effort: never aborts setup (provision script always
# exits 0; failures leave the prior CPU/degraded state for the fallback below).
# CUDA-capable in either build layout: old monolithic (libggml-cuda is a direct
# ldd dependency) or current split build (CUDA is a dlopen-ed backend, libggml-cuda.so*,
# beside the binary -- ldd will NOT list it). Checking only ldd is a false negative on
# current llama.cpp and would force a needless rebuild; a CPU-only build has no
# libggml-cuda.so at all, so its presence beside the binary is the reliable signal.
_have_cuda_llama_server() {
[ -x "$LLAMA_SERVER_BIN" ] && ldd "$LLAMA_SERVER_BIN" 2>/dev/null | grep -qi 'libggml-cuda'
[ -x "$LLAMA_SERVER_BIN" ] || return 1
ldd "$LLAMA_SERVER_BIN" 2>/dev/null | grep -qi 'libggml-cuda' && return 0
for _so in "$(dirname "$LLAMA_SERVER_BIN")"/libggml-cuda.so*; do [ -e "$_so" ] && return 0; done
return 1
}
if [ "$_HOST_SYSTEM" = "Linux" ] \
&& { [ "$_HOST_MACHINE" = "aarch64" ] || [ "$_HOST_MACHINE" = "arm64" ]; } \