Source llama.cpp prebuilts from unslothai/llama.cpp (CUDA, ROCm, macOS) (#5963)
* Studio: route arm64 Linux CUDA hosts to linux-arm64-cuda prebuilts
* Studio: SM-aware selection for windows-cuda app bundles
* Studio: select published ROCm bundles by gfx target (linux + windows)
* Studio: route macOS installs to the fork's prebuilt bundles
* Studio: fix windows cuda13 driver-13.0 gate and ROCm gfx prefix overreach
* Fix Blackwell Windows pin shadowing native app-bundle (b9360 over b9457)
* Match Windows cuda12 driver floor to Linux (12.x minor-version compat)
* Fix Windows app-bundle dropped when runtime DLLs come from torch/lib
* Fold the manifest resolver into the simple-path resolver (one entry, no dormant full path)
* Remove unused UNSLOTH_LLAMA_PUBLISHED_REPO override
* Route Windows GPU hosts to the fork prebuilts in setup.ps1
* Document sm_103 path divergence and mark --simple-policy as a no-op
* Note sm_103 coverage now comes from the producer manifest
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove the now-vestigial --simple-policy flag (one resolver handles all hosts)
* Unify the fork onto the manifest path; drop the linux-x64 filename path and hardcoded coverage tables
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip whitespace from manifest gfx_target/mapped_targets when parsing
* Windows CUDA: sort coverage-unknown bundles last so they can't outrank targeted ones
* Share the SM-coverage sort key between the linux and windows selectors via _sm_range
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reject approved releases with an exact source archive but no source repo to clone from
* Studio: accept the fork's windows-rocm kind in the Windows reinstall check
* Studio: accept a manifest-bundle source repo in the exact-source release check
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: route Linux hosts to the fork only when a usable GPU is present
* Fix Windows AMD lemonade tag resolution for PR #5963
The fork release scan passes each scanned release's upstream tag
(b9518, ...) to the lemonade lookup, but lemonade publishes its own tag
series (b1292, ...) that never contains upstream tag numbers. On a
Windows AMD host every scanned release therefore 404s the lemonade
fetch twice, the upstream HIP zip is dropped by the approved-hash gate,
and the scan walks the whole release history until it dies on the
unauthenticated GitHub rate limit or falls to a HIP source build. The
Linux path already passes the requested tag ("latest") and works.
Thread the requested tag through resolve_release_asset_choice ->
resolve_asset_choice -> resolve_upstream_asset_choice as lemonade_tag,
used only by the lemonade lookups. Upstream asset names keep the
concrete per-release tag and all new parameters default to the old
behavior.
Verified on a gfx1151 box: before, the native Windows install scanned
b9518..b8811 and aborted on rate limit; after, it selects
llama-b1292-windows-rocm-gfx1151-x64.zip (lemonade) from fork release
b9518, passes staged validation, and the installed llama-server
enumerates ROCm0. WSL keeps selecting the matching ubuntu bundle.
Adds a regression test pinning that the Windows fork path resolves
lemonade via /releases/latest, never /releases/tags/<fork-tag>.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Plan lemonade for Linux ROCm hosts on the ggml-org direct path for PR #5963
Audit follow-up to 72f32364 across the other selection pathways. The
ggml-org direct planner kept its lemonade attempt for Windows ROCm
hosts but planned only the CPU tarball for Linux ROCm hosts, so an AMD
Linux box routed to ggml-org (for example a --published-repo override)
silently installed the CPU build. That lemonade planning used to live
in the --simple-policy dispatcher this PR removed.
Add the lemonade attempt ahead of the CPU tarball in the Linux x86_64
branch, mirroring the Windows branch, with the lookup keyed to the
requested tag. Adds a regression test asserting lemonade is the first
attempt for a Linux ROCm host on the direct path.
Also re-verified the other pathways on a gfx1151 box: the fork-routed
flows pass the requested tag everywhere, repeat runs over an existing
lemonade install correctly skip with "already matches selected release
b9518" on both native Windows and WSL, and macOS, CUDA and CPU
selection are untouched. Suites: 328 passed on Linux, Windows matches
the pre-existing baseline.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
This commit is contained in:
oobabooga
GitHub
parent
8bca7bcfc9
commit
cc1a724efc
8 changed files with 1311 additions and 1060 deletions
|
|
@ -812,6 +812,7 @@ if ! command -v rocminfo >/dev/null 2>&1 && [ -x /opt/rocm/bin/rocminfo ]; then
|
|||
PATH="$PATH:/opt/rocm/bin"
|
||||
fi
|
||||
_setup_amd_detected=false
|
||||
_setup_nvidia_usable=false
|
||||
_setup_gfx_all=""
|
||||
_setup_mkt=""
|
||||
if command -v rocminfo >/dev/null 2>&1 && \
|
||||
|
|
@ -832,6 +833,7 @@ fi
|
|||
|
||||
if command -v nvidia-smi >/dev/null 2>&1 && \
|
||||
nvidia-smi -L 2>/dev/null | awk '/^GPU[[:space:]]+[0-9]+:/{found=1} END{exit !found}'; then
|
||||
_setup_nvidia_usable=true
|
||||
step "gpu" "NVIDIA GPU detected"
|
||||
elif [ "$_setup_amd_detected" = true ]; then
|
||||
_setup_vis="${HIP_VISIBLE_DEVICES:-${ROCR_VISIBLE_DEVICES:-}}"
|
||||
|
|
@ -910,34 +912,46 @@ _HOST_SYSTEM="$(uname -s 2>/dev/null || true)"
|
|||
_HOST_MACHINE="$(uname -m 2>/dev/null || true)"
|
||||
|
||||
# Pick the release repo install_llama_prebuilt.py plans against.
|
||||
# unslothai/llama.cpp ships only Linux CUDA bundles, so CPU-only Linux
|
||||
# x86_64 routes to ggml-org for bin-ubuntu-x64.tar.gz. Anything with a
|
||||
# GPU tool installed stays on unslothai (CUDA bundle / ROCm source build).
|
||||
# The fork ships CUDA (Linux x64/arm64, Windows), ROCm (Linux/Windows) and
|
||||
# macOS bundles. Only the plain CPU/Vulkan bundles still come from ggml-org, so
|
||||
# CPU-only Linux (x86_64 and arm64) routes there; GPU Linux, Windows and macOS
|
||||
# use unslothai.
|
||||
_LINUX_HAS_GPU=false
|
||||
for _GPU_TOOL in nvidia-smi rocminfo amd-smi hipconfig hipinfo; do
|
||||
if command -v "$_GPU_TOOL" >/dev/null 2>&1; then
|
||||
_LINUX_HAS_GPU=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
# Route to the fork only for a usable GPU. NVIDIA counts only when a device is
|
||||
# actually enumerated (_setup_nvidia_usable, from the nvidia-smi -L probe above)
|
||||
# AND not hidden via CUDA_VISIBLE_DEVICES=-1 -- mirroring install_llama_prebuilt.py's
|
||||
# has_usable_nvidia. Mere nvidia-smi presence (CPU-only CUDA-toolkit containers,
|
||||
# broken drivers) or a hidden GPU therefore takes the ggml-org CPU prebuilt
|
||||
# instead of a slow source build. AMD is deliberately left on tooling presence,
|
||||
# not usability: an unusable NVIDIA host has a good CPU prebuilt to fall back to,
|
||||
# whereas tightening AMD would regress ROCm hosts exposing only hipconfig/hipinfo
|
||||
# into an unnecessary CPU build.
|
||||
if [ "$_setup_nvidia_usable" = true ] && [ "${CUDA_VISIBLE_DEVICES:-}" != "-1" ]; then
|
||||
_LINUX_HAS_GPU=true
|
||||
else
|
||||
for _GPU_TOOL in rocminfo amd-smi hipconfig hipinfo; do
|
||||
if command -v "$_GPU_TOOL" >/dev/null 2>&1; then
|
||||
_LINUX_HAS_GPU=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
if [ "$_HOST_SYSTEM" = "Darwin" ]; then
|
||||
_HELPER_RELEASE_REPO="ggml-org/llama.cpp"
|
||||
elif [ "$_HOST_SYSTEM" = "Linux" ] \
|
||||
if [ "$_HOST_SYSTEM" = "Linux" ] \
|
||||
&& [ "$_HOST_MACHINE" = "x86_64" ] \
|
||||
&& [ "$_LINUX_HAS_GPU" = false ]; then
|
||||
_HELPER_RELEASE_REPO="ggml-org/llama.cpp"
|
||||
elif [ "$_HOST_SYSTEM" = "Linux" ] \
|
||||
&& { [ "$_HOST_MACHINE" = "aarch64" ] || [ "$_HOST_MACHINE" = "arm64" ]; } \
|
||||
&& [ "$_LINUX_HAS_GPU" = false ]; then
|
||||
# Linux ARM64 (Ampere Altra, Raspberry Pi 5, GitHub `ubuntu-24.04-arm`,
|
||||
# CPU-only Jetson rescue mode, ...). unslothai/llama.cpp only ships
|
||||
# the Linux CUDA bundles, so without this branch the prebuilt
|
||||
# resolver returns 0 attempts on every release and the installer
|
||||
# falls all the way back to a source build. Upstream ggml-org ships
|
||||
# CPU-only Linux ARM64 (Ampere Altra, Raspberry Pi 5, GitHub
|
||||
# `ubuntu-24.04-arm`, CPU-only Jetson rescue mode, ...). The fork ships no
|
||||
# arm64 CPU bundle, so without this branch the prebuilt resolver returns 0
|
||||
# attempts and the installer falls back to a source build. ggml-org ships
|
||||
# llama-bNNNN-bin-ubuntu-arm64.tar.gz from at least b9072 onward.
|
||||
_HELPER_RELEASE_REPO="ggml-org/llama.cpp"
|
||||
else
|
||||
# GPU Linux (x64 CUDA/ROCm, arm64 CUDA), Windows (CUDA/ROCm), and macOS.
|
||||
_HELPER_RELEASE_REPO="unslothai/llama.cpp"
|
||||
fi
|
||||
unset _GPU_TOOL
|
||||
|
|
@ -1000,7 +1014,6 @@ else
|
|||
--install-dir "$LLAMA_CPP_DIR"
|
||||
--llama-tag "$_REQUESTED_LLAMA_TAG"
|
||||
--published-repo "$_HELPER_RELEASE_REPO"
|
||||
--simple-policy
|
||||
)
|
||||
if [ -n "${UNSLOTH_LLAMA_RELEASE_TAG:-}" ]; then
|
||||
_PREBUILT_CMD+=(--published-release-tag "$UNSLOTH_LLAMA_RELEASE_TAG")
|
||||
|
|
@ -1555,7 +1568,6 @@ if [ "$_LLAMA_CPP_DEGRADED" = true ] \
|
|||
--install-dir "$LLAMA_CPP_DIR"
|
||||
--llama-tag "$_REQUESTED_LLAMA_TAG"
|
||||
--published-repo "ggml-org/llama.cpp"
|
||||
--simple-policy
|
||||
--cpu-fallback
|
||||
)
|
||||
# Trust the installer's exit code: it validates the server before exiting 0,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue