ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: download paired cudart bundle on Windows CUDA installs (#5322)

Browse source 
* Studio: download paired cudart bundle on Windows CUDA installs

Upstream ggml-org/llama.cpp publishes Windows CUDA in two archives
that the release notes explicitly say are both required:

  llama-<tag>-bin-win-cuda-X.Y-x64.zip       (binaries + ggml DLLs)
  cudart-llama-bin-win-cuda-X.Y-x64.zip      (cudart64, cublas64, cublasLt64)

Studio's installer was downloading only the first one. The
``runtime_name`` / ``runtime_url`` fields on AssetChoice existed but
were never populated, and ``install_from_archives`` only handled
``choice.url``. With the cudart DLLs missing from
``install_dir/build/bin/Release``, the prebuilt binary's LoadLibrary
calls only resolved at runtime when the user happened to have a
version-matched system CUDA toolkit on PATH. That is the underlying
cause for the Windows reports in #5106 ("GPU detected but model
loaded entirely on RAM"): the prebuilt's CUDA backend silently fails
to load and llama-server falls back to CPU regardless of ``-ngl`` or
``--fit on``.

Wires the pairing through end to end:

* ``windows_cuda_attempts`` and ``published_windows_cuda_attempts``
  look up the matching ``cudart-llama-bin-win-cuda-X.Y-x64.zip``
  asset URL alongside the main archive and store it as
  ``runtime_url`` / ``runtime_name`` on the AssetChoice. We only
  pair when the selected main archive is the binary archive
  (``llama-...zip``) so the legacy cudart-only naming path is
  unaffected.

* ``apply_approved_hashes`` resolves the runtime archive's hash from
  the approved manifest. If the manifest does not list the runtime
  archive, the pairing is dropped rather than installing without
  checksum coverage. Preserves the supply-chain guarantee for
  published bundles; upstream installs with no manifest are
  unaffected (same risk surface as the existing main-archive
  download).

* ``install_from_archives`` now downloads the runtime archive into a
  separate temp dir and runs ``copy_globs`` against both source dirs.
  Separate dirs avoid the "ambiguous archive layout" guard tripping
  on shared filenames like LICENSE.txt, while the second
  ``copy_globs`` overlay drops the cudart DLLs into the same
  ``install_dir/build/bin/Release`` directory as the main binary.

Adds a ``runtime_sha256`` field on AssetChoice to carry the
verified hash through to the download step, alongside the existing
``runtime_name`` / ``runtime_url`` slots.

Tests: 5 new cases in tests/studio/install/test_selection_logic.py:
* upstream pairing populates runtime_url / runtime_name
* graceful degrade when cudart asset is absent in the release
* legacy cudart-only naming path does not self-pair
* apply_approved_hashes threads runtime_sha256 when the manifest
  lists it
* apply_approved_hashes drops the pair when the runtime hash is
  missing rather than installing without verification

130 install tests pass (125 baseline + 5 new). No regressions.

Refs #5106

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Trim comments to be more succinct

* Studio: refresh installs that pre-date the paired cudart bundle

expected_install_fingerprint did not hash the new runtime_name /
runtime_sha256 fields, and runtime_payload_health_groups for windows-
cuda only checked llama.dll / ggml-cuda.dll. The combination meant that
an install made before this PR -- the exact installs reporting #5106 --
would still match the post-PR choice: same main asset name + sha, same
llama.dll, same ggml-cuda.dll, missing cudart64_*.dll, but
existing_install_matches_choice returned True and the cudart download
path in install_from_archives never ran. Fresh installs got the fix;
existing affected installs did not.

This commit:
 * Adds runtime_asset and runtime_sha256 to the fingerprint payload so
   any change to (or first introduction of) the cudart pair invalidates
   pre-existing installs.
 * Refactors write_prebuilt_metadata to call expected_install_fingerprint
   so the recorded fingerprint cannot drift from the expected one when
   new keys are added.
 * Extends runtime_payload_health_groups for windows-cuda to require
   cudart64_*.dll and cublas64_*.dll *only when the choice carries a
   paired runtime archive*. Gating on choice.runtime_name keeps the
   no-pair fallback path (manifest missing cudart hash, upstream
   without paired bundle) from looping on reinstall.

New tests:
 * test_existing_install_matches_plan_windows_cuda_paired_requires_cudart
   -- paired choice rejects installs missing cudart / cublas.
 * test_existing_install_matches_plan_windows_cuda_unpaired_skips_cudart_check
   -- unpaired choice still accepts legacy cudart-less installs.
 * test_existing_install_fingerprint_changes_when_cudart_pair_added
   -- direct fingerprint mismatch between the legacy and paired choice.

Refs #5106

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: tighten paired Windows CUDA install gates

Three follow-ups from a 12-reviewer batch over 526894a4 (PR #5322):

1. (12/12) Health check required cudart64_*.dll and cublas64_*.dll but
   not cublasLt64_*.dll. The upstream cudart-llama-bin-win-cuda-X.Y-x64
   bundle ships all three (verified against b9103 cuda-12.4 and
   cuda-13.1: 3 DLLs, no executables), and a Windows install missing
   any one of them still fails CUDA initialisation. Adding
   cublasLt64_*.dll to runtime_payload_health_groups so a partial
   install or a deletion of the third DLL triggers reinstall instead
   of silently staying broken.

2. The runtime overlay copy used the same broad runtime_patterns_for_choice
   set as the main archive (windows-cuda returns *.exe and *.dll). A
   malformed runtime zip that contained a llama-server.exe alongside
   the real cudart DLLs would have overwritten the main archive's
   server binary. Introduced paired_runtime_dll_patterns() that
   returns the cudart bundle's three specific filename patterns and
   nothing else, and use that for the second copy_globs pass.
   New end-to-end regression test packs a fake runtime zip with an
   extra llama-server.exe and asserts the main binary survives.

3. (7/12) python_runtime_dirs in install_llama_prebuilt.py and
   _windows_pip_nvidia_dll_dirs in llama_cpp.py walked different path
   sets. The installer side missed nvidia/<pkg>/Library/bin (conda
   layout) and nvidia/<pkg>/bin/x86_64 (current CUDA 13 unsuffixed
   wheel layout), so preflight CUDA detection could fail even when
   usable DLLs were present. Mirrored the same six-path set the
   backend resolver uses, including arch subdirs.

New tests:
 - test_paired_runtime_dll_patterns_excludes_executables
 - test_runtime_overlay_cannot_overwrite_main_archive_payload (end-to-end)
 - test_python_runtime_dirs_covers_cu13_and_library_bin
 - extended test_existing_install_matches_plan_windows_cuda_paired_requires_cudart
   with a cublasLt-missing case

Upstream cudart bundle contents verified empirically by downloading
the b9103 release artifacts directly: each cuda-X.Y bundle contains
exactly cudart64_X.dll + cublas64_X.dll + cublasLt64_X.dll, no exes.

Refs #5106

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-05-11 05:42:05 -07:00 • committed by GitHub
commit e346193ae8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 803 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

196
studio/install_llama_prebuilt.py
View file

@ -205,8 +205,12 @@ class AssetChoice:
name: str
url: str
source_label: str
# Paired runtime archive (Windows CUDA cudart bundle). When set,
# install_from_archives also downloads it and overlays its DLLs on
# top of the main install. See unslothai/unsloth#5106.
runtime_name: str | None = None
runtime_url: str | None = None
runtime_sha256: str | None = None
is_ready_bundle: bool = False
install_kind: str = ""
bundle_profile: str | None = None
@ -2922,6 +2926,30 @@ def windows_cuda_attempts(
+ ",".join(windows_cuda_upstream_asset_names(llama_tag, runtime))
)
continue
# Pair the cudart bundle when upstream ships it. Without this
# the binary needs a system CUDA toolkit on PATH at runtime
# (#5106). Only pair when the selected main archive is the
# binary archive, not the cudart archive itself.
runtime_archive_name: str | None = None
runtime_archive_url: str | None = None
if selected_name.startswith(f"llama-"):
cudart_name = f"cudart-llama-bin-win-cuda-{runtime}-x64.zip"
cudart_url = upstream_assets.get(cudart_name)
if cudart_url and cudart_url != asset_url:
runtime_archive_name = cudart_name
runtime_archive_url = cudart_url
attempt_log = list(selection_log) + [
f"windows_cuda_selection: selected {selected_name} runtime={runtime}"
]
if runtime_archive_name:
attempt_log.append(
f"windows_cuda_selection: paired runtime archive {runtime_archive_name}"
)
else:
attempt_log.append(
"windows_cuda_selection: no paired runtime archive found; "
"binary will rely on a system CUDA toolkit at runtime"
)
attempts.append(
AssetChoice(
repo = UPSTREAM_REPO,
@ -2931,10 +2959,9 @@ def windows_cuda_attempts(
source_label = "upstream",
install_kind = "windows-cuda",
runtime_line = runtime_line,
selection_log = list(selection_log)
+ [
f"windows_cuda_selection: selected {selected_name} runtime={runtime}"
],
runtime_name = runtime_archive_name,
runtime_url = runtime_archive_url,
selection_log = attempt_log,
)
)
return attempts
@ -2982,6 +3009,24 @@ def published_windows_cuda_attempts(
asset_url = release.assets.get(artifact.asset_name)
if not asset_url:
continue
# See windows_cuda_attempts: pair the cudart bundle.
runtime_archive_name: str | None = None
runtime_archive_url: str | None = None
if artifact.asset_name.startswith("llama-"):
runtime = runtime_by_line[runtime_line]
cudart_name = f"cudart-llama-bin-win-cuda-{runtime}-x64.zip"
cudart_url = release.assets.get(cudart_name)
if cudart_url and cudart_url != asset_url:
runtime_archive_name = cudart_name
runtime_archive_url = cudart_url
attempt_log = list(ordered_attempt.selection_log or []) + [
"windows_cuda_selection: selected published asset "
f"{artifact.asset_name} for runtime_line={runtime_line}"
]
if runtime_archive_name:
attempt_log.append(
f"windows_cuda_selection: paired published runtime archive {runtime_archive_name}"
)
attempts.append(
AssetChoice(
repo = release.repo,
@ -2991,11 +3036,9 @@ def published_windows_cuda_attempts(
source_label = "published",
install_kind = "windows-cuda",
runtime_line = runtime_line,
selection_log = list(ordered_attempt.selection_log or [])
+ [
"windows_cuda_selection: selected published asset "
f"{artifact.asset_name} for runtime_line={runtime_line}"
],
runtime_name = runtime_archive_name,
runtime_url = runtime_archive_url,
selection_log = attempt_log,
)
)
break
@ -3701,6 +3744,17 @@ def overlay_directory_for_choice(
return path
def paired_runtime_dll_patterns(choice: AssetChoice) -> list[str]:
"""Filename patterns the paired runtime archive is allowed to drop
into the install. Used for the second copy_globs pass in
install_from_archives, narrower than runtime_patterns_for_choice so
the runtime archive cannot overwrite main-archive payload like
llama-server.exe. Only Windows CUDA has paired runtimes today."""
if choice.install_kind == "windows-cuda":
return ["cudart64_*.dll", "cublas64_*.dll", "cublasLt64_*.dll"]
return []
def runtime_patterns_for_choice(choice: AssetChoice) -> list[str]:
if choice.install_kind in {"linux-cpu", "linux-cuda", "linux-rocm"}:
return [
@ -4020,14 +4074,52 @@ def install_from_archives(
install_dir.mkdir(parents = True, exist_ok = True)
extract_dir = Path(tempfile.mkdtemp(prefix = "extract-", dir = work_dir))
runtime_extract_dir: Path | None = None
try:
extract_archive(main_archive, extract_dir)
# Download the paired runtime archive into its own temp dir to
# avoid copy_globs's ambiguous-layout guard on shared names
# like LICENSE.txt. Two passes of copy_globs land both archives
# in the same overlay dir. Fixes #5106.
if choice.runtime_url and choice.runtime_name:
runtime_archive = work_dir / choice.runtime_name
log(
f"downloading paired runtime archive {choice.runtime_name} "
f"from {choice.source_label} release"
)
download_file_verified(
choice.runtime_url,
runtime_archive,
expected_sha256 = choice.runtime_sha256,
label = f"prebuilt runtime archive {choice.runtime_name}",
)
runtime_extract_dir = Path(
tempfile.mkdtemp(prefix = "extract-runtime-", dir = work_dir)
)
extract_archive(runtime_archive, runtime_extract_dir)
source_dir = extract_dir
overlay_dir = overlay_directory_for_choice(install_dir, choice, host)
copy_globs(
source_dir, overlay_dir, runtime_patterns_for_choice(choice), required = True
)
if runtime_extract_dir is not None:
# The runtime archive only contributes the CUDA DLLs.
# Restrict the overlay to the cudart bundle's known
# filenames (cudart64_X.dll / cublas64_X.dll /
# cublasLt64_X.dll) rather than the broad ``*.exe`` /
# ``*.dll`` set from runtime_patterns_for_choice, so a
# malformed runtime archive can never overwrite
# llama-server.exe or other main-archive payload. The
# upstream cudart-llama-bin-win-cuda-X.Y-x64.zip currently
# ships exactly these three DLLs (verified against b9103
# cuda-12.4 and cuda-13.1 bundles).
copy_globs(
runtime_extract_dir,
overlay_dir,
paired_runtime_dll_patterns(choice),
required = False,
)
copy_globs(
source_dir,
install_dir,
@ -4036,6 +4128,8 @@ def install_from_archives(
)
finally:
remove_tree(extract_dir)
if runtime_extract_dir is not None:
remove_tree(runtime_extract_dir)
if host.is_windows:
exec_dir = install_dir / "build" / "bin" / "Release"
@ -4239,8 +4333,27 @@ def python_runtime_dirs() -> list[str]:
for root in search_roots:
if not root.is_dir():
continue
# ``nvidia/<pkg>/lib`` -- Linux convention; harmless on Windows
# where the directory simply does not exist on real wheels.
candidates.extend(root.glob("nvidia/*/lib"))
# ``nvidia/<pkg>/bin`` -- legacy modular Windows wheels
# (``nvidia-cuda-runtime-cu12``, ``nvidia-cublas-cu12``).
candidates.extend(root.glob("nvidia/*/bin"))
# ``nvidia/<pkg>/bin/x86_64`` and ``.../bin/x64`` -- current
# CUDA 13 Windows wheel layout (the unsuffixed
# ``nvidia-cuda-runtime`` 13.x and ``nvidia-cublas`` 13.x
# packages ship under ``nvidia/cu13/bin/x86_64/cudart64_13.dll``).
# Without these, Windows preflight CUDA detection misses cu13
# installs and falls back to the upstream cudart bundle path
# even when usable DLLs are already on disk (#5106). Kept in
# sync with the backend resolver
# ``llama_cpp.LlamaCppBackend._windows_pip_nvidia_dll_dirs``.
candidates.extend(root.glob("nvidia/*/bin/x86_64"))
candidates.extend(root.glob("nvidia/*/bin/x64"))
# ``nvidia/<pkg>/Library/bin`` -- conda-style wheel repacks.
candidates.extend(root.glob("nvidia/*/Library/bin"))
candidates.extend(root.glob("nvidia/*/Library/bin/x86_64"))
candidates.extend(root.glob("nvidia/*/Library/bin/x64"))
candidates.extend(root.glob("torch/lib"))
return dedupe_existing_dirs(candidates)
@ -4743,6 +4856,17 @@ def apply_approved_hashes(
missing_assets.append(attempt.name)
continue
attempt.expected_sha256 = approved.sha256
# Resolve the paired runtime archive's hash too. Drop the pair
# if the manifest does not list it -- never install an
# unverified archive.
if attempt.runtime_name and attempt.runtime_url:
runtime_approved = checksums.artifacts.get(attempt.runtime_name)
if runtime_approved is None:
attempt.runtime_name = None
attempt.runtime_url = None
attempt.runtime_sha256 = None
else:
attempt.runtime_sha256 = runtime_approved.sha256
approved_attempts.append(attempt)
if not approved_attempts:
missing_text = ", ".join(missing_assets) if missing_assets else "none"
@ -4906,24 +5030,19 @@ def write_prebuilt_metadata(
approved_checksums,
llama_tag,
)
fingerprint_payload = {
"published_repo": approved_checksums.repo,
"release_tag": release_tag,
"upstream_tag": llama_tag,
"asset": choice.name,
"asset_sha256": choice.expected_sha256,
"source": choice.source_label,
"source_asset": source_asset_name,
"source_sha256": source_sha256,
"runtime_line": choice.runtime_line,
"bundle_profile": choice.bundle_profile,
"coverage_class": choice.coverage_class,
}
fingerprint = hashlib.sha256(
json.dumps(fingerprint_payload, sort_keys = True, separators = (",", ":")).encode(
"utf-8"
)
).hexdigest()
# expected_install_fingerprint is the source of truth for what the
# fingerprint must contain. Calling it here -- instead of inlining a
# parallel payload -- prevents drift where new keys (e.g. the cudart
# pair fields added for #5106) are added to one side but not the
# other, which would cause every install to look stale.
fingerprint = expected_install_fingerprint(
llama_tag = llama_tag,
release_tag = release_tag,
choice = choice,
approved_checksums = approved_checksums,
)
if fingerprint is None:
raise PrebuiltFallback(f"cannot compute install fingerprint for {choice.name}")
metadata = {
"requested_tag": requested_tag,
"tag": llama_tag,
@ -4974,6 +5093,14 @@ def expected_install_fingerprint(
"source_asset": source_asset_name,
"source_sha256": source_sha256,
"runtime_line": choice.runtime_line,
# Including the paired runtime archive (Windows cudart bundle)
# in the fingerprint is what forces existing #5106 installs to
# refresh: pre-PR installs hashed nothing in this slot, post-PR
# paired installs hash the cudart sha. Without these two keys
# an existing cudart-less install would keep matching the new
# choice and never re-overlay the cudart DLLs.
"runtime_asset": choice.runtime_name,
"runtime_sha256": choice.runtime_sha256,
"bundle_profile": choice.bundle_profile,
"coverage_class": choice.coverage_class,
}
@ -5034,7 +5161,20 @@ def runtime_payload_health_groups(choice: AssetChoice) -> list[list[str]]:
if choice.install_kind == "windows-cpu":
return [["llama.dll"]]
if choice.install_kind == "windows-cuda":
return [["llama.dll"], ["ggml-cuda.dll"]]
groups = [["llama.dll"], ["ggml-cuda.dll"]]
# When the cudart bundle was paired in (#5106) require all
# three of its DLLs alongside the main archive's payload.
# install_kind alone is not enough -- legacy installs without
# the cudart pair must still pass the health check on the
# no-pair fallback path, otherwise pair-less builds would loop
# on reinstall forever. The upstream cudart bundle ships
# cudart64_X.dll + cublas64_X.dll + cublasLt64_X.dll; missing
# any one of them still breaks GPU initialisation.
if choice.runtime_name:
groups.append(["cudart64_*.dll"])
groups.append(["cublas64_*.dll"])
groups.append(["cublasLt64_*.dll"])
return groups
if choice.install_kind == "windows-hip":
return [["llama.dll"], ["*hip*.dll"]]
return []

516
tests/studio/install/test_install_llama_prebuilt_logic.py
View file

@ -769,7 +769,11 @@ def write_linux_install_shape(install_dir: Path) -> None:
def write_windows_install_shape(
install_dir: Path, *, include_llama_dll: bool = True, include_cuda_dll: bool = False
install_dir: Path,
*,
include_llama_dll: bool = True,
include_cuda_dll: bool = False,
include_cudart_dlls: bool = False,
) -> None:
runtime_dir = install_dir / "build" / "bin" / "Release"
runtime_dir.mkdir(parents = True, exist_ok = True)
@ -779,6 +783,11 @@ def write_windows_install_shape(
(runtime_dir / "llama.dll").write_bytes(b"DLL")
if include_cuda_dll:
(runtime_dir / "ggml-cuda.dll").write_bytes(b"DLL")
if include_cudart_dlls:
# cudart bundle DLLs that ship in cudart-llama-bin-win-cuda-*-x64.zip
(runtime_dir / "cudart64_12.dll").write_bytes(b"DLL")
(runtime_dir / "cublas64_12.dll").write_bytes(b"DLL")
(runtime_dir / "cublasLt64_12.dll").write_bytes(b"DLL")
(install_dir / "convert_hf_to_gguf.py").write_text(
"#!/usr/bin/env python3\n", encoding = "utf-8"
)
@ -1153,6 +1162,330 @@ def test_existing_install_matches_plan_windows_cuda_requires_cuda_dll(tmp_path:
assert existing_install_matches_plan(install_dir, host, plan) is False
def test_existing_install_matches_plan_windows_cuda_paired_requires_cudart(
tmp_path: Path,
):
"""When the choice ships a paired cudart bundle (#5106), the install
is considered stale unless cudart64_*.dll and cublas64_*.dll are
actually on disk. Otherwise existing broken installs would keep
matching and skip the reinstall that drops cudart in."""
install_dir = tmp_path / "llama.cpp"
install_dir.mkdir()
write_windows_install_shape(
install_dir,
include_llama_dll = True,
include_cuda_dll = True,
include_cudart_dlls = True,
)
host = HostInfo(
system = "Windows",
machine = "AMD64",
is_windows = True,
is_linux = False,
is_macos = False,
is_x86_64 = True,
is_arm64 = False,
nvidia_smi = None,
driver_cuda_version = (12, 4),
compute_caps = [],
visible_cuda_devices = None,
has_physical_nvidia = False,
has_usable_nvidia = True,
)
choice = AssetChoice(
repo = "unslothai/llama.cpp",
tag = "release-1",
name = "llama-b9001-bin-win-cuda-12.4-x64.zip",
url = "https://example.com/x.zip",
source_label = "published",
install_kind = "windows-cuda",
runtime_line = "cuda12",
expected_sha256 = "a" * 64,
runtime_name = "cudart-llama-bin-win-cuda-12.4-x64.zip",
runtime_url = "https://example.com/cudart.zip",
runtime_sha256 = "c" * 64,
)
checksums = ApprovedReleaseChecksums(
repo = "unslothai/llama.cpp",
release_tag = "release-1",
upstream_tag = "b9001",
source_commit = "deadbeef",
artifacts = {
source_archive_logical_name("b9001"): ApprovedArtifactHash(
asset_name = source_archive_logical_name("b9001"),
sha256 = "b" * 64,
repo = "ggml-org/llama.cpp",
kind = "upstream-source",
),
choice.name: ApprovedArtifactHash(
asset_name = choice.name,
sha256 = choice.expected_sha256,
repo = "unslothai/llama.cpp",
kind = "prebuilt",
),
choice.runtime_name: ApprovedArtifactHash(
asset_name = choice.runtime_name,
sha256 = choice.runtime_sha256,
repo = "unslothai/llama.cpp",
kind = "prebuilt",
),
},
)
plan = INSTALL_LLAMA_PREBUILT.InstallReleasePlan(
requested_tag = "latest",
llama_tag = "b9001",
release_tag = "release-1",
attempts = [choice],
approved_checksums = checksums,
)
write_prebuilt_metadata(
install_dir,
requested_tag = "latest",
llama_tag = "b9001",
release_tag = "release-1",
choice = choice,
approved_checksums = checksums,
prebuilt_fallback_used = False,
)
# Fully populated install (main archive + cudart DLLs) matches.
assert existing_install_matches_plan(install_dir, host, plan) is True
# cublas missing -- stale, must reinstall.
(install_dir / "build" / "bin" / "Release" / "cublas64_12.dll").unlink()
assert existing_install_matches_plan(install_dir, host, plan) is False
# cudart missing -- stale, must reinstall.
write_windows_install_shape(
install_dir,
include_llama_dll = True,
include_cuda_dll = True,
include_cudart_dlls = True,
)
(install_dir / "build" / "bin" / "Release" / "cudart64_12.dll").unlink()
assert existing_install_matches_plan(install_dir, host, plan) is False
# cublasLt missing -- stale, must reinstall. The upstream cudart
# bundle ships all three of cudart / cublas / cublasLt; a user with
# cudart + cublas but no cublasLt is still missing a required GPU
# initialisation DLL and Studio must refresh the install.
write_windows_install_shape(
install_dir,
include_llama_dll = True,
include_cuda_dll = True,
include_cudart_dlls = True,
)
(install_dir / "build" / "bin" / "Release" / "cublasLt64_12.dll").unlink()
assert existing_install_matches_plan(install_dir, host, plan) is False
def test_existing_install_matches_plan_windows_cuda_unpaired_skips_cudart_check(
tmp_path: Path,
):
"""If the choice has no paired runtime archive (manifest dropped it,
or upstream did not ship cudart), legacy installs without cudart on
disk must still pass the health check -- otherwise the installer
would loop on reinstall forever because install_from_archives has no
cudart source to drop in."""
install_dir = tmp_path / "llama.cpp"
install_dir.mkdir()
write_windows_install_shape(
install_dir,
include_llama_dll = True,
include_cuda_dll = True,
include_cudart_dlls = False,
)
host = HostInfo(
system = "Windows",
machine = "AMD64",
is_windows = True,
is_linux = False,
is_macos = False,
is_x86_64 = True,
is_arm64 = False,
nvidia_smi = None,
driver_cuda_version = (12, 4),
compute_caps = [],
visible_cuda_devices = None,
has_physical_nvidia = False,
has_usable_nvidia = True,
)
choice = AssetChoice(
repo = "unslothai/llama.cpp",
tag = "release-1",
name = "llama-b9001-bin-win-cuda-12.4-x64.zip",
url = "https://example.com/x.zip",
source_label = "published",
install_kind = "windows-cuda",
runtime_line = "cuda12",
expected_sha256 = "a" * 64,
)
checksums = ApprovedReleaseChecksums(
repo = "unslothai/llama.cpp",
release_tag = "release-1",
upstream_tag = "b9001",
source_commit = "deadbeef",
artifacts = {
source_archive_logical_name("b9001"): ApprovedArtifactHash(
asset_name = source_archive_logical_name("b9001"),
sha256 = "b" * 64,
repo = "ggml-org/llama.cpp",
kind = "upstream-source",
),
choice.name: ApprovedArtifactHash(
asset_name = choice.name,
sha256 = choice.expected_sha256,
repo = "unslothai/llama.cpp",
kind = "prebuilt",
),
},
)
plan = INSTALL_LLAMA_PREBUILT.InstallReleasePlan(
requested_tag = "latest",
llama_tag = "b9001",
release_tag = "release-1",
attempts = [choice],
approved_checksums = checksums,
)
write_prebuilt_metadata(
install_dir,
requested_tag = "latest",
llama_tag = "b9001",
release_tag = "release-1",
choice = choice,
approved_checksums = checksums,
prebuilt_fallback_used = False,
)
assert existing_install_matches_plan(install_dir, host, plan) is True
def test_existing_install_fingerprint_changes_when_cudart_pair_added(
tmp_path: Path,
):
"""Existing pre-#5322 Windows CUDA installs (no paired cudart) must
be treated as stale once the choice gains a runtime archive,
otherwise the fingerprint match would keep skipping the reinstall
that drops the cudart DLLs in. This is the install-cache half of the
#5106 fix -- the health-check half lives in the test above."""
install_dir = tmp_path / "llama.cpp"
install_dir.mkdir()
write_windows_install_shape(
install_dir,
include_llama_dll = True,
include_cuda_dll = True,
include_cudart_dlls = False,
)
host = HostInfo(
system = "Windows",
machine = "AMD64",
is_windows = True,
is_linux = False,
is_macos = False,
is_x86_64 = True,
is_arm64 = False,
nvidia_smi = None,
driver_cuda_version = (12, 4),
compute_caps = [],
visible_cuda_devices = None,
has_physical_nvidia = False,
has_usable_nvidia = True,
)
legacy_choice = AssetChoice(
repo = "unslothai/llama.cpp",
tag = "release-1",
name = "llama-b9001-bin-win-cuda-12.4-x64.zip",
url = "https://example.com/x.zip",
source_label = "published",
install_kind = "windows-cuda",
runtime_line = "cuda12",
expected_sha256 = "a" * 64,
)
paired_choice = AssetChoice(
repo = "unslothai/llama.cpp",
tag = "release-1",
name = "llama-b9001-bin-win-cuda-12.4-x64.zip",
url = "https://example.com/x.zip",
source_label = "published",
install_kind = "windows-cuda",
runtime_line = "cuda12",
expected_sha256 = "a" * 64,
runtime_name = "cudart-llama-bin-win-cuda-12.4-x64.zip",
runtime_url = "https://example.com/cudart.zip",
runtime_sha256 = "c" * 64,
)
checksums = ApprovedReleaseChecksums(
repo = "unslothai/llama.cpp",
release_tag = "release-1",
upstream_tag = "b9001",
source_commit = "deadbeef",
artifacts = {
source_archive_logical_name("b9001"): ApprovedArtifactHash(
asset_name = source_archive_logical_name("b9001"),
sha256 = "b" * 64,
repo = "ggml-org/llama.cpp",
kind = "upstream-source",
),
legacy_choice.name: ApprovedArtifactHash(
asset_name = legacy_choice.name,
sha256 = legacy_choice.expected_sha256,
repo = "unslothai/llama.cpp",
kind = "prebuilt",
),
paired_choice.runtime_name: ApprovedArtifactHash(
asset_name = paired_choice.runtime_name,
sha256 = paired_choice.runtime_sha256,
repo = "unslothai/llama.cpp",
kind = "prebuilt",
),
},
)
# Install metadata was written for the legacy (no-pair) choice.
write_prebuilt_metadata(
install_dir,
requested_tag = "latest",
llama_tag = "b9001",
release_tag = "release-1",
choice = legacy_choice,
approved_checksums = checksums,
prebuilt_fallback_used = False,
)
# New plan offers the paired choice -- fingerprint must differ so
# the install is refreshed. The health check would also catch this
# because cudart64_*.dll is missing on disk; we test the fingerprint
# half explicitly by comparing the two fingerprints directly.
legacy_fingerprint = INSTALL_LLAMA_PREBUILT.expected_install_fingerprint(
llama_tag = "b9001",
release_tag = "release-1",
choice = legacy_choice,
approved_checksums = checksums,
)
paired_fingerprint = INSTALL_LLAMA_PREBUILT.expected_install_fingerprint(
llama_tag = "b9001",
release_tag = "release-1",
choice = paired_choice,
approved_checksums = checksums,
)
assert legacy_fingerprint != paired_fingerprint, (
"expected_install_fingerprint must hash runtime_name/runtime_sha256 "
"so pre-#5322 installs are not falsely considered up-to-date"
)
paired_plan = INSTALL_LLAMA_PREBUILT.InstallReleasePlan(
requested_tag = "latest",
llama_tag = "b9001",
release_tag = "release-1",
attempts = [paired_choice],
approved_checksums = checksums,
)
assert existing_install_matches_plan(install_dir, host, paired_plan) is False
def test_existing_install_matches_plan_macos_requires_dylibs(tmp_path: Path):
install_dir = tmp_path / "llama.cpp"
install_dir.mkdir()
@ -2050,3 +2383,184 @@ def test_existing_install_matches_choice_fails_when_install_tree_incomplete_maco
)
is False
)
def test_paired_runtime_dll_patterns_excludes_executables() -> None:
"""The paired runtime archive must only contribute CUDA DLLs to
the install. The narrow pattern list -- not the broad
runtime_patterns_for_choice ``*.exe`` / ``*.dll`` -- is what
prevents a malformed cudart bundle from overwriting
llama-server.exe at install time.
"""
paired_runtime_dll_patterns = INSTALL_LLAMA_PREBUILT.paired_runtime_dll_patterns
paired_choice = AssetChoice(
repo = "x",
tag = "t",
name = "llama-b9001-bin-win-cuda-12.4-x64.zip",
url = "u",
source_label = "published",
install_kind = "windows-cuda",
runtime_line = "cuda12",
expected_sha256 = "a" * 64,
runtime_name = "cudart-llama-bin-win-cuda-12.4-x64.zip",
runtime_url = "https://example.com/cudart.zip",
runtime_sha256 = "c" * 64,
)
patterns = paired_runtime_dll_patterns(paired_choice)
assert "cudart64_*.dll" in patterns
assert "cublas64_*.dll" in patterns
assert "cublasLt64_*.dll" in patterns
assert "*.exe" not in patterns
assert "*.dll" not in patterns
for kind in (
"linux-cpu",
"linux-cuda",
"linux-rocm",
"macos-arm64",
"macos-x64",
"windows-cpu",
"windows-hip",
):
non_windows = AssetChoice(
repo = "x",
tag = "t",
name = "x",
url = "u",
source_label = "published",
install_kind = kind,
expected_sha256 = "a" * 64,
)
assert paired_runtime_dll_patterns(non_windows) == []
def test_runtime_overlay_cannot_overwrite_main_archive_payload(
tmp_path: Path,
) -> None:
"""End-to-end: a malformed runtime archive containing
``llama-server.exe`` alongside the real cudart DLLs must NOT
replace the main archive's ``llama-server.exe``.
"""
install_from_archives = INSTALL_LLAMA_PREBUILT.install_from_archives
work = tmp_path / "work"
install = tmp_path / "install"
archives = tmp_path / "archives"
work.mkdir()
install.mkdir()
archives.mkdir()
main_zip = archives / "llama-b9001-bin-win-cuda-12.4-x64.zip"
runtime_zip = archives / "cudart-llama-bin-win-cuda-12.4-x64.zip"
with zipfile.ZipFile(main_zip, "w", zipfile.ZIP_DEFLATED) as zf:
zf.writestr("llama-server.exe", b"MAIN-SERVER")
zf.writestr("llama-quantize.exe", b"MAIN-Q")
zf.writestr("llama.dll", b"DLL-llama")
zf.writestr("ggml-cuda.dll", b"DLL-ggml")
import hashlib
main_sha = hashlib.sha256(main_zip.read_bytes()).hexdigest()
with zipfile.ZipFile(runtime_zip, "w", zipfile.ZIP_DEFLATED) as zf:
zf.writestr("cudart64_12.dll", b"DLL-cudart")
zf.writestr("cublas64_12.dll", b"DLL-cublas")
zf.writestr("cublasLt64_12.dll", b"DLL-cublasLt")
zf.writestr("llama-server.exe", b"RUNTIME-OVERWRITE")
runtime_sha = hashlib.sha256(runtime_zip.read_bytes()).hexdigest()
choice = AssetChoice(
repo = "unslothai/llama.cpp",
tag = "release-1",
name = main_zip.name,
url = f"https://example.com/{main_zip.name}",
source_label = "published",
install_kind = "windows-cuda",
runtime_line = "cuda12",
expected_sha256 = main_sha,
runtime_name = runtime_zip.name,
runtime_url = f"https://example.com/{runtime_zip.name}",
runtime_sha256 = runtime_sha,
)
host = HostInfo(
system = "Windows",
machine = "AMD64",
is_windows = True,
is_linux = False,
is_macos = False,
is_x86_64 = True,
is_arm64 = False,
nvidia_smi = None,
driver_cuda_version = (12, 4),
compute_caps = [],
visible_cuda_devices = None,
has_physical_nvidia = False,
has_usable_nvidia = True,
)
import shutil as _shutil
orig_download = INSTALL_LLAMA_PREBUILT.download_file_verified
def fake_download(url, target_path, *, expected_sha256 = None, label = None, **kw):
src = main_zip if "cudart" not in url else runtime_zip
_shutil.copy2(src, target_path)
if expected_sha256:
actual = hashlib.sha256(Path(target_path).read_bytes()).hexdigest()
if actual != expected_sha256:
raise INSTALL_LLAMA_PREBUILT.PrebuiltFallback(
f"sha256 mismatch on {label}"
)
INSTALL_LLAMA_PREBUILT.download_file_verified = fake_download
try:
install_from_archives(choice, host, install, work)
finally:
INSTALL_LLAMA_PREBUILT.download_file_verified = orig_download
release_dir = install / "build" / "bin" / "Release"
server = release_dir / "llama-server.exe"
assert server.exists()
assert server.read_bytes() == b"MAIN-SERVER", (
"runtime archive overwrote main llama-server.exe; "
f"got {server.read_bytes()!r}"
)
for name in ("cudart64_12.dll", "cublas64_12.dll", "cublasLt64_12.dll"):
assert (release_dir / name).exists(), f"missing {name}"
def test_python_runtime_dirs_covers_cu13_and_library_bin(
monkeypatch, tmp_path: Path
) -> None:
"""Installer-side runtime DLL discovery must scan the same path
set as the backend ``_windows_pip_nvidia_dll_dirs``: legacy
``nvidia/<pkg>/bin``, current ``nvidia/<pkg>/bin/x86_64``
(cu13 layout), conda-style ``nvidia/<pkg>/Library/bin``, plus
``torch/lib``. Otherwise installer preflight and backend launch
can disagree about which DLLs are actually present.
"""
import site as _site
python_runtime_dirs = INSTALL_LLAMA_PREBUILT.python_runtime_dirs
site_dir = tmp_path / "Lib" / "site-packages"
# cu12-style modular wheel
cu12_bin = site_dir / "nvidia" / "cuda_runtime" / "bin"
cu12_bin.mkdir(parents = True)
# cu13-style unsuffixed wheel
cu13_arch = site_dir / "nvidia" / "cu13" / "bin" / "x86_64"
cu13_arch.mkdir(parents = True)
# conda-style repack
library_bin = site_dir / "nvidia" / "cublas" / "Library" / "bin"
library_bin.mkdir(parents = True)
# PyTorch bundled-CUDA wheel
torch_lib = site_dir / "torch" / "lib"
torch_lib.mkdir(parents = True)
monkeypatch.setattr(sys, "path", [str(site_dir)])
monkeypatch.setattr(_site, "getsitepackages", lambda: [str(site_dir)])
monkeypatch.setattr(_site, "getusersitepackages", lambda: "")
dirs = python_runtime_dirs()
assert str(cu12_bin) in dirs
assert str(cu13_arch) in dirs
assert str(library_bin) in dirs
assert str(torch_lib) in dirs

120
tests/studio/install/test_selection_logic.py
View file

@ -1839,6 +1839,126 @@ class TestWindowsCudaAttempts:
assert result[0].name == "cudart-llama-bin-win-cuda-13.1-x64.zip"
assert result[1].name == "cudart-llama-bin-win-cuda-12.4-x64.zip"
def test_cudart_runtime_archive_is_paired(self, monkeypatch):
# #5106: cudart bundle must surface on runtime_url so
# install_from_archives downloads it.
mock_windows_runtime(monkeypatch, ["cuda13", "cuda12"])
host = make_host(system = "Windows", machine = "AMD64", driver_cuda_version = (13, 1))
assets = {
f"llama-{self.TAG}-bin-win-cuda-13.1-x64.zip": f"https://example.com/llama-{self.TAG}-bin-win-cuda-13.1-x64.zip",
"cudart-llama-bin-win-cuda-13.1-x64.zip": "https://example.com/cudart-llama-bin-win-cuda-13.1-x64.zip",
f"llama-{self.TAG}-bin-win-cuda-12.4-x64.zip": f"https://example.com/llama-{self.TAG}-bin-win-cuda-12.4-x64.zip",
"cudart-llama-bin-win-cuda-12.4-x64.zip": "https://example.com/cudart-llama-bin-win-cuda-12.4-x64.zip",
}
result = windows_cuda_attempts(host, self.TAG, assets, None)
assert len(result) == 2
# cuda13 first (host driver supports 13.1)
assert result[0].name == f"llama-{self.TAG}-bin-win-cuda-13.1-x64.zip"
assert result[0].runtime_name == "cudart-llama-bin-win-cuda-13.1-x64.zip"
assert result[0].runtime_url == (
"https://example.com/cudart-llama-bin-win-cuda-13.1-x64.zip"
)
# cuda12 second
assert result[1].name == f"llama-{self.TAG}-bin-win-cuda-12.4-x64.zip"
assert result[1].runtime_name == "cudart-llama-bin-win-cuda-12.4-x64.zip"
def test_no_runtime_archive_when_cudart_absent(self, monkeypatch):
# Older releases without the cudart split must still install.
mock_windows_runtime(monkeypatch, ["cuda12"])
host = make_host(system = "Windows", machine = "AMD64", driver_cuda_version = (12, 4))
assets = {
f"llama-{self.TAG}-bin-win-cuda-12.4-x64.zip": f"https://example.com/llama-{self.TAG}-bin-win-cuda-12.4-x64.zip",
}
result = windows_cuda_attempts(host, self.TAG, assets, None)
assert len(result) == 1
assert result[0].runtime_url is None
assert result[0].runtime_name is None
def test_cudart_only_assets_do_not_self_pair(self, monkeypatch):
# Legacy cudart-only naming path must not self-pair.
mock_windows_runtime(monkeypatch, ["cuda13", "cuda12"])
host = make_host(system = "Windows", machine = "AMD64", driver_cuda_version = (13, 1))
assets = self._upstream("13.1", "12.4", current_names = True)
result = windows_cuda_attempts(host, self.TAG, assets, None)
assert len(result) == 2
for attempt in result:
assert attempt.runtime_url is None
assert attempt.runtime_name is None
# ===========================================================================
# N.1. apply_approved_hashes -- runtime archive checksum threading
# ===========================================================================
class TestApplyApprovedHashesRuntimePair:
"""Runtime archive must inherit a manifest hash, or be dropped."""
TAG = "b8508"
def _runtime_paired_attempt(self) -> AssetChoice:
return AssetChoice(
repo = "unslothai/llama.cpp",
tag = self.TAG,
name = f"llama-{self.TAG}-bin-win-cuda-13.1-x64.zip",
url = f"https://x/llama-{self.TAG}-bin-win-cuda-13.1-x64.zip",
source_label = "published",
install_kind = "windows-cuda",
runtime_line = "cuda13",
runtime_name = "cudart-llama-bin-win-cuda-13.1-x64.zip",
runtime_url = "https://x/cudart-llama-bin-win-cuda-13.1-x64.zip",
)
def test_runtime_hash_threaded_when_present(self):
attempt = self._runtime_paired_attempt()
checksums = ApprovedReleaseChecksums(
repo = "unslothai/llama.cpp",
release_tag = self.TAG,
upstream_tag = self.TAG,
artifacts = {
attempt.name: ApprovedArtifactHash(
asset_name = attempt.name,
sha256 = "0" * 64,
repo = "unslothai/llama.cpp",
kind = "windows-cuda",
),
"cudart-llama-bin-win-cuda-13.1-x64.zip": ApprovedArtifactHash(
asset_name = "cudart-llama-bin-win-cuda-13.1-x64.zip",
sha256 = "1" * 64,
repo = "unslothai/llama.cpp",
kind = "windows-cuda",
),
},
)
result = apply_approved_hashes([attempt], checksums)
assert len(result) == 1
assert result[0].expected_sha256 == "0" * 64
assert result[0].runtime_sha256 == "1" * 64
assert result[0].runtime_name == "cudart-llama-bin-win-cuda-13.1-x64.zip"
def test_runtime_pair_dropped_when_hash_missing(self):
# Drop the pair rather than install an unverified runtime.
attempt = self._runtime_paired_attempt()
checksums = ApprovedReleaseChecksums(
repo = "unslothai/llama.cpp",
release_tag = self.TAG,
upstream_tag = self.TAG,
artifacts = {
attempt.name: ApprovedArtifactHash(
asset_name = attempt.name,
sha256 = "0" * 64,
repo = "unslothai/llama.cpp",
kind = "windows-cuda",
),
},
)
result = apply_approved_hashes([attempt], checksums)
assert len(result) == 1
assert result[0].expected_sha256 == "0" * 64
assert result[0].runtime_url is None
assert result[0].runtime_name is None
assert result[0].runtime_sha256 is None
# ===========================================================================
# O. resolve_upstream_asset_choice -- platform routing