* ROCm/AMD CI coverage: arch-table parity, native-Linux lib prepend, RDNA4 grouped_mm, discovery-based shell suite
Three merged ROCm fixes shipped without tests, and the CI wiring that
would have run them was gated on files the fixes do not touch.
Tests added (113):
tests/studio/install/test_rocm_arch_table_parity.py (27)
diffs the four duplicated gfx -> AMD pip-index tables across
install.sh, install.ps1, studio/setup.ps1 and install_python_stack.py,
plus the GPU-name -> arch tables and the torch 2.11 pin allowlist.
tests/studio/install/test_rocm_native_linux_lib_dirs.py (26)
covers #7233: system-ROCm lib dirs prepended ahead of bundled
libggml-hip, the /dev/kfd + not-WSL + libhsa gate, the opt-out env
var, root resolution order, and source parity between the two copies.
studio/backend/tests/test_grouped_mm_rdna4_fallback.py (46)
covers #7292: registration on the CUDA dispatch key, grouped and
ungrouped numerics, bias/dtype promotion, and the Linux HIP<7.13 +
RDNA4 name gate, executed from the shipped source rather than a copy.
tests/studio/test_ci_shell_suite_coverage.py (14)
fails if either shell runner goes back to a hardcoded list or skips
a file without a recorded reason.
CI wiring:
studio-backend-ci.yml: add install.sh / install.ps1 to the path filter
(the suites it runs assert against those two files, so install-only
changes -- the shape most AMD/ROCm routing fixes take -- skipped it),
and replace the 13-file hardcoded shell list with directory
discovery. That list had fallen seven files behind, including
test_strixhalo_wsl_reroute.sh, the only shell coverage of the ROCm
WSL reroute, which had never run on a PR.
tests/run_all.sh: same discovery loop so local and CI agree.
* Test review fixes: assert on outcomes, not on the code under test
Self-review of the previous commit found four tests that passed for the
wrong reason.
1. The arch-table parity test pinned expected gfx ids copied out of the
shipped tables, which enshrined three upstream inaccuracies as
correct: RX 9070 (non-XT) is gfx1201 not gfx1200, RX 7800 XT is
gfx1101 not gfx1100, and PRO V710 is gfx1101 not gfx1102 per AMD's
ROCm compatibility matrix. The expectation is now the AMD pip index
leaf -- the thing the tables exist to produce, and what a wrong
answer costs the user. The three known drifts are listed explicitly
with a test asserting they stay cosmetic, i.e. that the wrong and
right ids still map to the same wheel index. That test turns red the
day one of them starts routing users to the wrong wheel.
2. The RDNA4 device-name test extracted the regex from worker.py and
then matched with it, so it could not fail. Widening the pattern --
the dangerous edit, since it forces the slow Python mm fallback onto
RDNA3 users -- would have been silently accepted. It now reads the
live pattern and checks it against fixed cases, plus asserts the
name match stays guarded by `not _lin_arch` and that the name is
lowercased before matching.
3. The CI-coverage test matched a verbatim line of studio-backend-ci.yml,
so reindenting the step would fail the build while a real regression
to a hardcoded list could slip past a reformat. It now parses the
YAML, finds the step by name, and asserts on the glob plus the
absence of individual filenames. The path-filter test likewise reads
the parsed trigger instead of scanning raw text.
4. A set comprehension in the parity helper had a ternary whose branches
were identical.
Mutation-tested: widening the RDNA4 regex, desyncing one copy of the
name table, dropping install.sh from the path filter, and re-skipping
the ROCm WSL shell suite each fail at least two tests. Verified on
Linux (WSL Ubuntu 24.04) with CI's torch pin: 86 + 48 pass.
* Fix three wrong gfx ids in the GPU-name arch tables
The name -> gfx tables disagreed with AMD's ROCm compatibility matrix on
three entries. Corrected against the "Radeon GPU" list at
rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html:
RX 9070, RX 9070 GRE gfx1200 -> gfx1201 (Navi 48, same die as the XT)
RX 7800 XT, RX 7700 XT gfx1100 -> gfx1101 (Navi 32, not Navi 31)
PRO W7700 gfx1100 -> gfx1101
PRO V710 gfx1102 -> gfx1101 (Navi 32, not Navi 33)
No wheel changes for anyone: gfx1200/gfx1201 both resolve to gfx120X-all
and gfx1100/gfx1101/gfx1102 all resolve to gfx110X-all, in all four copies
of the index-family map. That collapse is why the errors survived being
copied into six places -- the leaf-level tests could not see them.
It was not purely cosmetic, though. install.sh's second copy feeds
"Tip: set UNSLOTH_ROCM_GFX_ARCH=<arch>", so a 7800 XT user following the
printed advice exported gfx1100 and made a wrong id authoritative for
every later run. It would also have become a real misroute the moment AMD
split a family across index leaves, as they already do for gfx1151/gfx1150.
Fixed in all six places, which is two more than the table's own "kept in
sync with" comments claim exist:
install.sh _infer_amd_gfx_arch_from_gpu_name
install.sh case "$_gpu_disp_mkt" (banner + env tip; undocumented)
studio/setup.sh
install.ps1
studio/setup.ps1
studio/install_python_stack.py
Ordering is preserved: the gfx1102 arm still precedes gfx1101 in the shell
copies so "RX 7700S" cannot fall onto the "RX 7700" glob, and the
PowerShell copies keep the (?!S) lookahead.
Test changes:
- test_rocm_arch_table_parity.py gains _AMD_DOCUMENTED_ARCH, exact gfx
ids transcribed from AMD rather than from the tables. Agreement between
six copies proves nothing when all six were transcribed from the same
mistake, so the ground truth has to come from outside. Verified it
catches the bug: against the pre-fix tables it fails 6 tests.
- The parity check now covers all six copies. It had four; the two
install.sh copies were being treated as one, and
_WIN_GPU_NAME_ARCH_TABLE was not checked at all.
- test_rocm_support.py's TestGfxArchNameFallback pinned two of the wrong
ids as expected values; updated, and extended with a 9060 XT and a
7900 XTX case so each RDNA3/4 die is represented.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard against unregistered copies of the GPU-name arch table
Counting the copies by hand is what let them drift: the in-code "kept in
sync with" comments claimed four, the arch-id fix found six, and scanning
the tree turns up a seventh.
TestNoUnregisteredArchTable rediscovers the copies from the source tree
instead of trusting a hand-maintained list. A table line is one that names
a card and gives its arch; real tables score 9-17 such lines and the only
other hits in the repo are two single-line prose comments, so the
three-line threshold is not load-bearing. A companion test asserts the
scan still finds the known copies, so the heuristic cannot go blind and
pass by finding nothing.
The seventh copy is tests/_zoo_rocm_spoof.py, the fixture other ROCm tests
build their fake AMD host from. It states the mapping backwards (gfx ->
the name torch should report), which makes it an independent witness: it
had gfx1101 -> RX 7800 XT and gfx1201 -> RX 9070 XT right while all six
installer copies were wrong, and nothing compared the two. Now they are
round-tripped against each other.
RX 6700 XT is pinned as a known divergence rather than normalised. AMD's
compatibility matrix documents no consumer RX 6000 card and no gfx1031 at
all, the installer arm is commented "gfx103X family", and gfx1031 appears
only as an index-family key, never as a value a name table emits. With no
external source to correct against, changing shipped behaviour would be
guesswork. A test fails if the divergence ever disappears, so the
exemption cannot go stale.
Also adds the reverse of the AMD-matrix check: a documented card that
matches no arm anywhere is a silent CPU fallback rather than a wrong id.
This cannot detect hardware nobody transcribed, which would need a live
fetch of AMD's matrix and a non-hermetic suite; the docstring says so
rather than implying coverage that is not there.
Verified on Linux: 478 passed, plus all five new guards mutation-tested
to confirm each fails when its invariant is broken.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Docstring said six copies; the list under it now has seven
* tests: run discovered shell tests with bash, not sh
tests/run_all.sh discovered tests/sh/ instead of listing files, but still
invoked each one with sh. Every file there declares a bash shebang, and on
Debian/Ubuntu /bin/sh is dash: test_apt_distro_prompt.sh,
test_studio_home_node_dir.sh and test_with_llama_cpp_dir_link_behavior.sh
fail on bashisms under dash and pass under bash. The old hand-written list
happened to name only dash-clean files, so switching to discovery is what
surfaced it. Backend CI already used bash, so this was a local-only break.
Guarded by a new test asserting both runners invoke tests/sh/ with bash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Krackan Point (Radeon 860M/840M) routed to the gfx1150 wheel index
The GPU-name tables map 860M/840M and the Ryzen AI 7 350 / AI 5 340 CPU
strings to gfx1150, but Krackan Point is gfx1152. AMD's own lemonade table
(src/cpp/server/system_info.cpp) maps both Krackan iGPUs to gfx1152.
Unlike the three ids already fixed here, this one is not wheel-neutral:
repo.amd.com publishes gfx1150 and gfx1152 as separate index leaves with
separately built torch wheels, so these laptops were installing wheels
built for a different LLVM target. gfx1152 was absent from the codebase
entirely, so it needed the index-family maps, the torch 2.11 floor lists
(same _grouped_mm bug as gfx1150/1151), the Strix reroute set and the
Windows arch allowlist as well as the seven name tables.
The parity test added in this PR did not catch it because its AMD-matrix
expectations stopped at 890M/880M. Added the APU rows, so the case that
actually changes a wheel is now covered: reverting the tables fails 9
tests naming 860M, 840M and Krackan.
gfx1153 (Ryzen AI 5 430 era) is left alone; AMD publishes no gfx1153
wheel family, so there is nothing to route it to.
Verified: bash -n on both shell installers, PowerShell AST parse on both
.ps1 files, python ast.parse on all touched modules, install suite 1334
passed with no new failures against main, shell suite 20 files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add gfx1152 to unified-memory classifiers, make parity allowlist set-based
Krackan Point (gfx1152, Radeon 860M/840M) is the third RDNA 3.5 APU and
shares one GPU/system-RAM pool exactly like Strix Point (gfx1150) and
Strix Halo (gfx1151), but only the installers knew about it. The two
runtime classifiers still had two-element arch sets, so a Krackan laptop
got the 0.90 discrete headroom factor on a shared pool and ran llama.cpp
without GGML_CUDA_ENABLE_UNIFIED_MEMORY.
- worker.py _rocm_classify_unified_memory: add gfx1152 to the arch set,
and 860m/840m to the device-name fallback. The NVIDIA GeForce 840M
cannot collide there: the function is only reached under _hw.IS_ROCM.
- llama_cpp.py _amd_apu_wants_unified_memory: add gfx1152 to the arch set.
- Tests for both, including the :sramecc-:xnack- suffix form.
TestGfx211AllowlistParity compared four hardcoded allowlist strings, so
adding gfx1152 to all four installers correctly turned three assertions
red without any installer actually disagreeing with another. Each test
now extracts the set its installer holds and compares it to one EXPECTED
constant. Order and spacing are free, membership is not, and the next
leaf is a one-line edit instead of four.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
253 lines
10 KiB
Python
253 lines
10 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Unit tests for _rocm_classify_unified_memory (ROCm OOM-guard classifier).
|
|
|
|
Three paths: (1) canonical gcnArchName, (2) alternate-spelling attr, (3) all
|
|
arch attrs absent -> device-name substring match.
|
|
|
|
Regression: Strix Halo (gfx1151) was misclassified as discrete on Radeon wheels
|
|
that set props.name="Radeon 8060S Graphics" but no gcnArchName, applying the
|
|
wrong headroom factor on a 128 GiB unified-memory pool.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from core.training.worker import _rocm_classify_unified_memory
|
|
|
|
|
|
# ── helpers ──────────────────────────────────────────────────────────────────
|
|
|
|
|
|
def _props(**kwargs) -> SimpleNamespace:
|
|
"""Build a fake device-properties object with the given attributes."""
|
|
return SimpleNamespace(**kwargs)
|
|
|
|
|
|
# ── Path 0: props.is_integrated (driver's own unified-memory answer) ─────────
|
|
|
|
|
|
class TestIsIntegratedSignal:
|
|
"""hipDeviceProp_t.integrated wins when truthy; 0/absent never downgrades.
|
|
|
|
Same universal gate PR #5988's UMA safetensors fast-load uses -- keeps
|
|
Unsloth's two unified-memory consumers on one signal."""
|
|
|
|
def test_integrated_upgrades_unknown_apu(self) -> None:
|
|
# gfx1103 Phoenix iGPU: outside the hardcoded arch set, but the
|
|
# driver says integrated -> unified.
|
|
props = _props(gcnArchName = "gfx1103", name = "Radeon 780M", is_integrated = 1)
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == "gfx1103"
|
|
assert is_unified is True
|
|
|
|
def test_integrated_wins_without_any_arch(self) -> None:
|
|
props = _props(name = "Some Future APU", is_integrated = 1)
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == ""
|
|
assert is_unified is True
|
|
|
|
def test_zero_does_not_downgrade_known_apu(self) -> None:
|
|
# A wheel that zeroes the field must not flip Strix Halo to discrete.
|
|
props = _props(gcnArchName = "gfx1151", name = "x", is_integrated = 0)
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert is_unified is True
|
|
|
|
def test_absent_keeps_existing_behavior(self) -> None:
|
|
props = _props(gcnArchName = "gfx1201", name = "RX 9070 XT")
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert is_unified is False
|
|
|
|
def test_discrete_with_zero_stays_discrete(self) -> None:
|
|
props = _props(gcnArchName = "gfx1100", name = "RX 7900 XTX", is_integrated = 0)
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert is_unified is False
|
|
|
|
|
|
# ── Path 1: canonical gcnArchName ────────────────────────────────────────────
|
|
|
|
|
|
class TestCanonicalGcnArchName:
|
|
"""gcnArchName is present and populated."""
|
|
|
|
@pytest.mark.parametrize(
|
|
"arch, expected_unified",
|
|
[
|
|
("gfx1150", True), # Strix Point
|
|
("gfx1151", True), # Strix Halo
|
|
("gfx1152", True), # Krackan Point (Radeon 860M/840M)
|
|
("gfx1100", False), # Navi 31 (RX 7900 XTX) — discrete
|
|
("gfx906", False), # MI50 — discrete server GPU
|
|
("gfx1201", False), # RX 9070 XT — discrete
|
|
],
|
|
)
|
|
def test_canonical_attr(self, arch: str, expected_unified: bool) -> None:
|
|
props = _props(gcnArchName = arch, name = "irrelevant")
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == arch
|
|
assert is_unified is expected_unified
|
|
|
|
def test_arch_with_colon_suffix_stripped(self) -> None:
|
|
"""gcnArchName can carry xnack/sramecc suffix; only the base is kept."""
|
|
props = _props(gcnArchName = "gfx1151:xnack-", name = "irrelevant")
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == "gfx1151"
|
|
assert is_unified is True
|
|
|
|
def test_canonical_attr_wins_over_name(self) -> None:
|
|
"""Arch attr takes priority; device name is ignored."""
|
|
# Discrete arch, but name looks like a unified SKU — arch must win.
|
|
props = _props(gcnArchName = "gfx1100", name = "Radeon 890M")
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == "gfx1100"
|
|
assert is_unified is False
|
|
|
|
|
|
# ── Path 2: alternate-spelling fallback ──────────────────────────────────────
|
|
|
|
|
|
class TestAlternateSpellingFallback:
|
|
"""gcnArchName is missing but an alternate attr spelling is present."""
|
|
|
|
@pytest.mark.parametrize(
|
|
"attr_name",
|
|
["gcn_arch_name", "arch_name", "gfx_arch_name"],
|
|
)
|
|
def test_alternate_attr_unified(self, attr_name: str) -> None:
|
|
props = _props(**{attr_name: "gfx1151"}, name = "Radeon 8060S Graphics")
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == "gfx1151"
|
|
assert is_unified is True
|
|
|
|
@pytest.mark.parametrize(
|
|
"attr_name",
|
|
["gcn_arch_name", "arch_name", "gfx_arch_name"],
|
|
)
|
|
def test_alternate_attr_discrete(self, attr_name: str) -> None:
|
|
props = _props(**{attr_name: "gfx1201"}, name = "Radeon RX 9070 XT")
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == "gfx1201"
|
|
assert is_unified is False
|
|
|
|
def test_first_non_empty_attr_wins(self) -> None:
|
|
"""With multiple alternate attrs, the first non-empty one wins."""
|
|
props = _props(gcn_arch_name = "gfx1151", arch_name = "gfx1100", name = "irrelevant")
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == "gfx1151"
|
|
assert is_unified is True
|
|
|
|
|
|
# ── Path 3: device-name fallback ─────────────────────────────────────────────
|
|
|
|
|
|
class TestDeviceNameFallback:
|
|
"""ALL arch attrs absent — classifier must rely solely on device name."""
|
|
|
|
# --- unified-memory devices that MUST be detected ---
|
|
|
|
@pytest.mark.parametrize(
|
|
"device_name",
|
|
[
|
|
# gfx1150 Strix Point
|
|
"Radeon 890M",
|
|
"AMD Radeon 890M Graphics",
|
|
"RADEON 890M", # case-insensitive
|
|
"Radeon 880M",
|
|
"AMD Radeon 880M Graphics",
|
|
# gfx1151 Strix Halo — the regression case from the review
|
|
"Radeon 8060S Graphics", # Ryzen AI MAX+ 395 (as returned by torch)
|
|
"AMD Radeon 8060S",
|
|
"Radeon 8050S Graphics", # cut-down Strix Halo SKU
|
|
"AMD Radeon 8050S",
|
|
# gfx1151 Gorgon Halo (Ryzen AI Max 400 refresh)
|
|
"Radeon 8065S Graphics", # Ryzen AI Max+ 495
|
|
"AMD Radeon 8065S",
|
|
# gfx1152 Krackan Point (Ryzen AI 7 350 / AI 5 340)
|
|
"Radeon 860M",
|
|
"AMD Radeon 860M Graphics",
|
|
"Radeon 840M",
|
|
"AMD Radeon 840M Graphics",
|
|
# case variants
|
|
"RADEON 8060S GRAPHICS",
|
|
"radeon 8050s",
|
|
"RADEON 860M",
|
|
],
|
|
)
|
|
def test_unified_memory_detected(self, device_name: str) -> None:
|
|
props = _props(name = device_name)
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == "", f"expected empty gcn_arch, got {gcn!r}"
|
|
assert is_unified is True, f"device {device_name!r} should be classified as unified-memory"
|
|
|
|
# --- discrete devices that must NOT be mis-classified ---
|
|
|
|
@pytest.mark.parametrize(
|
|
"device_name",
|
|
[
|
|
"Radeon RX 9070 XT",
|
|
"AMD Radeon RX 7900 XTX",
|
|
"Radeon RX 6900 XT",
|
|
"Radeon Pro W7900",
|
|
"AMD Instinct MI300X",
|
|
# Superficially similar substrings but discrete
|
|
"Radeon RX 580",
|
|
"Radeon VII",
|
|
],
|
|
)
|
|
def test_discrete_not_misclassified(self, device_name: str) -> None:
|
|
props = _props(name = device_name)
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == ""
|
|
assert (
|
|
is_unified is False
|
|
), f"discrete device {device_name!r} should NOT be classified as unified-memory"
|
|
|
|
def test_empty_name_returns_false(self) -> None:
|
|
"""Absent name must not crash and must default to discrete."""
|
|
props = _props() # no 'name' attr at all
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == ""
|
|
assert is_unified is False
|
|
|
|
def test_none_name_returns_false(self) -> None:
|
|
props = _props(name = None)
|
|
gcn, is_unified = _rocm_classify_unified_memory(props)
|
|
assert gcn == ""
|
|
assert is_unified is False
|
|
|
|
|
|
# ── Fraction selection (source-pinned) ───────────────────────────────────────
|
|
|
|
|
|
_WORKER_PY = Path(__file__).resolve().parents[1] / "core" / "training" / "worker.py"
|
|
|
|
|
|
class TestMemFractionSelection:
|
|
"""Pin the per-platform fraction policy in worker.py section 1g.
|
|
|
|
On native Windows, torch.cuda.mem_get_info's total is the WDDM budget
|
|
the driver grants HIP -- the OS share of RAM is already outside it, so
|
|
a 0.80 cap double-taxes (field report: 48.49 GiB budget -> '38.79 GiB
|
|
allowed' OOM denying a 47.29 GiB load that fit in free memory). 1.0
|
|
removes the double-tax; current AMD Windows wheels enforce only
|
|
sub-1.0 fractions, so it behaves like torch's uncapped default with
|
|
WDDM arbitrating residency (measured on gfx1151)."""
|
|
|
|
def test_unified_win32_uses_budget_exact_fraction(self) -> None:
|
|
source = _WORKER_PY.read_text(encoding = "utf-8")
|
|
assert '1.0 if sys.platform == "win32" else 0.80' in source
|
|
|
|
def test_discrete_keeps_090(self) -> None:
|
|
source = _WORKER_PY.read_text(encoding = "utf-8")
|
|
assert "_mem_fraction = 0.90" in source
|
|
|
|
def test_win32_unified_logs_vgm_hint(self) -> None:
|
|
"""Users must learn the WDDM budget is raisable (BIOS UMA / AMD
|
|
Software Variable Graphics Memory) instead of assuming a bug."""
|
|
source = _WORKER_PY.read_text(encoding = "utf-8")
|
|
assert "Variable Graphics Memory" in source
|