Studio: fix per-GPU VRAM reporting on Windows ROCm (#7238)
* Studio: fix per-GPU VRAM reporting on Windows ROCm On Windows ROCm without a HIP SDK, amd-smi is disabled and the System tab fell back to torch mem_get_info, which reports free==total there (ROCm/ROCm#1909), so used VRAM showed as 0. The perf-counter fallback also summed every adapter into a single device with only GPU 0's total, hiding the second GPU. Read per-adapter Dedicated Usage (LUID-instanced) for used and take each GPU's total from torch properties, and treat the free==total case as unknown rather than 0, so every GPU shows real usage. NVIDIA, Linux ROCm, Apple and CPU paths are unchanged. Final validation needs a real Windows AMD box. Fixes #7072 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: report unknown VRAM instead of fabricating or zeroing it Two gaps in the Windows ROCm VRAM path. When more adapters are actively using VRAM than are visible to the process (a GPU outside the visibility mask), the per-adapter attribution paired usage by size and fabricated a per-GPU value; report unknown for every device in that case rather than mis-assign. And the System API turned an unknown (None) used value into 0 with ``or 0``, then reported the full card as free, re-hiding the exact case this change surfaces; keep None so the UI shows unknown. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Render unknown VRAM as Unknown instead of zero in the System tab The backend reports null usage when it is unknown (e.g. the Windows ROCm perf counter is unavailable or localized), but the System tab coerced null to 0 and derived free from it, fabricating a 0-used/full-free total. Preserve null and render the translated Unknown for per-device used, free and utilization, and mark the aggregate VRAM tile unknown when any device is unknown. * Render unknown VRAM as Unknown in the floating monitor and the util tile The floating VRAM monitor and the aggregate utilization ring both still coerced a null usage to 0, showing a fabricated 0.00 GiB / full free / 0% on the same Windows ROCm no-counter case the resources tab already handles. Guard both on whether every device reports a finite usage and render Unknown (value and percent) instead of a concrete 0. * Attribute per-adapter VRAM usage only when capacity forces the mapping On Windows/ROCm there is no shared key between LUID performance-counter instances and torch ordinals, so usage was paired to devices purely by capacity ranking. That pairing is only trustworthy when capacity forces it (a usage larger than every smaller device can sit on one card). When a smaller-capacity device could equally hold a strictly larger usage (for example an 8 GiB card near full beside a lightly used 48 GiB card), the two values are swappable without violating any capacity, so the ranking is a guess with no key to break the tie. A wrong guess both mislabels the System tab and feeds routes/training_vram.py a wrong per-index free value, driving a wrong keep-resident decision. Report unknown for every device when the assignment is ambiguous, keeping the attribution only for the capacity-forced case. Returning None is the conservative direction: training_vram treats a missing index as zero free, so it never keeps a chat model into an OOM. Add regression tests for the not-capacity-ordered, same-capacity, single-fits-both, and capacity-forced cases. * Report unknown VRAM usage when a hidden adapter survives the noise filter When HIP_VISIBLE_DEVICES exposes a subset of the physical adapters, the LUID usage counters cover cards outside the visibility mask too. The sub-64 MiB noise filter could drop a genuinely-idle visible card's real usage while keeping a hidden larger card's high usage, which was then clamped onto the smaller visible device and reported as fully used (for example a hidden 48 GiB card at 40 GiB shown as a visible 8 GiB card fully used, with its true 10 MiB usage filtered out). That fabricated reading also feeds routes/training_vram.py a wrong per-index free value. Flag extra adapters on the raw counter count (before the noise filter, since an idle visible card can itself fall below the floor) and, when a kept usage exceeds its ranked visible capacity, report unknown rather than clamp a hidden card's usage onto a visible device. The genuinely-idle-noise and capacity-forced single-model cases are unchanged. Add a regression test for the hidden high-use-adapter case in both counter orders. * Report unknown when only a placeholder adapter counter survives the noise filter When more raw counters than visible devices are present but every counter sits below the 64 MiB noise floor (an idle real GPU alongside a Windows Basic Render Driver placeholder), the non_trivial-or-raw fallback resurrected the raw magnitude-sorted counters and could attribute the placeholder to a real GPU while dropping a real card's reading. With a single visible device the swap-ambiguity check cannot catch it (it needs at least two ranks), so the fabricated value reached the System tab and automatic GPU selection. Return unknown for every device in that case instead of falling back to raw counters. With the earlier guards this completes the invariant: a concrete per-GPU usage is emitted only when the assignment is capacity-forced, and every ambiguous, extra-adapter, placeholder-fallback, or count-mismatch path reports unknown. Add a regression test for the placeholder fallback in both counter orders and the two-idle-GPU case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: attribute Windows/ROCm VRAM only when capacity forces a clean bijection With more raw adapter counters than visible devices, a survivor that merely fits a visible card was pinned to it by magnitude ranking, fabricating a hidden GPU's usage onto an idle visible card whose true reading was dropped by the sub-threshold noise filter (two visible 48/8 GiB cards using 40 GiB / 10 MiB beside a hidden 6 GiB adapter returned [40, 6]). Emit a concrete per-device value only when the supra-threshold counters number exactly the visible devices (every visible card has one real reading, the extras were sub-threshold placeholders) AND the ranked usage strictly exceeds every smaller visible card's capacity. When a visible card is idle (fewer supra-threshold counters than devices) a survivor could be the hidden GPU's usage, so every device reports unknown; more active counters than visible cards, the smallest card, and any merely-fitting usage stay unknown too. The reporter's loaded-card display is preserved (40 GiB / 0.5 GiB across 48/8 GiB -> [40, None]). Adds a regression test for the reported case plus an exhaustive capacity-forced/bijection matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: keep the unified-memory total when Windows-ROCm used is unknown _apply_unified_memory_correction gated both the total and the used update on torch_used_gb being known, so on a unified-memory APU (Strix Halo) where torch reports used=None (the Windows-ROCm free==total sentinel) but an authoritative full-GTT total, the device kept amd-smi's small dedicated carve-out and underreported its capacity on the System tab. Adopt torch's larger total independently of used; overwrite used only when torch's is known (otherwise keep amd-smi's dedicated-usage figure) and recompute utilization against the corrected total. Adds regression tests. * Tighten comments in the ROCm/Windows VRAM reporting path * Tighten comments further in the ROCm/Windows VRAM reporting path --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>
This commit is contained in:
Daniel Han
GitHub
parent
65587c2be7
commit
66808ab25d
5 changed files with 716 additions and 86 deletions
|
|
@ -1149,11 +1149,15 @@ def _get_cached_system_gpu_info(logger) -> dict[str, Any]:
|
|||
util = util_devices.get(idx, {})
|
||||
|
||||
total_vram = util.get("vram_total_gb") or dev.get("memory_total_gb") or 0
|
||||
used_vram = util.get("vram_used_gb") or 0
|
||||
# Keep None (usage unknown, e.g. Windows ROCm perf counter) so the UI
|
||||
# shows unknown, not a fabricated 0 used / full free.
|
||||
used_vram = util.get("vram_used_gb")
|
||||
|
||||
enriched_dev = dict(dev)
|
||||
enriched_dev["vram_used_gb"] = used_vram
|
||||
enriched_dev["vram_free_gb"] = round(total_vram - used_vram, 2) if total_vram else 0
|
||||
enriched_dev["vram_free_gb"] = (
|
||||
round(total_vram - used_vram, 2) if total_vram and used_vram is not None else None
|
||||
)
|
||||
enriched_dev["vram_utilization_pct"] = util.get("vram_utilization_pct")
|
||||
enriched_devices.append(enriched_dev)
|
||||
|
||||
|
|
|
|||
361
studio/backend/tests/test_rocm_windows_vram_7072.py
Normal file
361
studio/backend/tests/test_rocm_windows_vram_7072.py
Normal file
|
|
@ -0,0 +1,361 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""Regression tests for issue #7072 -- "VRAM Usage in System Tab is wrong".
|
||||
|
||||
Reporter: dual AMD (Radeon PRO W7900 ~48GB + W7500 8GB), Windows 10, ROCm 7.13,
|
||||
torch 2.11.0+rocm7.13. On Windows without a HIP SDK, amd-smi is permanently
|
||||
disabled (avoids a UAC/DiskPart prompt) and hipMemGetInfo returns free==total
|
||||
(used 0). Two symptoms followed:
|
||||
|
||||
* System tab (/api/system -> get_visible_gpu_utilization) showed ~0 VRAM used
|
||||
on every GPU (torch mem_get_info free==total quirk; ROCm/ROCm#1909).
|
||||
* get_gpu_utilization()'s Windows fallback SUMMED "GPU Adapter Memory\\Dedicated
|
||||
Usage" across all adapters into ONE fake device with only GPU 0's total, so
|
||||
the second GPU never appeared.
|
||||
|
||||
The fix reads the per-adapter (LUID-instanced) Dedicated Usage performance
|
||||
counter -- Task Manager's source -- for per-GPU used, takes per-GPU total from
|
||||
torch device properties, and guards the free==total mem_get_info quirk. CI has no
|
||||
AMD GPU/Windows, so torch, the performance counter, and platform are all mocked.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from utils.hardware import hardware as hw
|
||||
|
||||
GB = 1024**3
|
||||
MiB = 1024**2
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- #
|
||||
# Fakes
|
||||
# ----------------------------------------------------------------------------- #
|
||||
def _fake_torch(
|
||||
devices,
|
||||
*,
|
||||
free_equals_total = False,
|
||||
used_per_device = None,
|
||||
):
|
||||
"""Build a fake `torch` module. devices: list of (name, total_bytes)."""
|
||||
dev = list(devices)
|
||||
|
||||
class _Props:
|
||||
def __init__(self, name, total):
|
||||
self.name = name
|
||||
self.total_memory = total
|
||||
|
||||
def get_device_properties(i):
|
||||
name, total = dev[i]
|
||||
return _Props(name, total)
|
||||
|
||||
def mem_get_info(i):
|
||||
_, total = dev[i]
|
||||
if free_equals_total:
|
||||
return (total, total)
|
||||
used = used_per_device[i] if used_per_device is not None else 0
|
||||
return (total - used, total)
|
||||
|
||||
t = types.ModuleType("torch")
|
||||
t.__version__ = "2.11.0+rocm7.13"
|
||||
t.version = types.SimpleNamespace(hip = "7.13", cuda = None)
|
||||
t.cuda = types.SimpleNamespace(
|
||||
is_available = lambda: len(dev) > 0,
|
||||
device_count = lambda: len(dev),
|
||||
current_device = lambda: 0,
|
||||
get_device_properties = get_device_properties,
|
||||
mem_get_info = mem_get_info,
|
||||
memory_allocated = lambda i: 0,
|
||||
memory_reserved = lambda i: 0,
|
||||
)
|
||||
return t
|
||||
|
||||
|
||||
def _adapter_output(adapters):
|
||||
if not adapters:
|
||||
return "__NONE__\n"
|
||||
return "".join(f"{name}|{int(used)}\n" for name, used in adapters)
|
||||
|
||||
|
||||
def _subprocess_run(*, adapter_output = "__NONE__\n", util_output = "12.0\n"):
|
||||
def fake_run(cmd, *a, **k):
|
||||
joined = " ".join(cmd) if isinstance(cmd, list) else str(cmd)
|
||||
if "GPU Adapter Memory" in joined and "InstanceName" in joined:
|
||||
out = adapter_output
|
||||
elif "engtype_3D" in joined or "GPU Engine" in joined:
|
||||
out = util_output
|
||||
else:
|
||||
out = "-1\n"
|
||||
return subprocess.CompletedProcess(args = cmd, returncode = 0, stdout = out, stderr = "")
|
||||
|
||||
return fake_run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def win_rocm(monkeypatch):
|
||||
"""Configure the hardware module as a Windows ROCm host with 2 visible GPUs."""
|
||||
monkeypatch.setattr(hw, "get_device", lambda: hw.DeviceType.CUDA)
|
||||
monkeypatch.setattr(hw, "IS_ROCM", True)
|
||||
monkeypatch.setattr(hw.platform, "system", lambda: "Windows")
|
||||
monkeypatch.setattr(hw.sys, "platform", "win32")
|
||||
monkeypatch.setattr(hw, "_smi_query", lambda *a, **k: None) # amd-smi disabled
|
||||
# Visible set via HIP mask so we don't shell out to amd-smi for the count.
|
||||
monkeypatch.setenv("HIP_VISIBLE_DEVICES", "0,1")
|
||||
monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising = False)
|
||||
monkeypatch.delenv("ROCR_VISIBLE_DEVICES", raising = False)
|
||||
return monkeypatch
|
||||
|
||||
|
||||
REPORTER_ADAPTERS = [
|
||||
("luid_0x00000000_0x0000d1e2_phys_0", 40.0 * GB), # W7900, model loaded
|
||||
("luid_0x00000000_0x0000e34a_phys_0", 0.5 * GB), # W7500, idle
|
||||
("luid_0x00000000_0x0000f001_phys_0", 3 * MiB), # Basic Render Driver
|
||||
]
|
||||
DEVICES = [("AMD Radeon PRO W7900", 48 * GB), ("AMD Radeon PRO W7500", 8 * GB)]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- #
|
||||
# System tab (get_visible_gpu_utilization) -- the reporter's screenshot
|
||||
# ----------------------------------------------------------------------------- #
|
||||
def test_system_tab_shows_per_gpu_used(win_rocm, monkeypatch):
|
||||
monkeypatch.setitem(sys.modules, "torch", _fake_torch(DEVICES, free_equals_total = True))
|
||||
monkeypatch.setattr(
|
||||
hw.subprocess, "run", _subprocess_run(adapter_output = _adapter_output(REPORTER_ADAPTERS))
|
||||
)
|
||||
|
||||
devices = hw.get_visible_gpu_utilization()["devices"]
|
||||
by_idx = {d["index"]: d for d in devices}
|
||||
assert len(devices) == 2
|
||||
assert by_idx[0]["vram_total_gb"] == 48.0
|
||||
assert by_idx[0]["vram_used_gb"] == pytest.approx(40.0, abs = 0.01) # not 0
|
||||
assert by_idx[1]["vram_total_gb"] == 8.0 # own total
|
||||
# The 3 MiB Basic Render Driver counter makes this a hidden-adapter case: only
|
||||
# the 40 GiB is forced onto the 48 GiB card; the idle card reads Unknown.
|
||||
assert by_idx[1]["vram_used_gb"] is None
|
||||
assert by_idx[1]["vram_utilization_pct"] is None
|
||||
assert all(
|
||||
d["vram_used_gb"] <= d["vram_total_gb"] for d in devices if d["vram_used_gb"] is not None
|
||||
)
|
||||
|
||||
|
||||
def test_gpu_utilization_does_not_collapse(win_rocm, monkeypatch):
|
||||
monkeypatch.setitem(sys.modules, "torch", _fake_torch(DEVICES, free_equals_total = True))
|
||||
monkeypatch.setattr(
|
||||
hw.subprocess, "run", _subprocess_run(adapter_output = _adapter_output(REPORTER_ADAPTERS))
|
||||
)
|
||||
|
||||
result = hw.get_gpu_utilization()
|
||||
devices = result["devices"]
|
||||
assert sorted(d["index"] for d in devices) == [0, 1] # both GPUs, no collapse
|
||||
assert {d["vram_total_gb"] for d in devices} == {48.0, 8.0}
|
||||
assert result["vram_total_gb"] == 48.0 # legacy primary mirror preserved
|
||||
|
||||
|
||||
def test_localized_counter_reports_unknown_not_zero(win_rocm, monkeypatch):
|
||||
monkeypatch.setitem(sys.modules, "torch", _fake_torch(DEVICES, free_equals_total = True))
|
||||
monkeypatch.setattr(hw.subprocess, "run", _subprocess_run(adapter_output = "__NONE__\n"))
|
||||
|
||||
devices = hw.get_visible_gpu_utilization()["devices"]
|
||||
assert len(devices) == 2 # both still shown with correct totals
|
||||
assert {d["vram_total_gb"] for d in devices} == {48.0, 8.0}
|
||||
assert all(d["vram_used_gb"] is None for d in devices) # unknown, not fake 0
|
||||
assert all(d["vram_utilization_pct"] is None for d in devices)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- #
|
||||
# mem_get_info free==total guard scoping
|
||||
# ----------------------------------------------------------------------------- #
|
||||
def test_mem_get_info_guard_scopes_to_windows_rocm(monkeypatch):
|
||||
torch_mod = _fake_torch(DEVICES, free_equals_total = True)
|
||||
monkeypatch.setattr(hw, "get_device", lambda: hw.DeviceType.CUDA)
|
||||
monkeypatch.setitem(sys.modules, "torch", torch_mod)
|
||||
|
||||
# Windows ROCm -> used unknown (None), total kept.
|
||||
monkeypatch.setattr(hw, "IS_ROCM", True)
|
||||
monkeypatch.setattr(hw.sys, "platform", "win32")
|
||||
win = hw._torch_get_per_device_info([0, 1])
|
||||
assert [d["used_gb"] for d in win] == [None, None]
|
||||
assert [d["total_gb"] for d in win] == [48.0, 8.0]
|
||||
|
||||
# Linux ROCm -> unchanged numeric used.
|
||||
monkeypatch.setattr(hw.sys, "platform", "linux")
|
||||
assert [d["used_gb"] for d in hw._torch_get_per_device_info([0, 1])] == [0.0, 0.0]
|
||||
|
||||
# Windows NVIDIA -> guard must not fire.
|
||||
monkeypatch.setattr(hw, "IS_ROCM", False)
|
||||
monkeypatch.setattr(hw.sys, "platform", "win32")
|
||||
assert [d["used_gb"] for d in hw._torch_get_per_device_info([0, 1])] == [0.0, 0.0]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- #
|
||||
# Per-adapter attribution helpers (pure unit)
|
||||
# ----------------------------------------------------------------------------- #
|
||||
def test_match_adapter_pairs_and_clamps():
|
||||
assert hw._match_adapter_used_to_devices([40 * GB, 0.5 * GB], [48 * GB, 8 * GB]) == [
|
||||
40 * GB,
|
||||
0.5 * GB,
|
||||
]
|
||||
assert hw._match_adapter_used_to_devices([100 * GB], [48 * GB]) == [48 * GB] # clamp
|
||||
assert hw._match_adapter_used_to_devices([40 * GB], [48 * GB, 8 * GB]) == [40 * GB, None]
|
||||
|
||||
|
||||
def test_match_adapter_reports_unknown_when_more_active_than_visible():
|
||||
# More adapters actively using VRAM than are visible (a GPU outside the mask):
|
||||
# attribution would fabricate a value, so report unknown for every device.
|
||||
assert hw._match_adapter_used_to_devices([40 * GB, 0.5 * GB], [8 * GB]) == [None]
|
||||
|
||||
|
||||
def test_match_adapter_reports_unknown_when_hidden_high_use_adapter_survives_filter():
|
||||
# Idle 8 GiB card (10 MiB noise) beside a hidden 48 GiB card at 40 GiB: the
|
||||
# 40 GiB can't fit the 8 GiB device, so clamping there would fabricate. Unknown.
|
||||
assert hw._match_adapter_used_to_devices([40 * GB, 10 * MiB], [8 * GB]) == [None]
|
||||
# Order of the counters must not matter.
|
||||
assert hw._match_adapter_used_to_devices([10 * MiB, 40 * GB], [8 * GB]) == [None]
|
||||
|
||||
|
||||
def test_match_adapter_reports_unknown_for_placeholder_fallback():
|
||||
# Every counter below the 64 MiB floor plus a placeholder: no LUID-to-ordinal
|
||||
# mapping tells placeholder from idle GPU, so report unknown, not fabricate.
|
||||
# Single visible 8 GiB card idle (10 MiB) beside a 50 MiB placeholder counter.
|
||||
assert hw._match_adapter_used_to_devices([50 * MiB, 10 * MiB], [8 * GB]) == [None]
|
||||
# Order of the counters must not matter.
|
||||
assert hw._match_adapter_used_to_devices([10 * MiB, 50 * MiB], [8 * GB]) == [None]
|
||||
# Two idle visible GPUs plus a placeholder: all three counters below the floor.
|
||||
assert hw._match_adapter_used_to_devices([50 * MiB, 10 * MiB, 5 * MiB], [48 * GB, 8 * GB]) == [
|
||||
None,
|
||||
None,
|
||||
]
|
||||
|
||||
|
||||
def test_match_adapter_reports_unknown_when_usage_not_capacity_ordered():
|
||||
# 8 GiB card at 7 GiB beside a 48 GiB card at 5 GiB: the bigger usage still fits
|
||||
# the smaller card, so both pairings are feasible -> unknown.
|
||||
assert hw._match_adapter_used_to_devices([7 * GB, 5 * GB], [8 * GB, 48 * GB]) == [None, None]
|
||||
# Device order must not matter (same physical situation, ordinals flipped).
|
||||
assert hw._match_adapter_used_to_devices([7 * GB, 5 * GB], [48 * GB, 8 * GB]) == [None, None]
|
||||
# Same-capacity cards with unequal usage are equally unattributable.
|
||||
assert hw._match_adapter_used_to_devices([12 * GB, 8 * GB], [24 * GB, 24 * GB]) == [None, None]
|
||||
# A single usage that fits both cards can sit on either -> unknown.
|
||||
assert hw._match_adapter_used_to_devices([5 * GB], [48 * GB, 8 * GB]) == [None, None]
|
||||
# But a capacity-forced assignment (usage exceeds the smaller card) is kept:
|
||||
# 40 GiB can only be the 48 GiB card, so it is not fabrication.
|
||||
assert hw._match_adapter_used_to_devices([40 * GB], [48 * GB, 8 * GB]) == [40 * GB, None]
|
||||
|
||||
|
||||
def test_match_adapter_reports_unknown_when_hidden_usage_fits_visible_card():
|
||||
# A survivor that merely *fits* a visible card must not be pinned onto it. Two
|
||||
# cards (48/8 GiB) at 40 GiB / 10 MiB beside a hidden 6 GiB adapter: the 6 GiB
|
||||
# fits the idle 8 GiB card but isn't forced -> Unknown; only 40 GiB is forced.
|
||||
assert hw._match_adapter_used_to_devices([40 * GB, 10 * MiB, 6 * GB], [48 * GB, 8 * GB]) == [
|
||||
40 * GB,
|
||||
None,
|
||||
]
|
||||
# Counter order must not matter.
|
||||
assert hw._match_adapter_used_to_devices([6 * GB, 40 * GB, 10 * MiB], [48 * GB, 8 * GB]) == [
|
||||
40 * GB,
|
||||
None,
|
||||
]
|
||||
# A single visible card with a hidden adapter is never attributable: a fitting
|
||||
# survivor could be the hidden GPU's while the visible card is idle.
|
||||
assert hw._match_adapter_used_to_devices([6 * GB, 10 * MiB], [8 * GB]) == [None]
|
||||
|
||||
|
||||
def test_match_adapter_capacity_forced_matrix():
|
||||
"""Exhaustive hidden-adapter matrix for the capacity-forced rule.
|
||||
|
||||
A value is emitted only when the supra-threshold counters number exactly the
|
||||
visible devices AND a device's ranked usage strictly exceeds every smaller
|
||||
card's capacity. Otherwise (a visible card idle, a merely-fitting usage, or the
|
||||
smallest card) every device reports unknown.
|
||||
"""
|
||||
m = hw._match_adapter_used_to_devices
|
||||
# -- exactly-n supra-threshold counters, capacity-forced survivors are kept - #
|
||||
# Both visible cards have a real reading (the 3 MiB is a placeholder): 40 GiB
|
||||
# forced onto the 48 GiB card, 0.5 GiB not forced -> None.
|
||||
assert m([40 * GB, 0.5 * GB, 3 * MiB], [48 * GB, 8 * GB]) == [40 * GB, None]
|
||||
# Three visible cards all active (supra-threshold) + placeholder: 40 > 24 and
|
||||
# 20 > 8, both forced; the 8 GiB card is not forced -> None.
|
||||
assert m([40 * GB, 20 * GB, 5 * GB, 3 * MiB], [48 * GB, 24 * GB, 8 * GB]) == [
|
||||
40 * GB,
|
||||
20 * GB,
|
||||
None,
|
||||
]
|
||||
# -- fewer supra-threshold counters than visible cards -> all unknown ------ #
|
||||
# A visible card is idle, so even a "forced" 40 could be the hidden GPU's.
|
||||
assert m([40 * GB, 3 * MiB, 3 * MiB], [48 * GB, 8 * GB]) == [None, None]
|
||||
assert m([40 * GB, 10 * MiB, 10 * MiB], [48 * GB, 8 * GB]) == [None, None]
|
||||
assert m([40 * GB, 20 * GB, 3 * MiB, 3 * MiB], [48 * GB, 24 * GB, 8 * GB]) == [
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
]
|
||||
# Middle usage (6 GiB) fits both the 24 and 8 GiB cards, and only two cards are
|
||||
# active for three visible -> not a bijection -> all unknown.
|
||||
assert m([40 * GB, 6 * GB, 3 * MiB, 3 * MiB], [48 * GB, 24 * GB, 8 * GB]) == [
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
]
|
||||
# -- hidden larger than every visible card -> all unknown ----------------- #
|
||||
assert m([40 * GB, 10 * MiB], [8 * GB]) == [None]
|
||||
assert m([48 * GB, 3 * MiB, 3 * MiB], [24 * GB, 8 * GB]) == [None, None]
|
||||
# -- more active adapters than visible cards -> all unknown --------------- #
|
||||
assert m([40 * GB, 7 * GB, 6 * GB, 3 * MiB], [48 * GB, 8 * GB]) == [None, None]
|
||||
assert m([40 * GB, 7 * GB, 6 * GB, 3 * MiB, 3 * MiB], [48 * GB, 8 * GB]) == [None, None]
|
||||
# -- every counter below the noise floor (placeholder fallback) -> unknown - #
|
||||
assert m([50 * MiB, 10 * MiB], [8 * GB]) == [None]
|
||||
assert m([50 * MiB, 10 * MiB, 5 * MiB], [48 * GB, 8 * GB]) == [None, None]
|
||||
# -- equal-capacity cards with a hidden adapter: nothing is forced -------- #
|
||||
assert m([40 * GB, 40 * GB, 3 * MiB], [48 * GB, 48 * GB]) == [None, None]
|
||||
assert m([40 * GB, 30 * GB, 3 * MiB], [48 * GB, 48 * GB]) == [None, None]
|
||||
|
||||
|
||||
def test_perf_counter_parser_and_sentinel(monkeypatch):
|
||||
monkeypatch.setattr(hw.platform, "system", lambda: "Windows")
|
||||
monkeypatch.setattr(
|
||||
hw.subprocess, "run", _subprocess_run(adapter_output = _adapter_output(REPORTER_ADAPTERS))
|
||||
)
|
||||
parsed = hw._rocm_windows_perf_counter_vram_by_adapter()
|
||||
assert parsed is not None and len(parsed) == 3
|
||||
assert parsed[0][0].startswith("luid_")
|
||||
monkeypatch.setattr(hw.subprocess, "run", _subprocess_run(adapter_output = "__NONE__\n"))
|
||||
assert hw._rocm_windows_perf_counter_vram_by_adapter() is None
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- #
|
||||
# Unified-memory (Strix Halo APU) total reconciliation (Codex #7238)
|
||||
# ----------------------------------------------------------------------------- #
|
||||
def test_unified_memory_adopts_torch_total_even_when_used_unknown():
|
||||
"""Windows ROCm unified-memory APU: torch's used is None but its total (the full
|
||||
GTT pool) is authoritative. The correction must still adopt the larger total;
|
||||
used stays at amd-smi's figure when torch's is unknown."""
|
||||
metrics = {"vram_total_gb": 8.0, "vram_used_gb": 2.0, "vram_utilization_pct": 25.0}
|
||||
hw._apply_unified_memory_correction(metrics, {"total_gb": 124.0, "used_gb": None, "index": 0})
|
||||
assert metrics["vram_total_gb"] == 124.0 # full unified pool, not the 8 GB carve-out
|
||||
assert metrics["vram_used_gb"] == 2.0 # amd-smi used preserved (torch's was None)
|
||||
assert metrics["vram_utilization_pct"] == pytest.approx(round(2.0 / 124.0 * 100, 1))
|
||||
|
||||
|
||||
def test_unified_memory_overwrites_used_when_torch_used_known():
|
||||
"""When torch reports both a larger total and a known used, both are adopted
|
||||
and utilization is recomputed against the corrected total (unchanged path)."""
|
||||
metrics = {"vram_total_gb": 8.0, "vram_used_gb": 2.0, "vram_utilization_pct": 25.0}
|
||||
hw._apply_unified_memory_correction(metrics, {"total_gb": 124.0, "used_gb": 40.0, "index": 0})
|
||||
assert metrics["vram_total_gb"] == 124.0
|
||||
assert metrics["vram_used_gb"] == 40.0
|
||||
assert metrics["vram_utilization_pct"] == pytest.approx(round(40.0 / 124.0 * 100, 1))
|
||||
|
||||
|
||||
def test_unified_memory_no_op_when_torch_total_not_larger():
|
||||
"""A discrete GPU where torch total does not exceed amd-smi's is left untouched."""
|
||||
metrics = {"vram_total_gb": 48.0, "vram_used_gb": 10.0, "vram_utilization_pct": 20.8}
|
||||
hw._apply_unified_memory_correction(metrics, {"total_gb": 48.0, "used_gb": None, "index": 0})
|
||||
assert metrics["vram_total_gb"] == 48.0
|
||||
assert metrics["vram_used_gb"] == 10.0
|
||||
assert metrics["vram_utilization_pct"] == 20.8
|
||||
|
|
@ -538,21 +538,31 @@ def _torch_get_physical_gpu_count() -> Optional[int]:
|
|||
|
||||
|
||||
def _torch_get_per_device_info(device_indices: list[int]) -> list[Dict[str, Any]]:
|
||||
"""Query torch for per-GPU name, total VRAM, and used VRAM."""
|
||||
"""Query torch for per-GPU name, total VRAM, and used VRAM.
|
||||
|
||||
``used_gb`` is ``None`` on Windows ROCm when ``hipMemGetInfo`` reports
|
||||
``free == total`` (ROCm/ROCm#1909): that 0 means unknown, not empty.
|
||||
"""
|
||||
mod, _ = _torch_get_device_module()
|
||||
if mod is None:
|
||||
return []
|
||||
|
||||
# free==total is a Windows-ROCm-only quirk.
|
||||
_win_rocm = sys.platform == "win32" and IS_ROCM
|
||||
devices = []
|
||||
for ordinal, phys_idx in enumerate(device_indices):
|
||||
try:
|
||||
# torch ordinals are 0-based relative to CUDA_VISIBLE_DEVICES.
|
||||
props = mod.get_device_properties(ordinal)
|
||||
total_bytes = props.total_memory
|
||||
used_bytes: Optional[int]
|
||||
# Prefer mem_get_info (system-wide) so auto-select sees other consumers.
|
||||
if hasattr(mod, "mem_get_info"):
|
||||
free_bytes, total_bytes = mod.mem_get_info(ordinal)
|
||||
used_bytes = total_bytes - free_bytes
|
||||
# free==total is the broken-API sentinel, not an idle GPU.
|
||||
if _win_rocm and free_bytes == total_bytes:
|
||||
used_bytes = None
|
||||
else:
|
||||
used_bytes = mod.memory_allocated(ordinal)
|
||||
devices.append(
|
||||
|
|
@ -561,7 +571,7 @@ def _torch_get_per_device_info(device_indices: list[int]) -> list[Dict[str, Any]
|
|||
"visible_ordinal": ordinal,
|
||||
"name": props.name,
|
||||
"total_gb": round(total_bytes / (1024**3), 2),
|
||||
"used_gb": round(used_bytes / (1024**3), 2),
|
||||
"used_gb": round(used_bytes / (1024**3), 2) if used_bytes is not None else None,
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
|
|
@ -724,20 +734,30 @@ def _rocm_linux_sysfs_vram_gb() -> tuple[Optional[float], Optional[float]]:
|
|||
return None, None
|
||||
|
||||
|
||||
def _rocm_windows_perf_counter_vram_gb() -> tuple[Optional[float], Optional[float]]:
|
||||
"""Query system-wide dedicated GPU VRAM via Windows Performance Counters.
|
||||
# ── Windows AMD/ROCm per-adapter VRAM (issue #7072) ──────────────────────────
|
||||
# amd-smi is disabled and hipMemGetInfo reports free==total, so read used from the
|
||||
# per-LUID "GPU Adapter Memory" perf counters and take each total from torch, so
|
||||
# every GPU shows instead of one fake device with GPU 0's total.
|
||||
# Placeholder adapters (Basic Render Driver / idle iGPU) drop only when they would
|
||||
# outnumber the real torch devices.
|
||||
_ROCM_WIN_ADAPTER_MIN_BYTES = 64 * 1024 * 1024 # 64 MiB
|
||||
|
||||
Same data source as Task Manager, so cross-process usage is accurate.
|
||||
Works for any GPU vendor without amd-smi or nvidia-smi.
|
||||
Returns (used_gb, total_gb) or (None, None) on failure.
|
||||
|
||||
def _rocm_windows_perf_counter_vram_by_adapter() -> Optional[list[tuple[str, float]]]:
|
||||
"""Per-adapter dedicated VRAM usage on Windows via Performance Counters.
|
||||
|
||||
Returns ``[(instance_name, used_bytes)]`` (one per LUID-named adapter), or
|
||||
``None`` when the counter is unavailable/localized/empty so callers fall back.
|
||||
"""
|
||||
if platform.system() != "Windows":
|
||||
return None, None
|
||||
return None
|
||||
try:
|
||||
# Emit "<InstanceName>|<CookedValue>" per sample, or a __NONE__ sentinel.
|
||||
ps = (
|
||||
"$s=(Get-Counter '\\GPU Adapter Memory(*)\\Dedicated Usage'"
|
||||
" -ErrorAction SilentlyContinue).CounterSamples;"
|
||||
"if($s){($s|Measure-Object CookedValue -Sum).Sum}else{-1}"
|
||||
"if($s){$s|ForEach-Object{'{0}|{1}' -f $_.InstanceName,[int64]$_.CookedValue}}"
|
||||
"else{'__NONE__'}"
|
||||
)
|
||||
r = subprocess.run(
|
||||
["powershell", "-NoProfile", "-NonInteractive", "-Command", ps],
|
||||
|
|
@ -746,16 +766,167 @@ def _rocm_windows_perf_counter_vram_gb() -> tuple[Optional[float], Optional[floa
|
|||
timeout = 5,
|
||||
)
|
||||
if r.returncode != 0 or not r.stdout.strip():
|
||||
return None, None
|
||||
used_bytes = float(r.stdout.strip())
|
||||
if used_bytes < 0:
|
||||
return None, None
|
||||
import torch as _torch
|
||||
|
||||
total_bytes = _torch.cuda.get_device_properties(0).total_memory
|
||||
return round(used_bytes / (1024**3), 2), round(total_bytes / (1024**3), 2)
|
||||
return None
|
||||
adapters: list[tuple[str, float]] = []
|
||||
for line in r.stdout.splitlines():
|
||||
line = line.strip()
|
||||
if not line or line == "__NONE__" or "|" not in line:
|
||||
continue
|
||||
instance, _, raw = line.rpartition("|")
|
||||
try:
|
||||
used = float(raw.strip())
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
if used < 0:
|
||||
continue
|
||||
adapters.append((instance.strip(), used))
|
||||
return adapters or None
|
||||
except Exception:
|
||||
return None, None
|
||||
return None
|
||||
|
||||
|
||||
def _match_adapter_used_to_devices(
|
||||
adapter_useds: list[float], device_totals: list[float]
|
||||
) -> list[Optional[float]]:
|
||||
"""Attribute per-adapter used bytes to torch devices by capacity ranking.
|
||||
|
||||
Windows shares no key between LUID counters and torch ordinals, so usages are
|
||||
ranked against device totals and each is trusted only when capacity *forces* it
|
||||
(it exceeds every smaller device); an ambiguous ranking reports unknown
|
||||
(``None``) rather than fabricate a per-index free.
|
||||
|
||||
Extra counters mean a hidden/display adapter, and the noise filter may have
|
||||
dropped a real reading, so values are emitted only when the supra-threshold
|
||||
counters number EXACTLY the visible devices AND capacity forces the mapping;
|
||||
otherwise every device is unknown. Best-effort but correct for the common
|
||||
loaded-card case (#7072). Returns a list aligned to ``device_totals``.
|
||||
"""
|
||||
n = len(device_totals)
|
||||
if n == 0:
|
||||
return []
|
||||
useds = sorted(adapter_useds, reverse = True)
|
||||
ranked_positions = sorted(range(n), key = lambda i: -device_totals[i])
|
||||
ranked_totals = [device_totals[pos] for pos in ranked_positions]
|
||||
assigned: list[Optional[float]]
|
||||
# More counters than devices -> a hidden/display adapter (check before noise filter).
|
||||
if len(useds) > n:
|
||||
non_trivial = [u for u in useds if u >= _ROCM_WIN_ADAPTER_MIN_BYTES]
|
||||
if len(non_trivial) != n:
|
||||
# Not a clean bijection (a masked GPU is busy or a visible card idle):
|
||||
# no counter maps to a specific card, so report unknown.
|
||||
return [None] * n
|
||||
# Exactly n supra-threshold counters: extras were placeholders, so a
|
||||
# capacity-ranked bijection is plausible.
|
||||
useds = non_trivial
|
||||
ranked_useds = [useds[rank] for rank in range(n)]
|
||||
# A usage above its ranked capacity is a hidden larger GPU; clamping onto the
|
||||
# smaller card would fabricate a fully-used reading.
|
||||
for rank in range(n):
|
||||
if ranked_useds[rank] > ranked_totals[rank]:
|
||||
return [None] * n
|
||||
# Capacity forces the mapping only when the usage exceeds the next-smaller
|
||||
# capacity; the smallest card and merely-fitting usages stay unknown.
|
||||
# Keeps 40 GiB over 48/8 GiB -> [40, None].
|
||||
assigned = [None] * n
|
||||
for rank, pos in enumerate(ranked_positions):
|
||||
if rank + 1 < n and ranked_useds[rank] > ranked_totals[rank + 1]:
|
||||
assigned[pos] = min(ranked_useds[rank], device_totals[pos])
|
||||
return assigned
|
||||
# No hidden adapters: every counter is a visible card, so ranking is a permutation.
|
||||
ranked_useds = [useds[rank] if rank < len(useds) else 0.0 for rank in range(n)]
|
||||
# Ambiguous if a strictly larger usage also fits the next smaller card: the two
|
||||
# could be swapped without breaking capacity, so ranking can't tell them apart.
|
||||
for rank in range(n - 1):
|
||||
upper, lower = ranked_useds[rank], ranked_useds[rank + 1]
|
||||
if upper > lower and upper <= ranked_totals[rank + 1]:
|
||||
return [None] * n
|
||||
assigned = [None] * n
|
||||
for rank, pos in enumerate(ranked_positions):
|
||||
if rank < len(useds):
|
||||
assigned[pos] = min(useds[rank], device_totals[pos])
|
||||
return assigned
|
||||
|
||||
|
||||
def _rocm_windows_per_device_vram(device_indices: list[int]) -> list[Dict[str, Any]]:
|
||||
"""Per-GPU VRAM on Windows AMD/ROCm: total from torch properties (reliable),
|
||||
used from the per-adapter Dedicated Usage counter.
|
||||
|
||||
Returns ``{index, visible_ordinal, name, used_gb, total_gb}`` per visible GPU
|
||||
(``used_gb`` may be ``None`` when the counter is unavailable), or ``[]`` when
|
||||
torch can't enumerate devices so callers fall through to the torch last resort.
|
||||
"""
|
||||
if platform.system() != "Windows":
|
||||
return []
|
||||
mod, _ = _torch_get_device_module()
|
||||
if mod is None:
|
||||
return []
|
||||
# Totals/names from torch properties (mem_get_info's free==total quirk zeroes used).
|
||||
dev_meta: list[Dict[str, Any]] = []
|
||||
for ordinal, phys_idx in enumerate(device_indices):
|
||||
try:
|
||||
props = mod.get_device_properties(ordinal)
|
||||
dev_meta.append(
|
||||
{
|
||||
"index": phys_idx,
|
||||
"visible_ordinal": ordinal,
|
||||
"name": props.name,
|
||||
"total_bytes": int(props.total_memory),
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug("torch property probe failed for ordinal %d: %s", ordinal, e)
|
||||
if not dev_meta:
|
||||
return []
|
||||
|
||||
adapters = _rocm_windows_perf_counter_vram_by_adapter()
|
||||
if adapters:
|
||||
assigned = _match_adapter_used_to_devices(
|
||||
[used for _, used in adapters],
|
||||
[d["total_bytes"] for d in dev_meta],
|
||||
)
|
||||
else:
|
||||
# Counter unavailable: show every GPU with a correct total, used unknown.
|
||||
assigned = [None] * len(dev_meta)
|
||||
|
||||
devices: list[Dict[str, Any]] = []
|
||||
for meta, used_bytes in zip(dev_meta, assigned):
|
||||
total_gb = round(meta["total_bytes"] / (1024**3), 2)
|
||||
used_gb = round(used_bytes / (1024**3), 2) if used_bytes is not None else None
|
||||
devices.append(
|
||||
{
|
||||
"index": meta["index"],
|
||||
"visible_ordinal": meta["visible_ordinal"],
|
||||
"name": meta["name"],
|
||||
"used_gb": used_gb,
|
||||
"total_gb": total_gb,
|
||||
}
|
||||
)
|
||||
return devices
|
||||
|
||||
|
||||
def _rocm_windows_device_payload_entry(
|
||||
device: DeviceType, dev: Dict[str, Any], gpu_util_pct: Optional[float]
|
||||
) -> Dict[str, Any]:
|
||||
"""Build a ``get_gpu_utilization`` device entry from a per-device VRAM dict."""
|
||||
total_gb = dev["total_gb"]
|
||||
used_gb = dev["used_gb"]
|
||||
return {
|
||||
"available": True,
|
||||
"backend": _backend_label(device),
|
||||
"index": dev["index"],
|
||||
"visible_ordinal": dev["visible_ordinal"],
|
||||
"name": dev.get("name", "Unknown"),
|
||||
"gpu_utilization_pct": gpu_util_pct,
|
||||
"temperature_c": None,
|
||||
"vram_used_gb": used_gb,
|
||||
"vram_total_gb": total_gb,
|
||||
"vram_utilization_pct": round((used_gb / total_gb) * 100, 1)
|
||||
if total_gb and total_gb > 0 and used_gb is not None
|
||||
else None,
|
||||
"power_draw_w": None,
|
||||
"power_limit_w": None,
|
||||
"power_utilization_pct": None,
|
||||
}
|
||||
|
||||
|
||||
def _gpu_utilization_payload(
|
||||
|
|
@ -821,30 +992,24 @@ def get_gpu_utilization() -> Dict[str, Any]:
|
|||
index_kind = result.get("index_kind"),
|
||||
)
|
||||
|
||||
# Fallback Windows ROCm
|
||||
# Fallback Windows ROCm: per-adapter VRAM attribution (issue #7072), so
|
||||
# every visible GPU is shown instead of a sum collapsed onto one device.
|
||||
if IS_ROCM and platform.system() == "Windows":
|
||||
_win_used, _win_total = _rocm_windows_perf_counter_vram_gb()
|
||||
if _win_used is not None and _win_total is not None:
|
||||
_win_util = _rocm_windows_perf_counter_gpu_util_pct()
|
||||
_win_ids = _get_parent_visible_gpu_spec().get("numeric_ids")
|
||||
if not _win_ids:
|
||||
_win_ids = list(range(_torch_get_physical_gpu_count() or 0))
|
||||
_win_devices = _rocm_windows_per_device_vram(_win_ids)
|
||||
if _win_devices:
|
||||
# A single visible GPU can own the aggregate 3D-engine utilization;
|
||||
# across several GPUs the sum isn't per-device, so leave it unset.
|
||||
_win_util = (
|
||||
_rocm_windows_perf_counter_gpu_util_pct() if len(_win_devices) == 1 else None
|
||||
)
|
||||
return _gpu_utilization_payload(
|
||||
device,
|
||||
[
|
||||
{
|
||||
"available": True,
|
||||
"backend": _backend_label(device),
|
||||
"index": 0,
|
||||
"visible_ordinal": 0,
|
||||
"gpu_utilization_pct": _win_util,
|
||||
"temperature_c": None,
|
||||
"vram_used_gb": _win_used,
|
||||
"vram_total_gb": _win_total,
|
||||
"vram_utilization_pct": round((_win_used / _win_total) * 100, 1)
|
||||
if _win_total > 0
|
||||
else None,
|
||||
"power_draw_w": None,
|
||||
"power_limit_w": None,
|
||||
"power_utilization_pct": None,
|
||||
}
|
||||
_rocm_windows_device_payload_entry(device, _wd, _win_util)
|
||||
for _wd in _win_devices
|
||||
],
|
||||
)
|
||||
|
||||
|
|
@ -901,7 +1066,7 @@ def get_gpu_utilization() -> Dict[str, Any]:
|
|||
"vram_used_gb": _used,
|
||||
"vram_total_gb": _total,
|
||||
"vram_utilization_pct": round((_used / _total) * 100, 1)
|
||||
if _total > 0
|
||||
if _total > 0 and _used is not None
|
||||
else None,
|
||||
"power_draw_w": None,
|
||||
"power_limit_w": None,
|
||||
|
|
@ -995,19 +1160,27 @@ def _apply_unified_memory_correction(
|
|||
endpoints stay in sync on AMD iGPUs with unified memory.
|
||||
"""
|
||||
torch_total_gb = torch_info["total_gb"]
|
||||
torch_used_gb = torch_info.get("used_gb")
|
||||
smi_total_gb = device_metrics.get("vram_total_gb") or 0.0
|
||||
# torch sees the full unified (GTT) pool; amd-smi only the dedicated carve-out.
|
||||
# Adopt torch's larger total regardless of used: on Windows ROCm torch_used is
|
||||
# None (free==total sentinel) but its total stays authoritative. Overwrite used
|
||||
# only when torch's is known, then recompute utilization against whatever remains.
|
||||
if torch_total_gb > smi_total_gb:
|
||||
torch_used_gb = torch_info["used_gb"]
|
||||
device_metrics["vram_total_gb"] = torch_total_gb
|
||||
device_metrics["vram_used_gb"] = torch_used_gb
|
||||
if torch_used_gb is not None:
|
||||
device_metrics["vram_used_gb"] = torch_used_gb
|
||||
_used_for_pct = device_metrics.get("vram_used_gb")
|
||||
device_metrics["vram_utilization_pct"] = (
|
||||
round((torch_used_gb / torch_total_gb) * 100, 1) if torch_total_gb > 0 else None
|
||||
round((_used_for_pct / torch_total_gb) * 100, 1)
|
||||
if torch_total_gb > 0 and _used_for_pct is not None
|
||||
else None
|
||||
)
|
||||
logger.debug(
|
||||
"ROCm unified memory: replaced amd-smi VRAM (%.2f GB) with "
|
||||
"torch mem_get_info total (%.2f GB) for device %s",
|
||||
smi_total_gb,
|
||||
"ROCm unified memory: adopted torch mem_get_info total (%.2f GB) over "
|
||||
"amd-smi (%.2f GB) for device %s",
|
||||
torch_total_gb,
|
||||
smi_total_gb,
|
||||
torch_info.get("index"),
|
||||
)
|
||||
|
||||
|
|
@ -1067,6 +1240,49 @@ def get_visible_gpu_utilization() -> Dict[str, Any]:
|
|||
_reconcile_rocm_unified_memory(result, numeric_ids)
|
||||
return result
|
||||
|
||||
# Windows AMD/ROCm (issue #7072): the System tab's VRAM source. The torch
|
||||
# fallback below would report used==0 (free==total), so read per-adapter
|
||||
# Dedicated Usage instead; total from torch properties.
|
||||
if IS_ROCM and platform.system() == "Windows":
|
||||
win_numeric_ids = parent_visible_spec.get("numeric_ids")
|
||||
if win_numeric_ids:
|
||||
win_ids = win_numeric_ids
|
||||
win_index_kind = "physical"
|
||||
else:
|
||||
win_ids = list(range(_torch_get_physical_gpu_count() or 0))
|
||||
win_index_kind = "relative"
|
||||
win_devices = _rocm_windows_per_device_vram(win_ids)
|
||||
if win_devices:
|
||||
devices = []
|
||||
for wd in win_devices:
|
||||
total = wd["total_gb"]
|
||||
used = wd["used_gb"]
|
||||
devices.append(
|
||||
{
|
||||
"index": wd["index"],
|
||||
"index_kind": win_index_kind,
|
||||
"visible_ordinal": wd["visible_ordinal"],
|
||||
"name": wd.get("name"),
|
||||
"gpu_utilization_pct": None,
|
||||
"temperature_c": None,
|
||||
"vram_used_gb": used,
|
||||
"vram_total_gb": total,
|
||||
"vram_utilization_pct": round((used / total) * 100, 1)
|
||||
if total and total > 0 and used is not None
|
||||
else None,
|
||||
"power_draw_w": None,
|
||||
"power_limit_w": None,
|
||||
"power_utilization_pct": None,
|
||||
}
|
||||
)
|
||||
return {
|
||||
"available": True,
|
||||
"backend": _backend_label(device),
|
||||
"parent_visible_gpu_ids": win_numeric_ids or [],
|
||||
"devices": devices,
|
||||
"index_kind": win_index_kind,
|
||||
}
|
||||
|
||||
# Torch-based fallback for CUDA (nvidia-smi unavailable, AMD ROCm) and XPU (Intel)
|
||||
if device in (DeviceType.CUDA, DeviceType.XPU):
|
||||
parent_ids = get_parent_visible_gpu_ids()
|
||||
|
|
@ -1094,7 +1310,7 @@ def get_visible_gpu_utilization() -> Dict[str, Any]:
|
|||
"vram_used_gb": used,
|
||||
"vram_total_gb": total,
|
||||
"vram_utilization_pct": round((used / total) * 100, 1)
|
||||
if total > 0
|
||||
if total > 0 and used is not None
|
||||
else None,
|
||||
"power_draw_w": None,
|
||||
"power_limit_w": None,
|
||||
|
|
|
|||
|
|
@ -70,13 +70,18 @@ export function FloatingMonitor() {
|
|||
(sum, device) => sum + (device.memory_total_gb ?? 0),
|
||||
0,
|
||||
);
|
||||
const vramUsed = devices.reduce(
|
||||
(sum, device) => sum + (device.vram_used_gb ?? 0),
|
||||
0,
|
||||
);
|
||||
// null usage = unknown (e.g. Windows ROCm perf counter): treating it as 0
|
||||
// fabricates a 0-used readout, so the aggregate is unknown if any device is.
|
||||
const vramUsageKnown =
|
||||
devices.length > 0 &&
|
||||
devices.every((device) => Number.isFinite(device.vram_used_gb));
|
||||
const vramUsed = vramUsageKnown
|
||||
? devices.reduce((sum, device) => sum + (device.vram_used_gb ?? 0), 0)
|
||||
: 0;
|
||||
const vramPercent = clampPercent(
|
||||
vramTotal > 0 ? (vramUsed / vramTotal) * 100 : 0,
|
||||
vramUsageKnown && vramTotal > 0 ? (vramUsed / vramTotal) * 100 : 0,
|
||||
);
|
||||
const unknownLabel = t("settings.resources.environment.unknown");
|
||||
|
||||
const hasGpu = (systemInfo.gpu?.available ?? false) && devices.length > 0;
|
||||
|
||||
|
|
@ -164,17 +169,20 @@ export function FloatingMonitor() {
|
|||
<span
|
||||
className={cn(
|
||||
"shrink-0 tabular-nums",
|
||||
usageTextClass(vramPercent),
|
||||
vramUsageKnown
|
||||
? usageTextClass(vramPercent)
|
||||
: "text-muted-foreground",
|
||||
)}
|
||||
>
|
||||
{Math.round(vramPercent)}%
|
||||
{vramUsageKnown ? `${Math.round(vramPercent)}%` : "--"}
|
||||
</span>
|
||||
</div>
|
||||
<div className="text-xs text-muted-foreground font-mono tabular-nums">
|
||||
{formatGiB(vramUsed)} / {formatGiB(vramTotal)}
|
||||
{vramUsageKnown ? formatGiB(vramUsed) : unknownLabel} /{" "}
|
||||
{formatGiB(vramTotal)}
|
||||
</div>
|
||||
<Progress
|
||||
value={vramPercent}
|
||||
value={vramUsageKnown ? vramPercent : 0}
|
||||
className="mt-1 h-1.5 rounded-full bg-muted"
|
||||
indicatorClassName={usageIndicatorClass(vramPercent)}
|
||||
/>
|
||||
|
|
|
|||
|
|
@ -90,8 +90,11 @@ function MetricTile({
|
|||
label: string;
|
||||
value: string;
|
||||
detail: string;
|
||||
percent: number;
|
||||
// null = usage unknown (e.g. Windows ROCm perf counter): show a dash and
|
||||
// empty bar rather than a fabricated 0%.
|
||||
percent: number | null;
|
||||
}) {
|
||||
const percentKnown = isFiniteNumber(percent);
|
||||
const safePercent = clampPercent(percent);
|
||||
return (
|
||||
<div className="flex min-w-0 flex-col gap-2 rounded-md border border-border/60 bg-muted/20 p-3">
|
||||
|
|
@ -102,10 +105,10 @@ function MetricTile({
|
|||
<span
|
||||
className={cn(
|
||||
"shrink-0 font-mono text-xs tabular-nums",
|
||||
usageTextClass(safePercent),
|
||||
percentKnown ? usageTextClass(safePercent) : "text-muted-foreground",
|
||||
)}
|
||||
>
|
||||
{formatPercent(safePercent)}
|
||||
{percentKnown ? formatPercent(safePercent) : "--"}
|
||||
</span>
|
||||
</div>
|
||||
<div className="min-w-0">
|
||||
|
|
@ -117,7 +120,7 @@ function MetricTile({
|
|||
</div>
|
||||
</div>
|
||||
<Progress
|
||||
value={safePercent}
|
||||
value={percentKnown ? safePercent : 0}
|
||||
aria-label={label}
|
||||
className="h-1.5 rounded-full bg-muted"
|
||||
indicatorClassName={usageIndicatorClass(safePercent)}
|
||||
|
|
@ -194,18 +197,30 @@ export function ResourcesTab() {
|
|||
(sum, device) => sum + (device.memory_total_gb ?? 0),
|
||||
0,
|
||||
);
|
||||
const vramUsed = devices.reduce(
|
||||
(sum, device) => sum + (device.vram_used_gb ?? 0),
|
||||
0,
|
||||
);
|
||||
const vramFree = devices.reduce(
|
||||
(sum, device) =>
|
||||
sum +
|
||||
(device.vram_free_gb ??
|
||||
Math.max(0, (device.memory_total_gb ?? 0) - (device.vram_used_gb ?? 0))),
|
||||
0,
|
||||
);
|
||||
const vramPercent = vramTotal > 0 ? (vramUsed / vramTotal) * 100 : 0;
|
||||
// null usage = unknown (e.g. Windows ROCm perf counter): treating it as 0
|
||||
// fabricates a 0-used total, so the aggregate is unknown if any device is.
|
||||
const vramUsageKnown =
|
||||
devices.length > 0 &&
|
||||
devices.every((device) => isFiniteNumber(device.vram_used_gb));
|
||||
const vramUsed = vramUsageKnown
|
||||
? devices.reduce((sum, device) => sum + (device.vram_used_gb ?? 0), 0)
|
||||
: null;
|
||||
const vramFree = vramUsageKnown
|
||||
? devices.reduce(
|
||||
(sum, device) =>
|
||||
sum +
|
||||
(device.vram_free_gb ??
|
||||
Math.max(
|
||||
0,
|
||||
(device.memory_total_gb ?? 0) - (device.vram_used_gb ?? 0),
|
||||
)),
|
||||
0,
|
||||
)
|
||||
: null;
|
||||
const vramPercent =
|
||||
vramUsageKnown && isFiniteNumber(vramUsed) && vramTotal > 0
|
||||
? (vramUsed / vramTotal) * 100
|
||||
: 0;
|
||||
|
||||
return {
|
||||
devices,
|
||||
|
|
@ -218,6 +233,7 @@ export function ResourcesTab() {
|
|||
vramUsed,
|
||||
vramFree,
|
||||
vramPercent,
|
||||
vramUsageKnown,
|
||||
};
|
||||
}, [systemInfo]);
|
||||
|
||||
|
|
@ -259,6 +275,7 @@ export function ResourcesTab() {
|
|||
: modelsFolderLoaded
|
||||
? t("settings.resources.environment.unknown")
|
||||
: t("common.loading");
|
||||
const unknownLabel = t("settings.resources.environment.unknown");
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-6">
|
||||
|
|
@ -327,17 +344,21 @@ export function ResourcesTab() {
|
|||
label={t("settings.resources.liveMonitor.vram")}
|
||||
value={
|
||||
hasGpu
|
||||
? `${formatGiB(metrics.vramUsed)} / ${formatGiB(metrics.vramTotal)}`
|
||||
? metrics.vramUsageKnown
|
||||
? `${formatGiB(metrics.vramUsed)} / ${formatGiB(metrics.vramTotal)}`
|
||||
: `${unknownLabel} / ${formatGiB(metrics.vramTotal)}`
|
||||
: t("settings.resources.liveMonitor.noGpu")
|
||||
}
|
||||
detail={
|
||||
hasGpu
|
||||
? t("settings.resources.liveMonitor.free", {
|
||||
value: formatGiB(metrics.vramFree),
|
||||
})
|
||||
? metrics.vramUsageKnown
|
||||
? t("settings.resources.liveMonitor.free", {
|
||||
value: formatGiB(metrics.vramFree),
|
||||
})
|
||||
: unknownLabel
|
||||
: backendLabel
|
||||
}
|
||||
percent={metrics.vramPercent}
|
||||
percent={metrics.vramUsageKnown ? metrics.vramPercent : null}
|
||||
/>
|
||||
</div>
|
||||
</SettingsSection>
|
||||
|
|
@ -346,13 +367,33 @@ export function ResourcesTab() {
|
|||
{hasGpu ? (
|
||||
metrics.devices.map((device, index) => {
|
||||
const ordinal = deviceOrdinal(device);
|
||||
const total = device.memory_total_gb ?? 0;
|
||||
const used = device.vram_used_gb ?? 0;
|
||||
const free = device.vram_free_gb ?? Math.max(0, total - used);
|
||||
// Preserve null (unknown, e.g. Windows ROCm perf counter); coercing
|
||||
// to 0 would render a fabricated 0 used / full free.
|
||||
const total = device.memory_total_gb ?? null;
|
||||
const used = device.vram_used_gb ?? null;
|
||||
const free =
|
||||
device.vram_free_gb ??
|
||||
(isFiniteNumber(total) && isFiniteNumber(used)
|
||||
? Math.max(0, total - used)
|
||||
: null);
|
||||
const percent =
|
||||
device.vram_utilization_pct ??
|
||||
(total > 0 ? (used / total) * 100 : null);
|
||||
(isFiniteNumber(total) && total > 0 && isFiniteNumber(used)
|
||||
? (used / total) * 100
|
||||
: null);
|
||||
const safePercent = clampPercent(percent);
|
||||
const usedText = isFiniteNumber(used)
|
||||
? formatGiB(used)
|
||||
: unknownLabel;
|
||||
const freeText = isFiniteNumber(free)
|
||||
? formatGiB(free)
|
||||
: unknownLabel;
|
||||
const totalText = isFiniteNumber(total)
|
||||
? formatGiB(total)
|
||||
: unknownLabel;
|
||||
const percentText = isFiniteNumber(percent)
|
||||
? formatPercent(safePercent)
|
||||
: unknownLabel;
|
||||
return (
|
||||
<div
|
||||
key={`${device.index ?? index}-${device.name ?? "gpu"}`}
|
||||
|
|
@ -374,7 +415,7 @@ export function ResourcesTab() {
|
|||
</div>
|
||||
<div className="shrink-0 font-mono text-xs tabular-nums text-muted-foreground">
|
||||
<span>
|
||||
{formatPercent(safePercent)}{" "}
|
||||
{percentText}{" "}
|
||||
{t("settings.resources.gpu.vramUtilization")}
|
||||
</span>
|
||||
</div>
|
||||
|
|
@ -382,17 +423,17 @@ export function ResourcesTab() {
|
|||
<div className="grid gap-1 text-xs text-muted-foreground sm:grid-cols-3 sm:gap-2">
|
||||
<span className="min-w-0 truncate font-mono tabular-nums">
|
||||
{t("settings.resources.gpu.used", {
|
||||
value: formatGiB(used),
|
||||
value: usedText,
|
||||
})}
|
||||
</span>
|
||||
<span className="min-w-0 truncate font-mono tabular-nums sm:text-center">
|
||||
{t("settings.resources.gpu.free", {
|
||||
value: formatGiB(free),
|
||||
value: freeText,
|
||||
})}
|
||||
</span>
|
||||
<span className="min-w-0 truncate font-mono tabular-nums sm:text-right">
|
||||
{t("settings.resources.gpu.total", {
|
||||
value: formatGiB(total),
|
||||
value: totalText,
|
||||
})}
|
||||
</span>
|
||||
</div>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue