* Vulkan GPUs: real device names and selectable ordinals
Rebases the durable half of #7356 onto the inference_gpu transport #7476
landed on main. Those two PRs solve an overlapping problem and disagree on
the data model, so merging #7356 as-is would ship two parallel Vulkan
device concepts with different index semantics. This keeps main's transport
and adds what #7356 had that #7476 does not.
- _vulkan_probe.py emits a 5th column, ggml's device description, sanitized
for the tab protocol and UTF-8 safe. Reader tolerates 4- or 5-column
output so an older probe still parses.
- llama_cpp gains _run_vulkan_probe (shared parse) and
vulkan_device_inventory (names + is_igpu + real totals).
- get_vulkan_inference_gpu_info reports the real name and an explicit
is_igpu instead of "Vulkan<i>" and a total == 0 guess.
- index_kind becomes "vulkan", not "relative", and gpu_ids picks are
supported on Vulkan builds once the probe enumerated ordinals. The XPU ban
no longer applies to them: a Vulkan pick is a ggml ordinal, not a torch-xpu
index, so it works on an Intel host too.
- Frontend picker reads the Vulkan inventory as the pickable set.
Memory deliberately still comes from _get_gpu_memory, not the inventory.
That path applies _apply_igpu_host_reserve_mib and zeroes a shared total;
budgeting an APU off its raw shared total would hand out the whole machine's
RAM with no OS headroom. Identity is joined onto it by ordinal, so a probe
failure degrades to Vulkan<i> names with the memory readings intact.
Dropped from #7356 as superseded: validate_vulkan_gpu_ids (main's
resolve_requested_gpu_ids already rejects duplicates and
_resolve_gguf_gpu_ids_for_request already probes for existence), the
gguf_devices transport, and the iGPU budget fallback in 71619891e, which
main's aggregateGpuMemoryTotalGb handles better by counting a shared pool
once.
Also keeps #7356's removal of the late diffusion raise, so the graceful
gpu_ids drop stays reachable for a GGUF only classified as diffusion after
download. #7415's real guard, _reject_vulkan_diffusion_gpu_ids_before_
teardown, is untouched.
Verified on Windows + Strix Halo: backend Vulkan/GPU-selection suites at the
same 4 pre-existing failures as main, tests/studio 1671 passed with no new
failures, frontend typecheck clean. Hardware confirmation of the underlying
behavior is on #7356 from @Bebiv24 (RX 9070 XT + RX 480).
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
da447d47ba
commit
74295d93d8
8 changed files with 297 additions and 70 deletions
|
|
@ -6,12 +6,14 @@
|
|||
Run in a short-lived subprocess (``python _vulkan_probe.py <bindir>``) so the
|
||||
Vulkan instance never lives in the long-running backend process. Loads the
|
||||
bundled ggml Vulkan backend from ``<bindir>`` and prints one
|
||||
``<idx>\\t<free_bytes>\\t<is_igpu>\\t<total_bytes>`` line per device to stdout.
|
||||
Indices are ggml's own Vulkan device ordinals, which need not match nvidia-smi
|
||||
order. ``is_igpu`` (from ggml's device type) is ``1`` for an integrated GPU
|
||||
sharing system RAM. ``total_bytes`` is the device-local heap; the reader uses
|
||||
it to reserve absolute headroom on a discrete card (parity with the CUDA/ROCm
|
||||
fit) and ignores it for an iGPU, whose "VRAM" is shared system RAM.
|
||||
``<idx>\\t<free_bytes>\\t<is_igpu>\\t<total_bytes>\\t<name>`` line per device to
|
||||
stdout. Indices are ggml's own Vulkan device ordinals, which need not match
|
||||
nvidia-smi order. ``is_igpu`` (from ggml's device type) is ``1`` for an
|
||||
integrated GPU sharing system RAM. ``total_bytes`` is the device-local heap;
|
||||
the reader uses it to reserve absolute headroom on a discrete card (parity
|
||||
with the CUDA/ROCm fit) and ignores it for an iGPU, whose "VRAM" is shared
|
||||
system RAM. ``name`` is ggml's device description (the marketing name, e.g.
|
||||
"AMD Radeon RX 9070 XT"); empty when the registry lookup fails.
|
||||
|
||||
Uses only the standard library so it stays runnable as a bare script.
|
||||
"""
|
||||
|
|
@ -24,15 +26,30 @@ import sys
|
|||
_GGML_BACKEND_DEVICE_TYPE_IGPU = 2
|
||||
|
||||
|
||||
def _igpu_flags(base, lib, count: int) -> list[bool]:
|
||||
"""Per-device integrated-GPU flags via ggml's backend registry.
|
||||
def _igpu_flags_and_names(base, lib, count: int) -> tuple[list[bool], list[str]]:
|
||||
"""Per-device integrated-GPU flags and descriptions via ggml's backend registry.
|
||||
|
||||
The Vulkan reg enumerates devices in the same order as
|
||||
``ggml_backend_vk_get_device_memory`` (each context uses ``ctx->device =
|
||||
i``), so reg index == device ordinal. Returns all-False on any failure so
|
||||
the reader never over-caps a discrete card.
|
||||
i``), so reg index == device ordinal. Returns all-False / empty-name on any
|
||||
failure so the reader never over-caps a discrete card and the memory
|
||||
readings still get through.
|
||||
"""
|
||||
flags = [False] * count
|
||||
names = [""] * count
|
||||
|
||||
# The name lookup is bound OUTSIDE the type-detection try: a ggml-base
|
||||
# without ggml_backend_dev_description (older/custom build) must degrade to
|
||||
# unnamed devices, not abort before the iGPU flags are read (which would
|
||||
# count an iGPU's shared RAM as VRAM).
|
||||
describe = None
|
||||
try:
|
||||
base.ggml_backend_dev_description.restype = ctypes.c_char_p
|
||||
base.ggml_backend_dev_description.argtypes = [ctypes.c_void_p]
|
||||
describe = base.ggml_backend_dev_description
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
lib.ggml_backend_vk_reg.restype = ctypes.c_void_p
|
||||
lib.ggml_backend_vk_reg.argtypes = []
|
||||
|
|
@ -45,17 +62,31 @@ def _igpu_flags(base, lib, count: int) -> list[bool]:
|
|||
|
||||
reg = lib.ggml_backend_vk_reg()
|
||||
if not reg:
|
||||
return flags
|
||||
return flags, names
|
||||
dev_count = base.ggml_backend_reg_dev_count(reg)
|
||||
for i in range(min(count, dev_count)):
|
||||
dev = base.ggml_backend_reg_dev_get(reg, i)
|
||||
if dev:
|
||||
flags[i] = base.ggml_backend_dev_type(dev) == _GGML_BACKEND_DEVICE_TYPE_IGPU
|
||||
if describe is not None:
|
||||
try:
|
||||
desc = describe(dev)
|
||||
if desc:
|
||||
# Tabs/newlines would corrupt the line protocol;
|
||||
# spaces are safe.
|
||||
names[i] = (
|
||||
desc.decode("utf-8", errors = "replace")
|
||||
.replace("\t", " ")
|
||||
.replace("\n", " ")
|
||||
.strip()
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception:
|
||||
# Best-effort: any failure degrades to "discrete" so the memory
|
||||
# readings still get through instead of crashing the probe.
|
||||
# Best-effort: any failure degrades to "discrete"/"unnamed" so the
|
||||
# memory readings still get through instead of crashing the probe.
|
||||
pass
|
||||
return flags
|
||||
return flags, names
|
||||
|
||||
|
||||
def main() -> int:
|
||||
|
|
@ -63,6 +94,14 @@ def main() -> int:
|
|||
return 0
|
||||
bindir = sys.argv[1]
|
||||
|
||||
# Device names can be non-ASCII (localized drivers); the platform-default
|
||||
# stdout encoding (e.g. cp1252) would raise on them and lose the whole
|
||||
# inventory. The reader decodes UTF-8 with the same error mode.
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding = "utf-8", errors = "replace")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Hold add_dll_directory's handle for the rest of main() (the documented
|
||||
# idiom) so bindir stays on the search path while the sibling ggml DLLs
|
||||
# resolve below.
|
||||
|
|
@ -96,12 +135,12 @@ def main() -> int:
|
|||
]
|
||||
|
||||
count = lib.ggml_backend_vk_get_device_count()
|
||||
igpu = _igpu_flags(base, lib, count)
|
||||
igpu, names = _igpu_flags_and_names(base, lib, count)
|
||||
rows = []
|
||||
for i in range(count):
|
||||
free, total = ctypes.c_size_t(0), ctypes.c_size_t(0)
|
||||
lib.ggml_backend_vk_get_device_memory(i, ctypes.byref(free), ctypes.byref(total))
|
||||
rows.append("%d\t%d\t%d\t%d" % (i, free.value, int(igpu[i]), total.value))
|
||||
rows.append("%d\t%d\t%d\t%d\t%s" % (i, free.value, int(igpu[i]), total.value, names[i]))
|
||||
sys.stdout.write("\n".join(rows))
|
||||
return 0
|
||||
|
||||
|
|
|
|||
|
|
@ -3501,18 +3501,17 @@ class LlamaCppBackend:
|
|||
return []
|
||||
|
||||
@staticmethod
|
||||
def _get_gpu_free_memory_vulkan(binary: Optional[str] = None) -> list[tuple[int, int, int]]:
|
||||
"""Query free (and total) VRAM per device via the bundled ggml Vulkan backend.
|
||||
def _run_vulkan_probe(binary: Optional[str] = None) -> list[dict]:
|
||||
"""Run ``_vulkan_probe.py`` and parse its per-device lines.
|
||||
|
||||
Loads ``libggml-vulkan`` in a short-lived subprocess (no Vulkan instance
|
||||
in this process) and returns (device_index, free_mib, total_mib) sorted
|
||||
by index. The index is ggml's compact Vulkan ordinal -- the one the
|
||||
registry names ``Vulkan<index>`` and load_model pins with ``--device``,
|
||||
NOT the raw ``GGML_VK_VISIBLE_DEVICES`` space. A user-set
|
||||
``GGML_VK_VISIBLE_DEVICES`` is honored by ggml (passed through), so the
|
||||
list already reflects it. iGPUs leave a host-RAM margin (see
|
||||
``_apply_igpu_host_reserve_mib``) and report total 0; discrete cards pass
|
||||
their real total through. [] when no Vulkan build or device is reachable.
|
||||
Returns raw (uncapped) rows sorted by index:
|
||||
``{"index", "free_mib", "total_mib", "is_igpu", "name"}``. The index is
|
||||
ggml's compact Vulkan ordinal -- the one the registry names
|
||||
``Vulkan<index>`` and load_model pins with ``--device``, NOT the raw
|
||||
``GGML_VK_VISIBLE_DEVICES`` space. A user-set ``GGML_VK_VISIBLE_DEVICES``
|
||||
is honored by ggml (passed through), so the list already reflects it.
|
||||
``name`` is ggml's device description; "" from an older 4-column probe.
|
||||
[] when no Vulkan build or device is reachable.
|
||||
"""
|
||||
binary = binary or LlamaCppBackend._find_llama_server_binary()
|
||||
if not binary:
|
||||
|
|
@ -3537,10 +3536,13 @@ class LlamaCppBackend:
|
|||
)
|
||||
probe_script = Path(__file__).with_name("_vulkan_probe.py")
|
||||
try:
|
||||
# UTF-8 to match the probe's stdout reconfigure: device names can be
|
||||
# non-ASCII, and the platform-default decode (cp1252) could throw.
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(probe_script), str(binary_dir)],
|
||||
capture_output = True,
|
||||
text = True,
|
||||
encoding = "utf-8",
|
||||
errors = "replace",
|
||||
timeout = 15,
|
||||
env = env,
|
||||
**_windows_hidden_subprocess_kwargs(),
|
||||
|
|
@ -3554,21 +3556,56 @@ class LlamaCppBackend:
|
|||
logger.debug(f"vulkan GPU probe failed: {e}")
|
||||
return []
|
||||
|
||||
gpus: list[tuple[int, int, int]] = []
|
||||
rows: list[dict] = []
|
||||
for line in result.stdout.strip().splitlines():
|
||||
parts = line.split("\t")
|
||||
if len(parts) != 4:
|
||||
# 4 columns from an older probe (no name); 5 with the name column.
|
||||
if len(parts) not in (4, 5):
|
||||
continue
|
||||
try:
|
||||
idx = int(parts[0])
|
||||
free_mib = int(parts[1]) // (1024 * 1024)
|
||||
is_igpu = parts[2] == "1"
|
||||
# iGPU "total" is shared RAM, not a VRAM budget -> keep 0 so the
|
||||
# fit stays on free*frac (the host reserve below is its
|
||||
# headroom); a discrete card passes its real total through.
|
||||
total_mib = 0 if is_igpu else int(parts[3]) // (1024 * 1024)
|
||||
rows.append(
|
||||
{
|
||||
"index": int(parts[0]),
|
||||
"free_mib": int(parts[1]) // (1024 * 1024),
|
||||
"is_igpu": parts[2] == "1",
|
||||
"total_mib": int(parts[3]) // (1024 * 1024),
|
||||
"name": parts[4].strip() if len(parts) == 5 else "",
|
||||
}
|
||||
)
|
||||
except ValueError:
|
||||
continue
|
||||
rows.sort(key = lambda r: r["index"])
|
||||
return rows
|
||||
|
||||
@staticmethod
|
||||
def vulkan_device_inventory(binary: Optional[str] = None) -> list[dict]:
|
||||
"""UI-facing Vulkan device list: the devices llama-server will actually
|
||||
use, with real totals (an iGPU keeps its shared-RAM total here -- the
|
||||
caller labels it, unlike the fit which zeroes it). Same rows as
|
||||
``_run_vulkan_probe``; names fall back to ``Vulkan<i>``.
|
||||
"""
|
||||
rows = LlamaCppBackend._run_vulkan_probe(binary)
|
||||
for row in rows:
|
||||
if not row["name"]:
|
||||
row["name"] = f"Vulkan{row['index']}"
|
||||
return rows
|
||||
|
||||
@staticmethod
|
||||
def _get_gpu_free_memory_vulkan(binary: Optional[str] = None) -> list[tuple[int, int, int]]:
|
||||
"""Query free (and total) VRAM per device via the bundled ggml Vulkan backend.
|
||||
|
||||
Fit-oriented view of ``_run_vulkan_probe``: returns (device_index,
|
||||
free_mib, total_mib) sorted by index. iGPUs leave a host-RAM margin (see
|
||||
``_apply_igpu_host_reserve_mib``) and report total 0; discrete cards pass
|
||||
their real total through. [] when no Vulkan build or device is reachable.
|
||||
"""
|
||||
gpus: list[tuple[int, int, int]] = []
|
||||
for row in LlamaCppBackend._run_vulkan_probe(binary):
|
||||
idx, free_mib, is_igpu = row["index"], row["free_mib"], row["is_igpu"]
|
||||
# iGPU "total" is shared RAM, not a VRAM budget -> keep 0 so the
|
||||
# fit stays on free*frac (the host reserve below is its
|
||||
# headroom); a discrete card passes its real total through.
|
||||
total_mib = 0 if is_igpu else row["total_mib"]
|
||||
capped = _apply_igpu_host_reserve_mib(free_mib, is_igpu)
|
||||
if capped < free_mib:
|
||||
logger.info(
|
||||
|
|
@ -3577,7 +3614,6 @@ class LlamaCppBackend:
|
|||
f"({free_mib}->{capped}MiB usable)"
|
||||
)
|
||||
gpus.append((idx, capped, total_mib))
|
||||
gpus.sort(key = lambda g: g[0])
|
||||
if gpus:
|
||||
logger.info(
|
||||
"Vulkan GPU memory detected: "
|
||||
|
|
@ -6635,12 +6671,23 @@ class LlamaCppBackend:
|
|||
# Block-diffusion GGUFs (DiffusionGemma) cannot run on llama-server;
|
||||
# serve them with the diffusion runner (same OpenAI-compat interface).
|
||||
if self._is_diffusion:
|
||||
# Final defense: route and pre-teardown preflights reject before Phase 1.
|
||||
if is_vulkan_backend and gpu_ids:
|
||||
raise ValueError(_VULKAN_DIFFUSION_GPU_IDS_ERROR)
|
||||
# Not a tensor/layer GGUF: clear any preserved-fallback flag from a
|
||||
# prior load (this path skips the command builder that clears it).
|
||||
self._layer_preserves_tensor_intent = False
|
||||
# On a Vulkan build gpu_ids are ggml Vulkan ordinals, but the diffusion
|
||||
# runner selects its device by CUDA physical index (_diffusion_gpu_arg
|
||||
# forwards gpu_ids[0] as a CUDA/DG_GPU token) with no mapping to them.
|
||||
# The route rejects a CONFIRMED-diffusion pick up front; an uncached GGUF
|
||||
# only classified as diffusion post-download still reaches here with a
|
||||
# pin, so drop it and serve on the default device (like an unpinned load).
|
||||
if gpu_ids and is_vulkan_backend:
|
||||
logger.warning(
|
||||
"Ignoring gpu_ids %s for diffusion GGUF on a Vulkan build: "
|
||||
"the diffusion runner cannot map ggml Vulkan ordinals; "
|
||||
"serving on the default device.",
|
||||
gpu_ids,
|
||||
)
|
||||
gpu_ids = None
|
||||
with self._lock:
|
||||
if self._cancel_event.is_set():
|
||||
logger.info("Load cancelled before diffusion server start")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue