* Vulkan GPUs: real device names and selectable ordinals
Rebases the durable half of #7356 onto the inference_gpu transport #7476
landed on main. Those two PRs solve an overlapping problem and disagree on
the data model, so merging #7356 as-is would ship two parallel Vulkan
device concepts with different index semantics. This keeps main's transport
and adds what #7356 had that #7476 does not.
- _vulkan_probe.py emits a 5th column, ggml's device description, sanitized
for the tab protocol and UTF-8 safe. Reader tolerates 4- or 5-column
output so an older probe still parses.
- llama_cpp gains _run_vulkan_probe (shared parse) and
vulkan_device_inventory (names + is_igpu + real totals).
- get_vulkan_inference_gpu_info reports the real name and an explicit
is_igpu instead of "Vulkan<i>" and a total == 0 guess.
- index_kind becomes "vulkan", not "relative", and gpu_ids picks are
supported on Vulkan builds once the probe enumerated ordinals. The XPU ban
no longer applies to them: a Vulkan pick is a ggml ordinal, not a torch-xpu
index, so it works on an Intel host too.
- Frontend picker reads the Vulkan inventory as the pickable set.
Memory deliberately still comes from _get_gpu_memory, not the inventory.
That path applies _apply_igpu_host_reserve_mib and zeroes a shared total;
budgeting an APU off its raw shared total would hand out the whole machine's
RAM with no OS headroom. Identity is joined onto it by ordinal, so a probe
failure degrades to Vulkan<i> names with the memory readings intact.
Dropped from #7356 as superseded: validate_vulkan_gpu_ids (main's
resolve_requested_gpu_ids already rejects duplicates and
_resolve_gguf_gpu_ids_for_request already probes for existence), the
gguf_devices transport, and the iGPU budget fallback in 71619891e, which
main's aggregateGpuMemoryTotalGb handles better by counting a shared pool
once.
Also keeps #7356's removal of the late diffusion raise, so the graceful
gpu_ids drop stays reachable for a GGUF only classified as diffusion after
download. #7415's real guard, _reject_vulkan_diffusion_gpu_ids_before_
teardown, is untouched.
Verified on Windows + Strix Halo: backend Vulkan/GPU-selection suites at the
same 4 pre-existing failures as main, tests/studio 1671 passed with no new
failures, frontend typecheck clean. Hardware confirmation of the underlying
behavior is on #7356 from @Bebiv24 (RX 9070 XT + RX 480).
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
da447d47ba
commit
74295d93d8
8 changed files with 297 additions and 70 deletions
|
|
@ -3501,18 +3501,17 @@ class LlamaCppBackend:
|
|||
return []
|
||||
|
||||
@staticmethod
|
||||
def _get_gpu_free_memory_vulkan(binary: Optional[str] = None) -> list[tuple[int, int, int]]:
|
||||
"""Query free (and total) VRAM per device via the bundled ggml Vulkan backend.
|
||||
def _run_vulkan_probe(binary: Optional[str] = None) -> list[dict]:
|
||||
"""Run ``_vulkan_probe.py`` and parse its per-device lines.
|
||||
|
||||
Loads ``libggml-vulkan`` in a short-lived subprocess (no Vulkan instance
|
||||
in this process) and returns (device_index, free_mib, total_mib) sorted
|
||||
by index. The index is ggml's compact Vulkan ordinal -- the one the
|
||||
registry names ``Vulkan<index>`` and load_model pins with ``--device``,
|
||||
NOT the raw ``GGML_VK_VISIBLE_DEVICES`` space. A user-set
|
||||
``GGML_VK_VISIBLE_DEVICES`` is honored by ggml (passed through), so the
|
||||
list already reflects it. iGPUs leave a host-RAM margin (see
|
||||
``_apply_igpu_host_reserve_mib``) and report total 0; discrete cards pass
|
||||
their real total through. [] when no Vulkan build or device is reachable.
|
||||
Returns raw (uncapped) rows sorted by index:
|
||||
``{"index", "free_mib", "total_mib", "is_igpu", "name"}``. The index is
|
||||
ggml's compact Vulkan ordinal -- the one the registry names
|
||||
``Vulkan<index>`` and load_model pins with ``--device``, NOT the raw
|
||||
``GGML_VK_VISIBLE_DEVICES`` space. A user-set ``GGML_VK_VISIBLE_DEVICES``
|
||||
is honored by ggml (passed through), so the list already reflects it.
|
||||
``name`` is ggml's device description; "" from an older 4-column probe.
|
||||
[] when no Vulkan build or device is reachable.
|
||||
"""
|
||||
binary = binary or LlamaCppBackend._find_llama_server_binary()
|
||||
if not binary:
|
||||
|
|
@ -3537,10 +3536,13 @@ class LlamaCppBackend:
|
|||
)
|
||||
probe_script = Path(__file__).with_name("_vulkan_probe.py")
|
||||
try:
|
||||
# UTF-8 to match the probe's stdout reconfigure: device names can be
|
||||
# non-ASCII, and the platform-default decode (cp1252) could throw.
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(probe_script), str(binary_dir)],
|
||||
capture_output = True,
|
||||
text = True,
|
||||
encoding = "utf-8",
|
||||
errors = "replace",
|
||||
timeout = 15,
|
||||
env = env,
|
||||
**_windows_hidden_subprocess_kwargs(),
|
||||
|
|
@ -3554,21 +3556,56 @@ class LlamaCppBackend:
|
|||
logger.debug(f"vulkan GPU probe failed: {e}")
|
||||
return []
|
||||
|
||||
gpus: list[tuple[int, int, int]] = []
|
||||
rows: list[dict] = []
|
||||
for line in result.stdout.strip().splitlines():
|
||||
parts = line.split("\t")
|
||||
if len(parts) != 4:
|
||||
# 4 columns from an older probe (no name); 5 with the name column.
|
||||
if len(parts) not in (4, 5):
|
||||
continue
|
||||
try:
|
||||
idx = int(parts[0])
|
||||
free_mib = int(parts[1]) // (1024 * 1024)
|
||||
is_igpu = parts[2] == "1"
|
||||
# iGPU "total" is shared RAM, not a VRAM budget -> keep 0 so the
|
||||
# fit stays on free*frac (the host reserve below is its
|
||||
# headroom); a discrete card passes its real total through.
|
||||
total_mib = 0 if is_igpu else int(parts[3]) // (1024 * 1024)
|
||||
rows.append(
|
||||
{
|
||||
"index": int(parts[0]),
|
||||
"free_mib": int(parts[1]) // (1024 * 1024),
|
||||
"is_igpu": parts[2] == "1",
|
||||
"total_mib": int(parts[3]) // (1024 * 1024),
|
||||
"name": parts[4].strip() if len(parts) == 5 else "",
|
||||
}
|
||||
)
|
||||
except ValueError:
|
||||
continue
|
||||
rows.sort(key = lambda r: r["index"])
|
||||
return rows
|
||||
|
||||
@staticmethod
|
||||
def vulkan_device_inventory(binary: Optional[str] = None) -> list[dict]:
|
||||
"""UI-facing Vulkan device list: the devices llama-server will actually
|
||||
use, with real totals (an iGPU keeps its shared-RAM total here -- the
|
||||
caller labels it, unlike the fit which zeroes it). Same rows as
|
||||
``_run_vulkan_probe``; names fall back to ``Vulkan<i>``.
|
||||
"""
|
||||
rows = LlamaCppBackend._run_vulkan_probe(binary)
|
||||
for row in rows:
|
||||
if not row["name"]:
|
||||
row["name"] = f"Vulkan{row['index']}"
|
||||
return rows
|
||||
|
||||
@staticmethod
|
||||
def _get_gpu_free_memory_vulkan(binary: Optional[str] = None) -> list[tuple[int, int, int]]:
|
||||
"""Query free (and total) VRAM per device via the bundled ggml Vulkan backend.
|
||||
|
||||
Fit-oriented view of ``_run_vulkan_probe``: returns (device_index,
|
||||
free_mib, total_mib) sorted by index. iGPUs leave a host-RAM margin (see
|
||||
``_apply_igpu_host_reserve_mib``) and report total 0; discrete cards pass
|
||||
their real total through. [] when no Vulkan build or device is reachable.
|
||||
"""
|
||||
gpus: list[tuple[int, int, int]] = []
|
||||
for row in LlamaCppBackend._run_vulkan_probe(binary):
|
||||
idx, free_mib, is_igpu = row["index"], row["free_mib"], row["is_igpu"]
|
||||
# iGPU "total" is shared RAM, not a VRAM budget -> keep 0 so the
|
||||
# fit stays on free*frac (the host reserve below is its
|
||||
# headroom); a discrete card passes its real total through.
|
||||
total_mib = 0 if is_igpu else row["total_mib"]
|
||||
capped = _apply_igpu_host_reserve_mib(free_mib, is_igpu)
|
||||
if capped < free_mib:
|
||||
logger.info(
|
||||
|
|
@ -3577,7 +3614,6 @@ class LlamaCppBackend:
|
|||
f"({free_mib}->{capped}MiB usable)"
|
||||
)
|
||||
gpus.append((idx, capped, total_mib))
|
||||
gpus.sort(key = lambda g: g[0])
|
||||
if gpus:
|
||||
logger.info(
|
||||
"Vulkan GPU memory detected: "
|
||||
|
|
@ -6635,12 +6671,23 @@ class LlamaCppBackend:
|
|||
# Block-diffusion GGUFs (DiffusionGemma) cannot run on llama-server;
|
||||
# serve them with the diffusion runner (same OpenAI-compat interface).
|
||||
if self._is_diffusion:
|
||||
# Final defense: route and pre-teardown preflights reject before Phase 1.
|
||||
if is_vulkan_backend and gpu_ids:
|
||||
raise ValueError(_VULKAN_DIFFUSION_GPU_IDS_ERROR)
|
||||
# Not a tensor/layer GGUF: clear any preserved-fallback flag from a
|
||||
# prior load (this path skips the command builder that clears it).
|
||||
self._layer_preserves_tensor_intent = False
|
||||
# On a Vulkan build gpu_ids are ggml Vulkan ordinals, but the diffusion
|
||||
# runner selects its device by CUDA physical index (_diffusion_gpu_arg
|
||||
# forwards gpu_ids[0] as a CUDA/DG_GPU token) with no mapping to them.
|
||||
# The route rejects a CONFIRMED-diffusion pick up front; an uncached GGUF
|
||||
# only classified as diffusion post-download still reaches here with a
|
||||
# pin, so drop it and serve on the default device (like an unpinned load).
|
||||
if gpu_ids and is_vulkan_backend:
|
||||
logger.warning(
|
||||
"Ignoring gpu_ids %s for diffusion GGUF on a Vulkan build: "
|
||||
"the diffusion runner cannot map ggml Vulkan ordinals; "
|
||||
"serving on the default device.",
|
||||
gpu_ids,
|
||||
)
|
||||
gpu_ids = None
|
||||
with self._lock:
|
||||
if self._cancel_event.is_set():
|
||||
logger.info("Load cancelled before diffusion server start")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue