6d27160dcc* Studio: fall back to text-only when a vision projector hard-crashes llama-server The text-only mmproj fallback (#6075) only fired when llama-server printed a recognizable projector-format error ("Unknown projector type", exit -6). An installed llama.cpp that predates a model's projector can instead SIGSEGV (exit -11) with no parseable output, e.g. unsloth/Qwen3.5-4B-MTP-GGUF + mmproj-F16 on an older gfx1151 prebuilt: llama-server crashes on load, the --fit off retry crashes the same way, and load_model gives up with a hard 500 instead of dropping vision. Generalize the decision: a vision (--mmproj) launch killed by a signal (POSIX returncode < 0, e.g. -11 SIGSEGV / -6 SIGABRT; Windows 0xC0000000+ access violation) is treated like a projector incompatibility, so the load retries once text-only. The retry is skipped if a cancel/unload is pending, mirroring the MTP guard. Clean non-zero exits (bad GGUF, port bind) and hung processes keep their own handling; non-vision launches are unaffected. Reproduced and verified on gfx1151 (Radeon 8060S, ROCm 7.2.1): a current prebuilt (llama.cpp b9596) loads the exact model + args fine, confirming the crash is a stale prebuilt. With a wrapper that SIGSEGVs on --mmproj, Studio now recovers: the load returns 200 (is_vision=false) and serves at ~31 tok/s text-only instead of failing. New _is_signal_crash helper plus tests pin the decision. Also normalize a few em-dashes to ASCII punctuation in existing comments. * Studio: refine mmproj hard-crash fallback (signal scope + last argv) - Limit _is_signal_crash to genuine program faults (SIGSEGV, SIGABRT, SIGILL, SIGFPE, SIGBUS) and Windows 0xC0000000+ statuses. SIGKILL, SIGTERM and SIGINT no longer count, so an OOM-killer, unload or supervisor kill is not masked as a projector incompatibility. - Strip --mmproj from the last attempted argv so the text-only retry keeps --fit off / --spec-default instead of resurrecting the original spec flags (matters for MTP vision models on an older llama.cpp). - Drop stray temp files committed by mistake and gitignore the "~" dir so they cannot be re-added. * Studio: tighten comments in mmproj hard-crash fallback * Studio: retry --flash-attn off before dropping vision on a startup crash When llama-server hard-crashes at startup, the recovery chain now tries the least-destructive mitigation first. Flash-attention kernels SIGSEGV at load on some ROCm/GPU builds (often inside the vision tower's attention); disabling flash attention keeps BOTH vision and MTP, so a hard program fault with --flash-attn on now retries once with --flash-attn off before the MTP-drop or the text-only (mmproj-strip) fallbacks. _is_signal_crash already gates this to genuine faults (SIGSEGV/SIGABRT/SIGILL/SIGFPE/SIGBUS), so an OOM-kill or unload (SIGKILL/SIGTERM/SIGINT) does not trigger a retry. Field context: a gfx1151 user crashes loading a vision GGUF even on the latest prebuilt, so an update cannot help, and the same model and args load fine on another gfx1151 box, pointing at a runtime/flash-attn fault. New _with_flash_attn_off helper plus tests. Verified on hardware with a wrapper that SIGSEGVs on --flash-attn on: Studio recovers with is_vision=true (vision and MTP intact) instead of failing or losing vision. * Studio: name the OOM kill on a too-large model load When the OS kills llama-server with no diagnostic output (SIGKILL/SIGTERM, almost always the OOM killer, e.g. a BF16 model too large for the WSL VM's RAM cap), the recovery ladder correctly does not retry an external kill, so this is the message the user sees. It fell through to the generic "is the GGUF valid / out of memory" text. Make it actionable: name the signal and point at a smaller or more quantized GGUF, a lower context length, or raising the WSL memory limit. Output-based diagnoses still win and a hard fault keeps the generic fallback. * Studio: refuse a model too large for system RAM on a unified-memory APU On gfx1150/gfx1151 APUs the weights load into shared system RAM (GGML unified memory). _get_gpu_free_memory reports the full ROCm/APU budget as free (often ~100 GB), but under WSL the VM's RAM cap is the real ceiling. Studio trusted the budget, spawned a load larger than RAM, and the OS killed it mid-flight, taking the Studio process with it (a silent "Terminated" with no error, the model resident in RAM not VRAM). Add a pre-flight guard on the APU path: if the weights exceed available system RAM (psutil, then /proc/meminfo), refuse before spawning with a clear message (smaller/more-quantized GGUF, lower context, or raise the WSL memory limit). Weights only so KV/context auto-reduction is not double-counted; unknown RAM never refuses; non-APU and discrete-GPU paths are untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refine recovery ladder (keep diagnosed errors, flip all flash-attn) Two review points on the hard-crash recovery ladder: 1. The signal-only text-only fallback stripped --mmproj on any hard fault, even when llama-server had already printed a non-projector cause (an OOM such as "cudaMalloc failed: out of memory", an unsupported architecture, or a tensor-parallel limit). That masked the real error and told the user to update llama.cpp for vision. New _output_has_nonprojector_diagnostic gates the signal path: it fires only when no such marker is present, so a bare SIGSEGV with no output still retries text-only, but a diagnosed OOM surfaces the real error instead of silently dropping vision. 2. _with_flash_attn_off only flipped the first --flash-attn. llama.cpp is last-wins, so a leftover enable from extra_args (--flash-attn on, -fa on, or the = form) could keep flash attention on and re-crash the retry. It now flips every occurrence and returns None only when nothing is flippable. test_llama_cpp_mmproj_fallback.py and the classification/APU suites: 103 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: pass the text-only retry's exit code to the failure classifier When the text-only fallback retry itself fails, read its exit code before _kill_process() clears it and forward it to _classify_llama_start_failure, so an OS-killed retry surfaces the actionable out-of-memory message instead of the generic one (matching the primary failure path). * Studio: scope APU RAM guard to selected GPUs, count MTP drafter, neutral SIGTERM Three refinements to the startup recovery work in this PR: - The unified-memory APU RAM guard fired whenever any visible GPU was a gfx1150/gfx1151 APU, so on a mixed APU+dGPU host it could refuse a valid load placed on the discrete GPU. Scope _amd_apu_wants_unified_memory to the selected gpu_indices (physical ids, mapped via CUDA_VISIBLE_DEVICES like _is_datacenter_gpu); None still means every visible GPU. Applied to both the RAM guard and the GGML_CUDA_ENABLE_UNIFIED_MEMORY env set. - The RAM guard counted only the main GGUF plus mmproj, so a separate MTP drafter (also resident in unified system RAM, even when offloaded to CPU) could push the load past the RAM cap and still get OS-killed mid-load. Add the drafter weights to the APU RAM total. - The startup classifier reported SIGTERM (-15) as 'most likely out of memory', but SIGTERM is also how an unload/cancel or a supervisor stops the server. Keep the OOM wording for SIGKILL (-9, the OOM killer) and report -15 neutrally. Tests updated/added accordingly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address review on the APU guard and decode-probe ladder - Map APU physical ids via the active ROCm mask (HIP, then ROCR, then CUDA), mirroring _get_gpu_memory, so a HIP_VISIBLE_DEVICES-selected APU is matched. - Only add the MTP drafter to the APU RAM total when MTP will actually engage, so a stale LLAMA_ARG_SPEC_DRAFT_MODEL cannot refuse a non-MTP load. - After an MTP first-decode hard fault, retry --flash-attn off (keeps MTP) before dropping speculative decoding, matching the startup rung. - Fold the --flash-attn= / -fa= rewrite into one branch. Tests: tensor-parallel decode-probe assertion updated for the FA-off rung. * Studio: tighten two comments in the APU guard and RAM preflight * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refine flash-attn retry and APU RAM guard per review - _with_flash_attn_off now decides on the effective last-wins value: it returns None when FA is already off (no wasted retry), and neutralizes a bare --flash-attn / -fa (which llama.cpp reads as on) so the retry cannot re-enable it. Length is preserved so downstream index slices stay valid. - _amd_apu_wants_unified_memory uses 'gpu_indices is not None' so an empty selection is respected (not treated as all-visible). - The APU RAM refusal now checks the base model only (main + mmproj); an optional MTP drafter is dropped by the existing MTP-drop fallback rather than causing a hard pre-spawn refusal of an otherwise loadable model. Tests: bare-flag / effective-off / empty-selection / HIP-mask cases added. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
117 lines
4.7 KiB
Python
117 lines
4.7 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""GGML_CUDA_ENABLE_UNIFIED_MEMORY must be set only for AMD unified-memory APUs
|
|
(gfx1150/gfx1151), never for discrete AMD, NVIDIA, CPU or macOS."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import types
|
|
|
|
import pytest
|
|
|
|
from core.inference.llama_cpp import LlamaCppBackend
|
|
|
|
|
|
def _fake_torch(
|
|
hip,
|
|
archs,
|
|
*,
|
|
cuda_ok = True,
|
|
):
|
|
t = types.ModuleType("torch")
|
|
t.version = types.SimpleNamespace(hip = hip)
|
|
t.cuda = types.SimpleNamespace(
|
|
is_available = lambda: cuda_ok,
|
|
device_count = lambda: len(archs),
|
|
get_device_properties = lambda i: types.SimpleNamespace(gcnArchName = archs[i]),
|
|
)
|
|
return t
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"hip,archs,expected",
|
|
[
|
|
("6.2.0", ["gfx1151:xnack-"], True), # Strix Halo APU (suffix stripped)
|
|
("6.2.0", ["gfx1150"], True), # Strix Point APU
|
|
("6.2.0", ["gfx1100"], False), # discrete RDNA3
|
|
("6.2.0", ["gfx1201"], False), # discrete RDNA4
|
|
("6.2.0", ["gfx942"], False), # MI300X (data center)
|
|
(None, ["sm_90"], False), # NVIDIA (no torch.version.hip)
|
|
("6.2.0", ["gfx1100", "gfx1151"], True), # mixed dGPU + APU
|
|
],
|
|
)
|
|
def test_apu_unified_memory_gating(monkeypatch, hip, archs, expected):
|
|
monkeypatch.setitem(sys.modules, "torch", _fake_torch(hip, archs))
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory() is expected
|
|
|
|
|
|
def test_apu_guard_scopes_to_selected_gpu(monkeypatch):
|
|
# Mixed host: physical id 0 = discrete gfx1100, 1 = gfx1151 APU.
|
|
for _m in ("HIP_VISIBLE_DEVICES", "ROCR_VISIBLE_DEVICES", "CUDA_VISIBLE_DEVICES"):
|
|
monkeypatch.delenv(_m, raising = False)
|
|
monkeypatch.setitem(sys.modules, "torch", _fake_torch("6.2.0", ["gfx1100", "gfx1151"]))
|
|
# Selecting only the dGPU, or an empty selection, must not be unified-memory.
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory([0]) is False
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory([]) is False
|
|
# Selecting the APU, or no selection, does.
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory([1]) is True
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory() is True
|
|
|
|
|
|
def test_apu_guard_honors_hip_visible_devices_mask(monkeypatch):
|
|
# ROCm resolves ids via HIP first: the mask exposes only the APU as ordinal 0
|
|
# but physical id 1, so the selection [1] must still match.
|
|
monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising = False)
|
|
monkeypatch.delenv("ROCR_VISIBLE_DEVICES", raising = False)
|
|
monkeypatch.setenv("HIP_VISIBLE_DEVICES", "1")
|
|
monkeypatch.setitem(sys.modules, "torch", _fake_torch("6.2.0", ["gfx1151"]))
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory([1]) is True
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory([0]) is False
|
|
|
|
|
|
def test_cpu_no_cuda_returns_false(monkeypatch):
|
|
monkeypatch.setitem(sys.modules, "torch", _fake_torch("6.2.0", [], cuda_ok = False))
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory() is False
|
|
|
|
|
|
def test_missing_torch_returns_false(monkeypatch):
|
|
monkeypatch.setitem(sys.modules, "torch", None)
|
|
assert LlamaCppBackend._amd_apu_wants_unified_memory() is False
|
|
|
|
|
|
_GB = 1024**3
|
|
_MIB_PER_GB = 1024
|
|
# Module-level (not a class attr) so it stays a plain function, not a bound method.
|
|
_shortfall = LlamaCppBackend._apu_ram_shortfall_message
|
|
|
|
|
|
class TestApuRamShortfall:
|
|
"""On a unified-memory APU the weights load into system RAM, so a model
|
|
larger than available RAM (the field case: a 64.6 GB GGUF on a WSL VM capped
|
|
well below the ROCm-reported APU budget) must be refused before spawning,
|
|
not left to OOM-kill the Studio process."""
|
|
|
|
def test_field_case_wsl_cap_refuses(self):
|
|
# 64.6 GB weights, ~46 GB available (WSL VM): refuse with guidance.
|
|
msg = _shortfall(int(64.6 * _GB), 46 * _MIB_PER_GB)
|
|
assert msg is not None
|
|
assert "65 GB" in msg and "46 GB" in msg
|
|
assert ".wslconfig" in msg
|
|
|
|
def test_bare_metal_fits_allows(self):
|
|
# Same model, ~92 GB available (no WSL cap): allow.
|
|
assert _shortfall(int(64.6 * _GB), 92 * _MIB_PER_GB) is None
|
|
|
|
def test_unknown_available_never_refuses(self):
|
|
assert _shortfall(int(64.6 * _GB), None) is None
|
|
|
|
def test_boundary_at_headroom(self):
|
|
# 20 GB weights, headroom 2 GB. avail 23 GB -> fits; 21 GB -> refuse.
|
|
assert _shortfall(20 * _GB, 23 * _MIB_PER_GB) is None
|
|
assert _shortfall(20 * _GB, 21 * _MIB_PER_GB) is not None
|
|
|
|
def test_available_system_memory_is_int_or_none(self):
|
|
v = LlamaCppBackend._available_system_memory_mib()
|
|
assert v is None or (isinstance(v, int) and v > 0)
|