f033213c0b* Studio: account for mmproj VRAM in GGUF fit budget (#5825) Vision GGUFs load the mmproj projector onto the GPU via --mmproj alongside the weights, but the context auto-sizing / GPU-selection budget sized off _get_gguf_size_bytes(model_path), which counts only the weight file(s). The projector was never added, so the budget was too optimistic: context got mis-estimated and tight vision loads spilled to system RAM / OOM'd. Resolve the launch projector once before GPU selection and fold its size into the fit budget. The same resolved path feeds both the budget and the --mmproj launch flag, so the two cannot disagree. The summary log now reports the projector size separately, keeping "GGUF size" accurate. Adds _mmproj_vram_bytes() + unit tests (no GPU / network / subprocess). * Studio: simplify mmproj summary-log concatenation (#5825) Address review: the summary log mixed explicit `+` with implicit f-string concatenation. Extract the optional projector fragment into `mmproj_note` so the logger.info uses uniform implicit concatenation. No behavioral change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: trim mmproj VRAM comments --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
40 lines
1.1 KiB
Python
40 lines
1.1 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Tests for mmproj VRAM accounting in GGUF fit budgeting (#5825)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
from core.inference.llama_cpp import LlamaCppBackend
|
|
|
|
|
|
def _write(path: Path, n_bytes: int) -> Path:
|
|
path.parent.mkdir(parents = True, exist_ok = True)
|
|
path.write_bytes(b"\x00" * n_bytes)
|
|
return path
|
|
|
|
|
|
def _backend() -> LlamaCppBackend:
|
|
return LlamaCppBackend.__new__(LlamaCppBackend)
|
|
|
|
|
|
def test_counts_resolved_projector_size(tmp_path: Path):
|
|
mmproj = _write(tmp_path / "Qwen3.5-9B-BF16-mmproj.gguf", 1024)
|
|
|
|
got = _backend()._mmproj_vram_bytes(str(mmproj))
|
|
|
|
assert got == 1024
|
|
|
|
|
|
def test_zero_when_no_projector_resolved(tmp_path: Path):
|
|
assert _backend()._mmproj_vram_bytes(None) == 0
|
|
|
|
|
|
def test_zero_when_projector_missing_on_disk(tmp_path: Path):
|
|
missing = tmp_path / "Qwen3.5-9B-BF16-mmproj.gguf" # never created
|
|
|
|
got = _backend()._mmproj_vram_bytes(str(missing))
|
|
|
|
assert got == 0
|