9311df2b29* [WIP] balanced device map for studio * gpus as a request parameter * API for multi GPU stuff * return multi gpu util in new API * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use balanced_low0 instead of balanced * Use balanced_low0 instead of balanced * Fix device_map typo, UUID parsing crash, set() filter bug, and broken tests - balanced_low0 -> balanced_low_0 (transformers/accelerate rejects the old string) - get_parent_visible_gpu_ids() now handles UUID/MIG CUDA_VISIBLE_DEVICES gracefully instead of crashing on int() parse - _get_backend_visible_gpu_info() set() or None bug: empty set is falsy so CUDA_VISIBLE_DEVICES=-1 would disable filtering and report all GPUs - test_gpu_selection.py: add missing get_visible_gpu_utilization import and add required job_id arg to start_training() calls * Smart GPU determinism using estimates * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * disallow gpu selection for gguf for now * cleanup * Slightly larger baseline * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat empty list as auto * Verbose logging/debug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup and revert unnecessary deletions * Cleanup excessive logs and guard against disk/cpu offload * auth for visibility API. cleanup redundant imports. Adjust QLoRA estimate * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * support for non cuda gpus * Fix multi-GPU auto-selection memory accounting The multi_gpu_factor was applied uniformly to all GPUs including the first one, which unfairly penalizes single-GPU capacity when transitioning to multi-GPU. This created a discontinuity where a model that barely fits 1 GPU would suddenly require 2 GPUs because the first GPU's free memory was discounted by 20%. Now the first GPU keeps its full free memory, and only additional GPUs have an overhead factor (0.85) applied to account for inter-GPU communication and sharding overhead. This gives more accurate auto-selection and avoids unnecessary multi-GPU for models that comfortably fit on one device. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add sandbox tests for multi-GPU selection logic 24 tests covering model size estimation, memory requirements, automatic GPU selection, device map generation, GPU ID validation, and multi-GPU overhead accounting. All tests use mocks so they run without GPUs on Linux, macOS, and Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix reviewer findings: 4bit inference estimate, fallback, GGUF gpu_ids, retry 1. 4-bit inference now uses reduced memory estimate (model_size/3 + buffer) instead of the FP16 1.3x multiplier. This prevents over-sharding quantized models across unnecessary GPUs. 2. When model size estimation fails, auto_select_gpu_ids now falls back to all visible GPUs instead of returning None (which could default to single-GPU loading for an unknown-size model). 3. GGUF inference route now treats gpu_ids=[] as auto-selection (same as None) instead of rejecting it as an unsupported explicit request. 4. Training retry path for "could not get source code" now preserves the gpu_ids parameter so the retry lands on the same GPUs. 5. Updated sandbox tests to cover the new 4-bit inference estimate branch. * Remove accidentally added unsloth-zoo submodule * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix UUID/MIG visibility and update test expectations 1. nvidia.py: When CUDA_VISIBLE_DEVICES uses UUID/MIG tokens, the visibility APIs now return "unresolved" with empty device lists instead of exposing all physical GPUs. This prevents the UI from showing GPUs that the backend process cannot actually use. 2. test_gpu_selection.py: Updated test expectations to match the new multi-GPU overhead accounting (first GPU at full capacity, 0.85x for additional GPUs) and 4-bit inference memory estimation formula. All 60 tests now pass. * Add CPU/disk offload guard to audio inference path The audio model loading branch returned before the common get_offloaded_device_map_entries() check, so audio models loaded with a multi-GPU device_map that spilled layers to CPU/disk would be accepted instead of rejected. Now audio loads also verify no modules are offloaded. * Improve VRAM requirement estimates * Replace balanced_low_0 with balanced * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refine calculations for slightly easier nums * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adjust estimates * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use nums instead of obj to avoid seralisation error * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden nvidia-smi parsing and fix fallback GPU list 1. nvidia.py: Wrap int() casts for GPU index and memory in try/except so MIG slices, N/A values, or unexpected nvidia-smi output skip the unparseable row instead of aborting the entire GPU list. 2. nvidia.py: Handle GPU names containing commas by using the last field as memory instead of a fixed positional index. 3. hardware.py: fallback_all now uses gpu_candidates (GPUs with verified VRAM data) instead of raw devices list, which could include GPUs with null VRAM that were excluded from the ranking. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cleanup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * consolidate raise_if_offload * Improve MoE support. Guard against nvidia-smi failures * Improve MoE support. Guard against nvidia-smi failures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix shared-expert LoRA undercount, torch VRAM fallback, and apply_gpu_ids edge case 1. vram_estimation.py: compute_lora_params now includes shared experts (n_shared_experts) alongside routed experts when computing MoE LoRA adapter parameters. Previously only n_experts were counted, causing the estimator to undercount adapter, optimizer, and gradient memory for DeepSeek/GLM-style models with shared experts. 2. hardware.py: _torch_get_per_device_info now uses mem_get_info (which reports system-wide VRAM usage) instead of memory_allocated (which only reports this process's PyTorch allocations). This prevents auto-selection from treating a GPU as mostly free when another process is consuming VRAM. Falls back to memory_allocated when mem_get_info is unavailable. 3. hardware.py: apply_gpu_ids([]) now returns early instead of setting CUDA_VISIBLE_DEVICES="" which would disable CUDA entirely. Empty list inherits the parent visibility, same as None. 4. hardware.py: Upgraded fallback_all GPU selection log from debug to warning so operators are notified when the model likely will not fit in available VRAM. * Guard nvidia-smi subprocess calls against OSError and TimeoutExpired get_visible_gpu_utilization and get_backend_visible_gpu_info now catch OSError (nvidia-smi not found) and TimeoutExpired internally instead of relying on callers to wrap every invocation. Returns the standard available=False sentinel on failure so the torch-based fallback in hardware.py can take over. * Guard get_primary_gpu_utilization and reset GPU caches between tests 1. nvidia.py: get_primary_gpu_utilization now catches OSError and TimeoutExpired internally, matching the pattern already used in get_visible_gpu_utilization and get_backend_visible_gpu_info. All three nvidia-smi callers are now self-contained. 2. test_gpu_selection.py: Added _GpuCacheResetMixin that resets the module-level _physical_gpu_count and _visible_gpu_count caches in tearDown. Applied to all test classes that exercise GPU selection, device map, or visibility functions. This prevents stale cache values from leaking between tests and causing flaky results on machines with real GPUs. * Fix nvidia-smi fallback regression and physical GPU count validation 1. hardware.py: get_gpu_utilization, get_visible_gpu_utilization, and get_backend_visible_gpu_info now check result.get("available") before returning the nvidia-smi result. When nvidia-smi is unavailable or returns no data (e.g., containers without nvidia-smi, UUID/MIG masks), the functions fall through to the torch-based fallback instead of returning an empty result. This fixes a regression where the internal exception handling in nvidia.py prevented the caller's except block from triggering the fallback. 2. hardware.py: resolve_requested_gpu_ids now separates negative-ID validation from physical upper-bound validation. The physical count check is only enforced when it is plausibly a true physical count (i.e., higher than the largest parent-visible ID), since torch.cuda.device_count() under CUDA_VISIBLE_DEVICES returns the visible count, not the physical total. The parent-visible-set check remains authoritative in all cases. This prevents valid physical IDs like [2, 3] from being rejected as "out of range" when nvidia-smi is unavailable and CUDA_VISIBLE_DEVICES="2,3" makes torch report only 2 devices. * Fix UUID/MIG torch fallback to enumerate devices by ordinal When CUDA_VISIBLE_DEVICES uses UUID or MIG identifiers, get_parent_visible_gpu_ids() returns [] because the tokens are non-numeric. The torch fallback in get_visible_gpu_utilization() and get_backend_visible_gpu_info() previously passed that empty list to _torch_get_per_device_info(), getting nothing back. Now both functions detect the empty-list case and fall back to enumerating torch-visible ordinals (0..device_count-1) with index_kind="relative". This means the UI and auto-selection still see real device data in Kubernetes, MIG, and Slurm-style UUID environments where nvidia-smi output cannot be mapped to physical indices. Updated test_uuid_parent_visibility to verify the new torch fallback path returns available=True with relative ordinals. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add type hint for gpu_ids parameter in InferenceOrchestrator.load_model --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>
544 lines
19 KiB
Python
544 lines
19 KiB
Python
#!/usr/bin/env python3
|
|
"""
|
|
Sandbox test for multi-GPU selection logic.
|
|
|
|
Tests the core GPU selection, memory estimation, and device_map logic
|
|
in an isolated environment. Can be run on Linux, macOS, and Windows
|
|
without requiring actual GPUs -- all hardware calls are mocked.
|
|
|
|
Usage:
|
|
python -m pytest studio/backend/tests/test_gpu_selection_sandbox.py -v
|
|
# or directly:
|
|
python studio/backend/tests/test_gpu_selection_sandbox.py
|
|
"""
|
|
|
|
import os
|
|
import sys
|
|
import unittest
|
|
from pathlib import Path
|
|
from unittest.mock import patch, MagicMock
|
|
|
|
# Ensure backend is on sys.path
|
|
_backend_root = Path(__file__).resolve().parent.parent
|
|
if str(_backend_root) not in sys.path:
|
|
sys.path.insert(0, str(_backend_root))
|
|
|
|
|
|
def _make_fake_config(
|
|
vocab_size = 32000,
|
|
hidden_size = 4096,
|
|
intermediate_size = 11008,
|
|
num_hidden_layers = 32,
|
|
num_attention_heads = 32,
|
|
num_key_value_heads = 8,
|
|
tie_word_embeddings = False,
|
|
):
|
|
"""Create a fake HF config-like object for estimation tests."""
|
|
from types import SimpleNamespace
|
|
|
|
return SimpleNamespace(
|
|
vocab_size = vocab_size,
|
|
hidden_size = hidden_size,
|
|
intermediate_size = intermediate_size,
|
|
num_hidden_layers = num_hidden_layers,
|
|
num_attention_heads = num_attention_heads,
|
|
num_key_value_heads = num_key_value_heads,
|
|
tie_word_embeddings = tie_word_embeddings,
|
|
)
|
|
|
|
|
|
class TestEstimateFP16ModelSizeFromConfig(unittest.TestCase):
|
|
"""Test the config-based model size estimation."""
|
|
|
|
def test_llama_8b_size_reasonable(self):
|
|
from utils.hardware.hardware import _estimate_fp16_model_size_bytes_from_config
|
|
|
|
config = _make_fake_config(
|
|
vocab_size = 128256,
|
|
hidden_size = 4096,
|
|
intermediate_size = 14336,
|
|
num_hidden_layers = 32,
|
|
num_attention_heads = 32,
|
|
num_key_value_heads = 8,
|
|
tie_word_embeddings = False,
|
|
)
|
|
size = _estimate_fp16_model_size_bytes_from_config(config)
|
|
self.assertIsNotNone(size)
|
|
size_gb = size / (1024**3)
|
|
# Llama 3.1 8B should be ~15GB in fp16
|
|
self.assertGreater(size_gb, 12)
|
|
self.assertLess(size_gb, 20)
|
|
|
|
def test_small_model(self):
|
|
from utils.hardware.hardware import _estimate_fp16_model_size_bytes_from_config
|
|
|
|
config = _make_fake_config(
|
|
vocab_size = 32000,
|
|
hidden_size = 2048,
|
|
intermediate_size = 5504,
|
|
num_hidden_layers = 22,
|
|
num_attention_heads = 32,
|
|
num_key_value_heads = 4,
|
|
)
|
|
size = _estimate_fp16_model_size_bytes_from_config(config)
|
|
self.assertIsNotNone(size)
|
|
size_gb = size / (1024**3)
|
|
# ~1B model should be ~2GB in fp16
|
|
self.assertGreater(size_gb, 1)
|
|
self.assertLess(size_gb, 5)
|
|
|
|
def test_returns_none_for_incomplete_config(self):
|
|
from utils.hardware.hardware import _estimate_fp16_model_size_bytes_from_config
|
|
from types import SimpleNamespace
|
|
|
|
config = SimpleNamespace(vocab_size = 32000) # Missing most fields
|
|
size = _estimate_fp16_model_size_bytes_from_config(config)
|
|
self.assertIsNone(size)
|
|
|
|
def test_moe_model(self):
|
|
from utils.hardware.hardware import _estimate_fp16_model_size_bytes_from_config
|
|
from types import SimpleNamespace
|
|
|
|
config = SimpleNamespace(
|
|
vocab_size = 152064,
|
|
hidden_size = 3584,
|
|
intermediate_size = 18944,
|
|
num_hidden_layers = 28,
|
|
num_attention_heads = 28,
|
|
num_key_value_heads = 4,
|
|
tie_word_embeddings = False,
|
|
num_local_experts = 64,
|
|
moe_intermediate_size = 2560,
|
|
)
|
|
size = _estimate_fp16_model_size_bytes_from_config(config)
|
|
self.assertIsNotNone(size)
|
|
size_gb = size / (1024**3)
|
|
# MoE model with 64 experts should be large
|
|
self.assertGreater(size_gb, 50)
|
|
|
|
|
|
class TestEstimateRequiredModelMemory(unittest.TestCase):
|
|
"""Test memory requirement estimation."""
|
|
|
|
def test_inference_fp16_uses_1_3x(self):
|
|
from utils.hardware.hardware import estimate_required_model_memory_gb
|
|
|
|
with patch(
|
|
"utils.hardware.hardware.estimate_fp16_model_size_bytes",
|
|
return_value = (10 * (1024**3), "config"), # 10GB model
|
|
):
|
|
required, meta = estimate_required_model_memory_gb(
|
|
"test/model",
|
|
training_type = None, # inference
|
|
load_in_4bit = False,
|
|
)
|
|
self.assertIsNotNone(required)
|
|
self.assertAlmostEqual(required, 13.0, places = 0)
|
|
self.assertEqual(meta["mode"], "inference")
|
|
|
|
def test_inference_4bit_uses_reduced_estimate(self):
|
|
from utils.hardware.hardware import estimate_required_model_memory_gb
|
|
|
|
with patch(
|
|
"utils.hardware.hardware.estimate_fp16_model_size_bytes",
|
|
return_value = (30 * (1024**3), "config"), # 30GB fp16 model
|
|
):
|
|
required, meta = estimate_required_model_memory_gb(
|
|
"test/model",
|
|
training_type = None, # inference
|
|
load_in_4bit = True,
|
|
)
|
|
self.assertIsNotNone(required)
|
|
# 4bit base = 30/3.2 = 9.375GB, required = 9.375 + max(9.375*0.3, 2) = 12.19GB
|
|
self.assertAlmostEqual(required, 12.2, places = 0)
|
|
|
|
def test_4bit_training_reduces_base(self):
|
|
from utils.hardware.hardware import estimate_required_model_memory_gb
|
|
|
|
with patch(
|
|
"utils.hardware.hardware.estimate_fp16_model_size_bytes",
|
|
return_value = (30 * (1024**3), "config"), # 30GB fp16 model
|
|
):
|
|
required, meta = estimate_required_model_memory_gb(
|
|
"test/model",
|
|
training_type = "LoRA/QLoRA",
|
|
load_in_4bit = True,
|
|
)
|
|
self.assertIsNotNone(required)
|
|
# fallback: base=30/3.2=9.375, lora=30*0.04=1.2, act=30*0.15=4.5, cuda=1.4
|
|
self.assertAlmostEqual(required, 16.5, places = 0)
|
|
|
|
def test_full_finetune_uses_3_5x(self):
|
|
from utils.hardware.hardware import estimate_required_model_memory_gb
|
|
|
|
with patch(
|
|
"utils.hardware.hardware.estimate_fp16_model_size_bytes",
|
|
return_value = (10 * (1024**3), "config"), # 10GB model
|
|
):
|
|
required, meta = estimate_required_model_memory_gb(
|
|
"test/model",
|
|
training_type = "Full Finetuning",
|
|
)
|
|
self.assertIsNotNone(required)
|
|
# fallback: 10 * 3.5 + 1.4 cuda overhead = 36.4
|
|
self.assertAlmostEqual(required, 36.4, places = 0)
|
|
|
|
def test_returns_none_when_unavailable(self):
|
|
from utils.hardware.hardware import estimate_required_model_memory_gb
|
|
|
|
with patch(
|
|
"utils.hardware.hardware.estimate_fp16_model_size_bytes",
|
|
return_value = (None, "unavailable"),
|
|
):
|
|
required, meta = estimate_required_model_memory_gb("test/model")
|
|
self.assertIsNone(required)
|
|
|
|
|
|
class TestAutoSelectGpuIds(unittest.TestCase):
|
|
"""Test automatic GPU selection based on model size and free memory."""
|
|
|
|
def _make_utilization(self, devices):
|
|
"""Create a fake utilization response."""
|
|
return {
|
|
"available": True,
|
|
"devices": [
|
|
{
|
|
"index": idx,
|
|
"vram_total_gb": total,
|
|
"vram_used_gb": total - free,
|
|
}
|
|
for idx, total, free in devices
|
|
],
|
|
}
|
|
|
|
def test_single_gpu_sufficient(self):
|
|
from utils.hardware.hardware import auto_select_gpu_ids
|
|
import utils.hardware.hardware as hw
|
|
|
|
with (
|
|
patch.object(hw, "get_device", return_value = hw.DeviceType.CUDA),
|
|
patch.object(
|
|
hw,
|
|
"estimate_required_model_memory_gb",
|
|
return_value = (
|
|
10.0,
|
|
{
|
|
"mode": "inference",
|
|
"required_gb": 10.0,
|
|
"model_size_source": "config",
|
|
"model_size_gb": 7.7,
|
|
},
|
|
),
|
|
),
|
|
patch.object(
|
|
hw,
|
|
"_get_parent_visible_gpu_spec",
|
|
return_value = {
|
|
"raw": "0,1,2,3",
|
|
"numeric_ids": [0, 1, 2, 3],
|
|
"supports_explicit_gpu_ids": True,
|
|
},
|
|
),
|
|
patch.object(hw, "get_parent_visible_gpu_ids", return_value = [0, 1, 2, 3]),
|
|
patch.object(
|
|
hw,
|
|
"get_visible_gpu_utilization",
|
|
return_value = self._make_utilization(
|
|
[
|
|
(0, 80.0, 75.0),
|
|
(1, 80.0, 78.0),
|
|
(2, 80.0, 70.0),
|
|
(3, 80.0, 72.0),
|
|
]
|
|
),
|
|
),
|
|
):
|
|
selected, meta = auto_select_gpu_ids("test/model")
|
|
# Should pick GPU 1 (most free memory: 78GB) -- enough for 10GB
|
|
self.assertEqual(len(selected), 1)
|
|
self.assertEqual(selected[0], 1)
|
|
|
|
def test_two_gpus_needed(self):
|
|
from utils.hardware.hardware import auto_select_gpu_ids
|
|
import utils.hardware.hardware as hw
|
|
|
|
with (
|
|
patch.object(hw, "get_device", return_value = hw.DeviceType.CUDA),
|
|
patch.object(
|
|
hw,
|
|
"estimate_required_model_memory_gb",
|
|
return_value = (
|
|
50.0,
|
|
{
|
|
"mode": "inference",
|
|
"required_gb": 50.0,
|
|
"model_size_source": "config",
|
|
"model_size_gb": 38.0,
|
|
},
|
|
),
|
|
),
|
|
patch.object(
|
|
hw,
|
|
"_get_parent_visible_gpu_spec",
|
|
return_value = {
|
|
"raw": "0,1",
|
|
"numeric_ids": [0, 1],
|
|
"supports_explicit_gpu_ids": True,
|
|
},
|
|
),
|
|
patch.object(hw, "get_parent_visible_gpu_ids", return_value = [0, 1]),
|
|
patch.object(
|
|
hw,
|
|
"get_visible_gpu_utilization",
|
|
return_value = self._make_utilization(
|
|
[
|
|
(0, 40.0, 30.0), # 30GB free
|
|
(1, 40.0, 35.0), # 35GB free
|
|
]
|
|
),
|
|
),
|
|
):
|
|
selected, meta = auto_select_gpu_ids("test/model")
|
|
# 35GB (first) + 30*0.85 (second) = 60.5GB > 50GB
|
|
self.assertEqual(len(selected), 2)
|
|
|
|
def test_non_cuda_returns_none(self):
|
|
from utils.hardware.hardware import auto_select_gpu_ids
|
|
import utils.hardware.hardware as hw
|
|
|
|
with patch.object(hw, "get_device", return_value = hw.DeviceType.CPU):
|
|
selected, meta = auto_select_gpu_ids("test/model")
|
|
self.assertIsNone(selected)
|
|
self.assertEqual(meta["selection_mode"], "non_cuda")
|
|
|
|
|
|
class TestGetDeviceMap(unittest.TestCase):
|
|
"""Test device_map string generation."""
|
|
|
|
def test_single_gpu_returns_sequential(self):
|
|
from utils.hardware.hardware import get_device_map
|
|
import utils.hardware.hardware as hw
|
|
|
|
with (
|
|
patch.object(hw, "get_device", return_value = hw.DeviceType.CUDA),
|
|
patch.object(
|
|
hw,
|
|
"_get_parent_visible_gpu_spec",
|
|
return_value = {
|
|
"raw": "0",
|
|
"numeric_ids": [0],
|
|
"supports_explicit_gpu_ids": True,
|
|
},
|
|
),
|
|
patch.object(hw, "get_visible_gpu_count", return_value = 1),
|
|
):
|
|
dm = get_device_map(gpu_ids = [0])
|
|
self.assertEqual(dm, "sequential")
|
|
|
|
def test_multi_gpu_returns_balanced(self):
|
|
from utils.hardware.hardware import get_device_map
|
|
import utils.hardware.hardware as hw
|
|
|
|
with patch.object(hw, "get_device", return_value = hw.DeviceType.CUDA):
|
|
dm = get_device_map(gpu_ids = [0, 1])
|
|
self.assertEqual(dm, "balanced")
|
|
|
|
def test_cpu_returns_sequential(self):
|
|
from utils.hardware.hardware import get_device_map
|
|
import utils.hardware.hardware as hw
|
|
|
|
with patch.object(hw, "get_device", return_value = hw.DeviceType.CPU):
|
|
dm = get_device_map(gpu_ids = None)
|
|
self.assertEqual(dm, "sequential")
|
|
|
|
|
|
class TestResolveRequestedGpuIds(unittest.TestCase):
|
|
"""Test GPU ID validation."""
|
|
|
|
def test_none_returns_parent_visible(self):
|
|
from utils.hardware.hardware import resolve_requested_gpu_ids
|
|
|
|
with (
|
|
patch.dict(os.environ, {"CUDA_VISIBLE_DEVICES": "2,3"}, clear = False),
|
|
patch("utils.hardware.hardware.get_physical_gpu_count", return_value = 8),
|
|
):
|
|
result = resolve_requested_gpu_ids(None)
|
|
self.assertEqual(result, [2, 3])
|
|
|
|
def test_empty_list_returns_parent_visible(self):
|
|
from utils.hardware.hardware import resolve_requested_gpu_ids
|
|
|
|
with (
|
|
patch.dict(os.environ, {"CUDA_VISIBLE_DEVICES": "2,3"}, clear = False),
|
|
patch("utils.hardware.hardware.get_physical_gpu_count", return_value = 8),
|
|
):
|
|
result = resolve_requested_gpu_ids([])
|
|
self.assertEqual(result, [2, 3])
|
|
|
|
def test_duplicates_rejected(self):
|
|
from utils.hardware.hardware import resolve_requested_gpu_ids
|
|
|
|
with (
|
|
patch.dict(os.environ, {"CUDA_VISIBLE_DEVICES": "0,1,2"}, clear = False),
|
|
patch("utils.hardware.hardware.get_physical_gpu_count", return_value = 8),
|
|
):
|
|
with self.assertRaises(ValueError):
|
|
resolve_requested_gpu_ids([1, 1])
|
|
|
|
def test_out_of_range_rejected(self):
|
|
from utils.hardware.hardware import resolve_requested_gpu_ids
|
|
|
|
with (
|
|
patch.dict(os.environ, {"CUDA_VISIBLE_DEVICES": "0,1"}, clear = False),
|
|
patch("utils.hardware.hardware.get_physical_gpu_count", return_value = 4),
|
|
):
|
|
with self.assertRaises(ValueError):
|
|
resolve_requested_gpu_ids([5])
|
|
|
|
def test_uuid_env_var_rejects_explicit_ids(self):
|
|
from utils.hardware.hardware import resolve_requested_gpu_ids
|
|
|
|
with (
|
|
patch.dict(
|
|
os.environ, {"CUDA_VISIBLE_DEVICES": "GPU-abc,GPU-def"}, clear = False
|
|
),
|
|
patch("utils.hardware.hardware.get_physical_gpu_count", return_value = 8),
|
|
):
|
|
with self.assertRaises(ValueError):
|
|
resolve_requested_gpu_ids([0])
|
|
|
|
|
|
class TestApplyGpuIds(unittest.TestCase):
|
|
"""Test CUDA_VISIBLE_DEVICES environment variable setting."""
|
|
|
|
def test_apply_list(self):
|
|
from utils.hardware.hardware import apply_gpu_ids
|
|
|
|
with patch.dict(os.environ, {}, clear = False):
|
|
apply_gpu_ids([3, 5])
|
|
self.assertEqual(os.environ.get("CUDA_VISIBLE_DEVICES"), "3,5")
|
|
|
|
def test_apply_none_does_nothing(self):
|
|
from utils.hardware.hardware import apply_gpu_ids
|
|
|
|
original = os.environ.get("CUDA_VISIBLE_DEVICES")
|
|
apply_gpu_ids(None)
|
|
self.assertEqual(os.environ.get("CUDA_VISIBLE_DEVICES"), original)
|
|
|
|
|
|
class TestMultiGpuOverheadAccounting(unittest.TestCase):
|
|
"""Test that multi-GPU overhead is applied correctly.
|
|
|
|
The first GPU should keep its full free memory, and only
|
|
additional GPUs should have the overhead factor applied.
|
|
"""
|
|
|
|
def _make_utilization(self, devices):
|
|
return {
|
|
"available": True,
|
|
"devices": [
|
|
{
|
|
"index": idx,
|
|
"vram_total_gb": total,
|
|
"vram_used_gb": total - free,
|
|
}
|
|
for idx, total, free in devices
|
|
],
|
|
}
|
|
|
|
def test_first_gpu_not_penalized(self):
|
|
"""A model that just fits on 1 GPU should not require 2 GPUs."""
|
|
from utils.hardware.hardware import auto_select_gpu_ids
|
|
import utils.hardware.hardware as hw
|
|
|
|
# Model requires 79GB, GPU has 80GB free
|
|
with (
|
|
patch.object(hw, "get_device", return_value = hw.DeviceType.CUDA),
|
|
patch.object(
|
|
hw,
|
|
"estimate_required_model_memory_gb",
|
|
return_value = (
|
|
79.0,
|
|
{
|
|
"mode": "inference",
|
|
"required_gb": 79.0,
|
|
"model_size_source": "config",
|
|
"model_size_gb": 60.0,
|
|
},
|
|
),
|
|
),
|
|
patch.object(
|
|
hw,
|
|
"_get_parent_visible_gpu_spec",
|
|
return_value = {
|
|
"raw": "0,1",
|
|
"numeric_ids": [0, 1],
|
|
"supports_explicit_gpu_ids": True,
|
|
},
|
|
),
|
|
patch.object(hw, "get_parent_visible_gpu_ids", return_value = [0, 1]),
|
|
patch.object(
|
|
hw,
|
|
"get_visible_gpu_utilization",
|
|
return_value = self._make_utilization(
|
|
[
|
|
(0, 80.0, 80.0),
|
|
(1, 80.0, 80.0),
|
|
]
|
|
),
|
|
),
|
|
):
|
|
selected, meta = auto_select_gpu_ids("test/model")
|
|
# Should fit on 1 GPU (80GB >= 79GB)
|
|
self.assertEqual(len(selected), 1)
|
|
|
|
def test_second_gpu_has_overhead(self):
|
|
"""When 2 GPUs are needed, the second one's contribution is reduced."""
|
|
from utils.hardware.hardware import auto_select_gpu_ids
|
|
import utils.hardware.hardware as hw
|
|
|
|
# Model requires 110GB. First GPU has 80GB, second has 40GB.
|
|
# With overhead: 80 + 40*0.85 = 114GB -- just enough
|
|
with (
|
|
patch.object(hw, "get_device", return_value = hw.DeviceType.CUDA),
|
|
patch.object(
|
|
hw,
|
|
"estimate_required_model_memory_gb",
|
|
return_value = (
|
|
110.0,
|
|
{
|
|
"mode": "inference",
|
|
"required_gb": 110.0,
|
|
"model_size_source": "config",
|
|
"model_size_gb": 85.0,
|
|
},
|
|
),
|
|
),
|
|
patch.object(
|
|
hw,
|
|
"_get_parent_visible_gpu_spec",
|
|
return_value = {
|
|
"raw": "0,1",
|
|
"numeric_ids": [0, 1],
|
|
"supports_explicit_gpu_ids": True,
|
|
},
|
|
),
|
|
patch.object(hw, "get_parent_visible_gpu_ids", return_value = [0, 1]),
|
|
patch.object(
|
|
hw,
|
|
"get_visible_gpu_utilization",
|
|
return_value = self._make_utilization(
|
|
[
|
|
(0, 80.0, 80.0),
|
|
(1, 80.0, 40.0),
|
|
]
|
|
),
|
|
),
|
|
):
|
|
selected, meta = auto_select_gpu_ids("test/model")
|
|
# Should use both GPUs
|
|
self.assertEqual(len(selected), 2)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|