* Studio: tune llama.cpp env for data-center GPUs
Detect datacenter/professional NVIDIA GPUs at llama-server launch and set
the llama.cpp env flags that help them, gated so consumer GeForce, AMD/ROCm,
CPU and macOS are never touched.
- GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1 for any DC GPU (FP32 cuBLAS
accumulation). On a B200 this is ~0% throughput cost with identical
perplexity (7.3230 wikitext-2-raw, baseline and on), where on GeForce the
same flag costs real throughput, hence the gate.
- GGML_CUDA_P2P=1 and CUDA_SCALE_LAUNCH_QUEUES=4x for multi-GPU DC boxes.
Benchmarked on 6x B200: +33-51% prompt processing on tensor (row) split and
+8-16% on the default pipeline (layer) split, with no regression on the
other split or on token generation.
Detection uses torch device names (A100/A30/H100/H200/H800/GH200/B200/GB200/
GB300/L40/L4/RTX PRO 6000/RTX 6000 Ada). A mixed box with one consumer GPU in
the selection is treated as non-DC. All writes are setdefault so a user value
always wins, and UNSLOTH_DISABLE_DC_TUNING=1 turns the whole thing off.
37 unit tests cover detection, multi-GPU gating, user-override precedence, the
disable flag and fail-open on error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix data-center GPU detection false positives and physical-id mapping
Two issues in the data-center llama.cpp env tuning gate:
- _is_datacenter_gpu matched the marker allowlist as unbounded substrings, so
workstation/laptop parts "NVIDIA RTX A1000" and "NVIDIA RTX A3000" matched
"a100"/"a30" and were wrongly tuned as data-center GPUs (forcing FP32 cuBLAS
accumulation and the multi-GPU env, which carry a real cost on those cards).
Switch to a word-boundary regex.
- gpu_indices carries physical GPU ids (translated from torch ordinals by
_get_gpu_free_memory via CUDA_VISIBLE_DEVICES), but they were passed straight
into torch.cuda.get_device_properties, which expects mask-relative ordinals.
On a masked host (e.g. CUDA_VISIBLE_DEVICES=4,5,6,7) a selection like [4,5]
fell out of range and silently dropped the tuning, and on a mixed mask it could
probe the wrong GPU class. Build a physical-id to device-name map mirroring
_get_gpu_free_memory, then look up the selection by physical id.
Add regression tests for the A1000/A3000 false positives and for masked-host
physical-id selection (reordered and mixed-class masks included).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten data-center GPU tuning comments
Comment-only pass over the DC tuning block and its tests: shorten verbose
docstrings/comments, drop ones that restate the code, collapse multi-line
blocks. Keep the load-bearing rationale (physical-id vs ordinal mapping, the
word-boundary reason, the B200 benchmark numbers). No code change: verified
with comment_tools.py check --strip-docstrings (code unchanged, comments only).
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han