ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: pair --numa distribute with numactl --interleave=all

Browse source 
Per llama.cpp multi-NUMA guidance (ggml-org/llama.cpp#19102), numactl
--interleave=all alone does not spread pages evenly across nodes; it must be paired
with --numa distribute. Add it when auto-interleave engages, unless the user already
set --numa.
This commit is contained in:
Daniel Han 2026-06-29 05:37:35 +00:00
commit 9af797ed5c
1 changed files with 7 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

10
studio/backend/core/inference/llama_cpp.py
View file

@ -5958,15 +5958,19 @@ class LlamaCppBackend:
cmd.extend(str(a) for a in extra_args)
logger.info(f"Appending user extra args to llama-server: {list(extra_args)}")
# NUMA auto-interleave: wrap with `numactl --interleave=all` when the
# model overflows one node but fits across all (else first-touch
# thrashes/OOMs). Applied at the Popen sites; decided before any spawn.
# NUMA auto-interleave: when the model overflows one node but fits
# across all (else first-touch thrashes/OOMs), wrap with
# `numactl --interleave=all` (applied at the Popen sites) AND pass
# `--numa distribute` -- both are needed, numactl alone doesn't spread
# pages evenly (llama.cpp #19102). User --numa wins.
self._numa_prefix = []
try:
from core.inference.numa import decide_interleave
_numa = decide_interleave(model_size, cpu_only = _cpu_only)
if _numa.interleave:
self._numa_prefix = list(_numa.prefix)
if not _extra_args_set_any_flag(extra_args, {"--numa"}):
cmd.extend(["--numa", "distribute"])
logger.info("NUMA: %s", _numa.reason)
elif _cpu_only and "numactl` is not installed" in _numa.reason:
logger.warning("NUMA: %s", _numa.reason)