Studio: default llama-server --threads to -1 (auto)
Previously we passed --threads only when the caller set an explicit value, which meant llama-server fell back to its internal default. That default has varied across llama.cpp builds (some versions use hardware concurrency including hyperthreads, which hurts throughput on CPU-heavy inference). Always passing --threads -1 pins the behaviour to llama.cpp's auto-detect (physical cores). Caller-supplied n_threads still wins when non-None.
This commit is contained in:
parent
6ced30cdaa
commit
4c86ee4bff
1 changed files with 4 additions and 2 deletions
|
|
@ -1528,8 +1528,10 @@ class LlamaCppBackend:
|
|||
# Model fits on selected GPU(s) -- offload all layers
|
||||
cmd.extend(["-ngl", "-1"])
|
||||
|
||||
if n_threads is not None:
|
||||
cmd.extend(["--threads", str(n_threads)])
|
||||
# -1 = llama.cpp auto-detect (physical cores). Pass explicitly so we
|
||||
# do not inherit llama-server's internal default, which has historically
|
||||
# varied (hardware concurrency incl. hyperthreads on some builds).
|
||||
cmd.extend(["--threads", str(n_threads if n_threads is not None else -1)])
|
||||
|
||||
# Always enable Jinja chat template rendering for proper template support
|
||||
cmd.extend(["--jinja"])
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue