5603ced75f
_get_gpu_free_memory was filtering by CUDA_VISIBLE_DEVICES, so with CUDA_VISIBLE_DEVICES='0' set by the training env, llama-server only saw 1 GPU and used --fit for CPU offloading instead of spreading across all 8 GPUs. Since llama-server manages its own GPU allocation (the _select_gpus method picks GPUs and sets CUDA_VISIBLE_DEVICES for the subprocess), the query must see ALL physical GPUs to make the right decision. |
||
|---|---|---|
| .. | ||
| assets | ||
| auth | ||
| core | ||
| loggers | ||
| models | ||
| plugins | ||
| requirements | ||
| routes | ||
| state | ||
| tests | ||
| utils | ||
| __init__.py | ||
| colab.py | ||
| main.py | ||
| run.py | ||