f8dc7c9a5c
Three improvements to the llama.cpp build step in setup.sh: 1. Detect GPU compute capability via nvidia-smi and limit CMAKE_CUDA_ARCHITECTURES to the current GPU. Without this, cmake builds for all default CUDA architectures which is very slow. 2. Use Ninja build generator when available. Ninja has better parallelism than Make for CUDA compilation. 3. Build both llama-server and llama-quantize targets in a single cmake --build invocation for better parallelism. 4. Add --threads=0 to CMAKE_CUDA_FLAGS for multi-threaded nvcc compilation. Measured on 192-core machine with B200 (sm_100): Make (all archs): very slow (minutes for each arch) Make (single arch): 1m 37s Ninja (single arch): 55s Speedup: ~1.7x Combined with the uv change, total setup goes from ~4m 35s to ~1m 40s. |
||
|---|---|---|
| .. | ||
| backend | ||
| frontend | ||
| __init__.py | ||
| install_python_stack.py | ||
| LICENSE.AGPL-3.0 | ||
| setup.bat | ||
| setup.ps1 | ||
| setup.sh | ||
| Unsloth_Studio_Colab.ipynb | ||