93ec05ced2
Two changes for GGUF variant selection: 1. Default variant preference now starts with UD-Q4_K_XL (Unsloth Dynamic quantization) which provides better quality per bit than standard Q4_K_M. Also added UD-Q2_K_XL, UD-IQ2_M, UD-IQ1_M, UD-IQ1_S as small fallback options. 2. If the selected variant doesn't fit on disk, automatically fall back to the smallest GGUF variant in the repo that does fit. Queries all GGUF file sizes via get_paths_info() and picks the smallest one under the free disk space limit. If nothing fits, raises a clear error. This means users with limited disk space won't get a download error -- they'll get a smaller quantization instead. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| audio_codecs.py | ||
| inference.py | ||
| llama_cpp.py | ||
| orchestrator.py | ||
| worker.py | ||