Roland Tannous
a0f03d3080
Add AGPL-3.0 SPDX headers to all source files
2026-03-09 20:17:45 +00:00
Roland Tannous
5f98d232d0
fix: align llama-server binary discovery with upstream unsloth-zoo paths
2026-03-03 17:03:01 +00:00
Roland Tannous
e7619a1291
Move llama.cpp clone/build from in-tree to ~/.unsloth/llama.cpp
...
- setup.sh: builds at ~/.unsloth/llama.cpp instead of ./llama.cpp
- setup.ps1: builds at %USERPROFILE%/.unsloth/llama.cpp
- inference llama_cpp.py: searches ~/.unsloth/ first, in-tree as legacy
- export.py: updated comments (unsloth-zoo handles path natively)
2026-03-02 04:04:41 +00:00
Roland Tannous
70d1567fe3
Download GGUF via huggingface_hub instead of llama-server -hf (fixes HTTPS not supported on Windows)
2026-03-01 13:05:10 +00:00
Roland Tannous
0430d22cc2
Auto-add CUDA DLLs to PATH when launching llama-server on Windows
2026-03-01 13:05:10 +00:00
Roland Tannous
afa1344452
Build llama.cpp in-tree, auto-detect driver CUDA version for compatible toolkit
2026-03-01 13:05:10 +00:00
Roland Tannous
af90c9c3d2
Fix llama-server binary lookup for Windows (.exe, Release dir, ~/.unsloth)
2026-03-01 13:05:10 +00:00
Roland Tannous
986bef4f99
fix: support mmproj for local vision GGUF models + fix Windows pipe deadlock
2026-03-01 12:58:38 +00:00
Manan17
b4311cca82
Aggregating sharded models, showing fit/oom for quantizations
2026-02-27 08:23:15 +00:00
Roland Tannous
2ebeba8588
Switch GGUF backend from /v1/completions to /v1/chat/completions
...
Fixes two bugs:
1. Chat template tags (<|im_start|>, <|im_end|>) leaking into output
because /v1/completions treated them as literal text
2. Image hallucination because image_b64 was never passed to llama-server
Now llama-server handles chat templates natively and receives images
as OpenAI-format multimodal content parts for vision models.
2026-02-24 19:21:01 +04:00
Roland Tannous
3ee4f1359a
Use llama-server -hf mode, add GGUF variant selector, fix vision detection
...
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
2026-02-24 19:03:06 +04:00
Roland Tannous
5b7555cd3f
Fix llama-server: build in-tree, fix path resolution, add LD_LIBRARY_PATH
2026-02-24 18:19:29 +04:00
Roland Tannous
2f985ccbb5
Add GGUF model inference via llama-server backend
2026-02-24 17:40:05 +04:00
