ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
772 commits 391 branches 58 tags 304 MiB
Commit graph

38 commits

Author SHA1 Message Date
Roland Tannous
2d7d3cd27e Merge pull request #287 from unslothai/fix/duplicate-def-inference 
Deleted duplicate definitions for load_for_eval, load_adapter, and load_model_simple in core Inference
 2026-03-04 10:06:04 +04:00
Roland Tannous
a4d2853fbc fix: align llama-server binary discovery with upstream unsloth-zoo paths 2026-03-03 17:03:01 +00:00
Roland Tannous
e280e457d1 Move llama.cpp clone/build from in-tree to ~/.unsloth/llama.cpp 
- setup.sh: builds at ~/.unsloth/llama.cpp instead of ./llama.cpp
- setup.ps1: builds at %USERPROFILE%/.unsloth/llama.cpp
- inference llama_cpp.py: searches ~/.unsloth/ first, in-tree as legacy
- export.py: updated comments (unsloth-zoo handles path natively)
 2026-03-02 04:04:41 +00:00
Roland Tannous
6e5a3d1744 Download GGUF via huggingface_hub instead of llama-server -hf (fixes HTTPS not supported on Windows) 2026-03-01 13:05:10 +00:00
Roland Tannous
12867f701b Auto-add CUDA DLLs to PATH when launching llama-server on Windows 2026-03-01 13:05:10 +00:00
Roland Tannous
3521de7040 Build llama.cpp in-tree, auto-detect driver CUDA version for compatible toolkit 2026-03-01 13:05:10 +00:00
Roland Tannous
f036a70681 Fix llama-server binary lookup for Windows (.exe, Release dir, ~/.unsloth) 2026-03-01 13:05:10 +00:00
Roland Tannous
ff93c97024 fix: support mmproj for local vision GGUF models + fix Windows pipe deadlock 2026-03-01 12:58:38 +00:00
samit
862b4100d2 deleted duplicate definitions 2026-02-27 06:00:28 -08:00
Manan17
168957a87a Aggregating sharded models, showing fit/oom for quantizations 2026-02-27 08:23:15 +00:00
Roland Tannous
01082b84e5 Merge branch 'nightly' into feat/gguf-llama-cpp-inference 2026-02-25 16:06:03 +04:00
Roland Tannous
0e7c8a2e5e Switch GGUF backend from /v1/completions to /v1/chat/completions 
Fixes two bugs:
1. Chat template tags (<|im_start|>, <|im_end|>) leaking into output
   because /v1/completions treated them as literal text
2. Image hallucination because image_b64 was never passed to llama-server

Now llama-server handles chat templates natively and receives images
as OpenAI-format multimodal content parts for vision models.
 2026-02-24 19:21:01 +04:00
Roland Tannous
ef1cd3ac98 Use llama-server -hf mode, add GGUF variant selector, fix vision detection 
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
 2026-02-24 19:03:06 +04:00
Roland Tannous
08aeeaee4b Fix llama-server: build in-tree, fix path resolution, add LD_LIBRARY_PATH 2026-02-24 18:19:29 +04:00
Roland Tannous
a40ebb1aab Add GGUF model inference via llama-server backend 2026-02-24 17:40:05 +04:00
Manan17
aeb198f52d Fixing base model export issue for vlms 2026-02-24 01:34:11 +00:00
Roland Tannous
dbbcdb4f09 feat: clear unsloth_compiled_cache on startup, shutdown, and between model loads 2026-02-23 07:26:22 +00:00
Roland Tannous
c051e3d532 fix: load proper vision processor from base model when FastVisionModel returns raw tokenizer, add tokenize=False to vision chat template 2026-02-21 04:40:29 +00:00
Manan17
f6ebeb1d42 Mapping proper tokenizer for VLMs 2026-02-21 01:57:05 +00:00
Manan17
3fa9e773c2 fixed the vlm's text only errors 2026-02-20 22:23:26 +00:00
Manan17
fdeccec259 Fixing compare feature 2026-02-19 20:15:44 +00:00
Shine1i
0db7da96cc feat: support disabling top-k sampling with -1 and standardize normalization logic 
- Updated top-k parameter range to accept -1 in models and frontend.
- Added utility to normalize top-k for backend compatibility.
 2026-02-16 21:33:24 +01:00
Roland Tannous
909955767b feat: add min_p sampling parameter to /chat/completions generation pipeline 2026-02-16 06:33:17 +00:00
Shine1i
571959e383 feat: add cancelation support for chat generation and streaming tasks 2026-02-15 18:23:27 +01:00
Roland Tannous
be3934860f strip extra debug statements 2026-02-14 19:23:51 +00:00
Roland Tannous
3ff3def555 replace model unloading and peft loading mechanism for compare feature 2026-02-14 19:18:49 +00:00
Roland Tannous
e7ae901737 del model.peft_config instead of using model.delete_adapter 2026-02-14 17:32:15 +00:00
Roland Tannous
7d8e991c1f added print statements for activate_lora_adapter 2026-02-14 17:25:37 +00:00
Roland Tannous
6fefbe9f0b swipped logger for print statements as logger isn't propagating 2026-02-14 17:21:26 +00:00
Roland Tannous
d0b94eae75 added logging 2026-02-14 17:09:07 +00:00
Roland Tannous
b5c8136957 exclude default from model.delete_adapter 2026-02-14 17:03:52 +00:00
Roland Tannous
35a6e40268 _apply_adapter_state now calls revert_to_base_model and activate_lora_adapter properly 2026-02-14 16:57:24 +00:00
Roland Tannous
f67ee58347 feat(inference): add use_adapter field for per-request adapter toggling in compare mode 2026-02-14 14:52:13 +00:00
Roland Tannous
418a374125 migrate _generate_vision_response to use TextIteratorStreamer + background thread 2026-02-14 09:30:32 +00:00
Roland Tannous
da1cde971c use get_device() for device selection and clear_gpu_cache() for GPU memory cleanup in inference, trainer, and export 2026-02-11 16:56:52 +00:00
Roland Tannous
b4ec0389f0 refactor/inference-api-routes-part-1 2026-02-03 16:57:57 +00:00
Roland Tannous
62ddcfa019 Refactor [dataset_utils.py](cci:7://file:///home/support/new-ui-prototype/studio/backend/utils/datasets/dataset_utils.py:0:0-0:0) into focused modules 2026-02-03 14:38:02 +00:00
Roland Tannous
544d6944d1 root studio folder 2026-02-02 09:13:49 +00:00