ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
963 commits 391 branches 58 tags 304 MiB
Commit graph

62 commits

Author SHA1 Message Date
Roland Tannous
a0f03d3080 Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
samit
6aa50d353f exposed trust_remote_code through the UI 2026-03-08 16:28:56 -07:00
Manan17
111caf636f Audio_VLM bug fix 2026-03-08 19:14:07 +00:00
Roland Tannous
08a6cf0c87 feat: route audio inference (TTS, ASR, Whisper) through orchestrator/worker subprocess 2026-03-08 18:25:27 +00:00
Roland Tannous
7db2c90cc6 merge nightly into audio branch (mock test) 2026-03-08 10:23:44 +00:00
Roland Tannous
ff85a45050 fix: clear stale model state on failed inference subprocess reload 2026-03-07 23:32:53 +00:00
Roland Tannous
dcebfe718a fix: prevent training hang on Windows by adding triton-windows support 2026-03-07 17:53:36 +00:00
Roland Tannous
c882a3d2f7 fix: propagate PYTHONPATH to child subprocesses, revert tokenizer patching 2026-03-07 11:28:24 +00:00
Roland Tannous
42bd976a2f fix: patch TokenizersBackend by model name - Qwen3.5→Qwen2Tokenizer, GLM→PreTrainedTokenizer 2026-03-07 10:29:59 +00:00
Roland Tannous
44e9b838ae fix: patch Qwen3.5 broken tokenizer_class TokenizersBackend across all backends 2026-03-07 09:43:25 +00:00
Roland Tannous
f101befca7 fix: bump transformers 5.x pin from 5.1.0 to 5.2.0 for Qwen3.5 support 2026-03-07 09:10:09 +00:00
Roland Tannous
2c4c598832 fix: fail fast if runtime pip install of transformers 5.x fails 2026-03-07 08:40:25 +00:00
Roland Tannous
717d5d621d fix: drain stale events from resp_queue after generation cancel 2026-03-07 08:12:16 +00:00
Roland Tannous
2558125a14 fix: serialize generation with _gen_lock to prevent concurrent queue readers 
Two overlapping /chat/completions requests could both read from the shared
resp_queue, consuming and dropping each other's token events. Replace the
request_id filtering (which silently dropped non-matching messages) with a
threading.Lock that serializes generation — correct for single-GPU inference.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-07 04:06:51 +00:00
Roland Tannous
7e59440029 fix: pin huggingface_hub==1.3.0 in .venv_t5 (satisfies transformers 5.x) 2026-03-06 06:19:28 +00:00
Roland Tannous
661ac4be96 feat: subprocess-based export, pin huggingface_hub==0.36.0 2026-03-06 06:03:09 +00:00
Roland Tannous
e866d16787 fix: indentation error in orchestrator load_model 2026-03-05 19:43:30 +00:00
Roland Tannous
b598cf0c14 fix: always spawn fresh subprocess per model load 
Reusing a subprocess after unsloth patches torch internals causes
inspect.getsource() failures when loading a different model type.
Each load now gets a clean Python interpreter.
 2026-03-05 19:15:37 +00:00
Roland Tannous
5c3efd610f fix: use mp.Event for instant cross-process generation cancel 
Replaces cmd_queue-based cancel polling with a shared mp.Event.
Fixes two issues:
- Loading a new model while generating no longer hangs (cancel is instant)
- Subprocess shuts down cleanly after explicit stop generation
 2026-03-05 18:54:17 +00:00
Roland Tannous
b70faf8cb7 feat: subprocess-based inference for transformers version switching 
Inference now runs in a persistent subprocess, solving the same
transformers version-switching problem that was fixed for training.
The subprocess stays alive between requests (model in GPU memory)
and is only restarted when switching transformers versions.

New files:
- core/inference/worker.py: subprocess entry point with command loop
- core/inference/orchestrator.py: parent-side proxy with same API

Modified:
- core/inference/__init__.py: exports orchestrator as default backend
- routes/inference.py: removed in-process ensure_transformers_version()
 2026-03-05 17:47:57 +00:00
Manan17
8203637d89 resolved merge conflicts 2026-03-05 07:59:43 +00:00
Roland Tannous
14a8fbc650 Merge pull request #287 from unslothai/fix/duplicate-def-inference 
Deleted duplicate definitions for load_for_eval, load_adapter, and load_model_simple in core Inference
 2026-03-04 10:06:04 +04:00
Roland Tannous
5f98d232d0 fix: align llama-server binary discovery with upstream unsloth-zoo paths 2026-03-03 17:03:01 +00:00
Roland Tannous
da00f5ed1d Merge branch 'nightly' into feature/support-for-audio-models 2026-03-02 15:55:25 +04:00
Roland Tannous
e7619a1291 Move llama.cpp clone/build from in-tree to ~/.unsloth/llama.cpp 
- setup.sh: builds at ~/.unsloth/llama.cpp instead of ./llama.cpp
- setup.ps1: builds at %USERPROFILE%/.unsloth/llama.cpp
- inference llama_cpp.py: searches ~/.unsloth/ first, in-tree as legacy
- export.py: updated comments (unsloth-zoo handles path natively)
 2026-03-02 04:04:41 +00:00
Roland Tannous
70d1567fe3 Download GGUF via huggingface_hub instead of llama-server -hf (fixes HTTPS not supported on Windows) 2026-03-01 13:05:10 +00:00
Roland Tannous
0430d22cc2 Auto-add CUDA DLLs to PATH when launching llama-server on Windows 2026-03-01 13:05:10 +00:00
Roland Tannous
afa1344452 Build llama.cpp in-tree, auto-detect driver CUDA version for compatible toolkit 2026-03-01 13:05:10 +00:00
Roland Tannous
af90c9c3d2 Fix llama-server binary lookup for Windows (.exe, Release dir, ~/.unsloth) 2026-03-01 13:05:10 +00:00
Roland Tannous
986bef4f99 fix: support mmproj for local vision GGUF models + fix Windows pipe deadlock 2026-03-01 12:58:38 +00:00
Manan17
8cdeb006b6 code cleanup 2026-03-01 08:04:38 +00:00
Manan17
9e89f31bc7 revamping up the code and adding inference 2026-03-01 02:30:31 +00:00
samit
46b1a78e52 deleted duplicate definitions 2026-02-27 06:00:28 -08:00
Manan17
b4311cca82 Aggregating sharded models, showing fit/oom for quantizations 2026-02-27 08:23:15 +00:00
Roland Tannous
efaa0bacfb Merge branch 'nightly' into feat/gguf-llama-cpp-inference 2026-02-25 16:06:03 +04:00
Roland Tannous
2ebeba8588 Switch GGUF backend from /v1/completions to /v1/chat/completions 
Fixes two bugs:
1. Chat template tags (<|im_start|>, <|im_end|>) leaking into output
   because /v1/completions treated them as literal text
2. Image hallucination because image_b64 was never passed to llama-server

Now llama-server handles chat templates natively and receives images
as OpenAI-format multimodal content parts for vision models.
 2026-02-24 19:21:01 +04:00
Roland Tannous
3ee4f1359a Use llama-server -hf mode, add GGUF variant selector, fix vision detection 
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
 2026-02-24 19:03:06 +04:00
Roland Tannous
5b7555cd3f Fix llama-server: build in-tree, fix path resolution, add LD_LIBRARY_PATH 2026-02-24 18:19:29 +04:00
Roland Tannous
2f985ccbb5 Add GGUF model inference via llama-server backend 2026-02-24 17:40:05 +04:00
Manan17
6fa6b0bf28 Fixing base model export issue for vlms 2026-02-24 01:34:11 +00:00
Roland Tannous
198433363a feat: clear unsloth_compiled_cache on startup, shutdown, and between model loads 2026-02-23 07:26:22 +00:00
Roland Tannous
ef118d0d05 fix: load proper vision processor from base model when FastVisionModel returns raw tokenizer, add tokenize=False to vision chat template 2026-02-21 04:40:29 +00:00
Manan17
e9710874e1 Mapping proper tokenizer for VLMs 2026-02-21 01:57:05 +00:00
Manan17
756aa56cd2 fixed the vlm's text only errors 2026-02-20 22:23:26 +00:00
Manan17
444ece6b07 Fixing compare feature 2026-02-19 20:15:44 +00:00
Shine1i
4be6eefed3 feat: support disabling top-k sampling with -1 and standardize normalization logic 
- Updated top-k parameter range to accept -1 in models and frontend.
- Added utility to normalize top-k for backend compatibility.
 2026-02-16 21:33:24 +01:00
Roland Tannous
6b839a1481 feat: add min_p sampling parameter to /chat/completions generation pipeline 2026-02-16 06:33:17 +00:00
Shine1i
f6397bf1ac feat: add cancelation support for chat generation and streaming tasks 2026-02-15 18:23:27 +01:00
Roland Tannous
4399687f93 strip extra debug statements 2026-02-14 19:23:51 +00:00
Roland Tannous
4d868e8d2b replace model unloading and peft loading mechanism for compare feature 2026-02-14 19:18:49 +00:00