ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
958 commits 391 branches 58 tags 304 MiB
Commit graph

61 commits

Author SHA1 Message Date
samit
86e94b5844 exposed trust_remote_code through the UI 2026-03-08 16:28:56 -07:00
Manan17
80b704d7b7 Audio_VLM bug fix 2026-03-08 19:14:07 +00:00
Roland Tannous
7ee81dd7df feat: route audio inference (TTS, ASR, Whisper) through orchestrator/worker subprocess 2026-03-08 18:25:27 +00:00
Roland Tannous
1435dbaf59 merge nightly into audio branch (mock test) 2026-03-08 10:23:44 +00:00
Roland Tannous
a7c34b42be fix: clear stale model state on failed inference subprocess reload 2026-03-07 23:32:53 +00:00
Roland Tannous
ef9184c731 fix: prevent training hang on Windows by adding triton-windows support 2026-03-07 17:53:36 +00:00
Roland Tannous
e25705a211 fix: propagate PYTHONPATH to child subprocesses, revert tokenizer patching 2026-03-07 11:28:24 +00:00
Roland Tannous
76c78afb8f fix: patch TokenizersBackend by model name - Qwen3.5→Qwen2Tokenizer, GLM→PreTrainedTokenizer 2026-03-07 10:29:59 +00:00
Roland Tannous
d60cd2843f fix: patch Qwen3.5 broken tokenizer_class TokenizersBackend across all backends 2026-03-07 09:43:25 +00:00
Roland Tannous
bd60562145 fix: bump transformers 5.x pin from 5.1.0 to 5.2.0 for Qwen3.5 support 2026-03-07 09:10:09 +00:00
Roland Tannous
0b3397cc3a fix: fail fast if runtime pip install of transformers 5.x fails 2026-03-07 08:40:25 +00:00
Roland Tannous
728420b290 fix: drain stale events from resp_queue after generation cancel 2026-03-07 08:12:16 +00:00
Roland Tannous
609e3168a1 fix: serialize generation with _gen_lock to prevent concurrent queue readers 
Two overlapping /chat/completions requests could both read from the shared
resp_queue, consuming and dropping each other's token events. Replace the
request_id filtering (which silently dropped non-matching messages) with a
threading.Lock that serializes generation — correct for single-GPU inference.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-07 04:06:51 +00:00
Roland Tannous
c3bc19494f fix: pin huggingface_hub==1.3.0 in .venv_t5 (satisfies transformers 5.x) 2026-03-06 06:19:28 +00:00
Roland Tannous
6b32af0bdc feat: subprocess-based export, pin huggingface_hub==0.36.0 2026-03-06 06:03:09 +00:00
Roland Tannous
31334cece1 fix: indentation error in orchestrator load_model 2026-03-05 19:43:30 +00:00
Roland Tannous
5bd6fac80e fix: always spawn fresh subprocess per model load 
Reusing a subprocess after unsloth patches torch internals causes
inspect.getsource() failures when loading a different model type.
Each load now gets a clean Python interpreter.
 2026-03-05 19:15:37 +00:00
Roland Tannous
7fc563731a fix: use mp.Event for instant cross-process generation cancel 
Replaces cmd_queue-based cancel polling with a shared mp.Event.
Fixes two issues:
- Loading a new model while generating no longer hangs (cancel is instant)
- Subprocess shuts down cleanly after explicit stop generation
 2026-03-05 18:54:17 +00:00
Roland Tannous
4eabc74f34 feat: subprocess-based inference for transformers version switching 
Inference now runs in a persistent subprocess, solving the same
transformers version-switching problem that was fixed for training.
The subprocess stays alive between requests (model in GPU memory)
and is only restarted when switching transformers versions.

New files:
- core/inference/worker.py: subprocess entry point with command loop
- core/inference/orchestrator.py: parent-side proxy with same API

Modified:
- core/inference/__init__.py: exports orchestrator as default backend
- routes/inference.py: removed in-process ensure_transformers_version()
 2026-03-05 17:47:57 +00:00
Manan17
9909111982 resolved merge conflicts 2026-03-05 07:59:43 +00:00
Roland Tannous
2d7d3cd27e Merge pull request #287 from unslothai/fix/duplicate-def-inference 
Deleted duplicate definitions for load_for_eval, load_adapter, and load_model_simple in core Inference
 2026-03-04 10:06:04 +04:00
Roland Tannous
a4d2853fbc fix: align llama-server binary discovery with upstream unsloth-zoo paths 2026-03-03 17:03:01 +00:00
Roland Tannous
87f2b2a9db Merge branch 'nightly' into feature/support-for-audio-models 2026-03-02 15:55:25 +04:00
Roland Tannous
e280e457d1 Move llama.cpp clone/build from in-tree to ~/.unsloth/llama.cpp 
- setup.sh: builds at ~/.unsloth/llama.cpp instead of ./llama.cpp
- setup.ps1: builds at %USERPROFILE%/.unsloth/llama.cpp
- inference llama_cpp.py: searches ~/.unsloth/ first, in-tree as legacy
- export.py: updated comments (unsloth-zoo handles path natively)
 2026-03-02 04:04:41 +00:00
Roland Tannous
6e5a3d1744 Download GGUF via huggingface_hub instead of llama-server -hf (fixes HTTPS not supported on Windows) 2026-03-01 13:05:10 +00:00
Roland Tannous
12867f701b Auto-add CUDA DLLs to PATH when launching llama-server on Windows 2026-03-01 13:05:10 +00:00
Roland Tannous
3521de7040 Build llama.cpp in-tree, auto-detect driver CUDA version for compatible toolkit 2026-03-01 13:05:10 +00:00
Roland Tannous
f036a70681 Fix llama-server binary lookup for Windows (.exe, Release dir, ~/.unsloth) 2026-03-01 13:05:10 +00:00
Roland Tannous
ff93c97024 fix: support mmproj for local vision GGUF models + fix Windows pipe deadlock 2026-03-01 12:58:38 +00:00
Manan17
c636fd5a42 code cleanup 2026-03-01 08:04:38 +00:00
Manan17
c48437848d revamping up the code and adding inference 2026-03-01 02:30:31 +00:00
samit
862b4100d2 deleted duplicate definitions 2026-02-27 06:00:28 -08:00
Manan17
168957a87a Aggregating sharded models, showing fit/oom for quantizations 2026-02-27 08:23:15 +00:00
Roland Tannous
01082b84e5 Merge branch 'nightly' into feat/gguf-llama-cpp-inference 2026-02-25 16:06:03 +04:00
Roland Tannous
0e7c8a2e5e Switch GGUF backend from /v1/completions to /v1/chat/completions 
Fixes two bugs:
1. Chat template tags (<|im_start|>, <|im_end|>) leaking into output
   because /v1/completions treated them as literal text
2. Image hallucination because image_b64 was never passed to llama-server

Now llama-server handles chat templates natively and receives images
as OpenAI-format multimodal content parts for vision models.
 2026-02-24 19:21:01 +04:00
Roland Tannous
ef1cd3ac98 Use llama-server -hf mode, add GGUF variant selector, fix vision detection 
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
 2026-02-24 19:03:06 +04:00
Roland Tannous
08aeeaee4b Fix llama-server: build in-tree, fix path resolution, add LD_LIBRARY_PATH 2026-02-24 18:19:29 +04:00
Roland Tannous
a40ebb1aab Add GGUF model inference via llama-server backend 2026-02-24 17:40:05 +04:00
Manan17
aeb198f52d Fixing base model export issue for vlms 2026-02-24 01:34:11 +00:00
Roland Tannous
dbbcdb4f09 feat: clear unsloth_compiled_cache on startup, shutdown, and between model loads 2026-02-23 07:26:22 +00:00
Roland Tannous
c051e3d532 fix: load proper vision processor from base model when FastVisionModel returns raw tokenizer, add tokenize=False to vision chat template 2026-02-21 04:40:29 +00:00
Manan17
f6ebeb1d42 Mapping proper tokenizer for VLMs 2026-02-21 01:57:05 +00:00
Manan17
3fa9e773c2 fixed the vlm's text only errors 2026-02-20 22:23:26 +00:00
Manan17
fdeccec259 Fixing compare feature 2026-02-19 20:15:44 +00:00
Shine1i
0db7da96cc feat: support disabling top-k sampling with -1 and standardize normalization logic 
- Updated top-k parameter range to accept -1 in models and frontend.
- Added utility to normalize top-k for backend compatibility.
 2026-02-16 21:33:24 +01:00
Roland Tannous
909955767b feat: add min_p sampling parameter to /chat/completions generation pipeline 2026-02-16 06:33:17 +00:00
Shine1i
571959e383 feat: add cancelation support for chat generation and streaming tasks 2026-02-15 18:23:27 +01:00
Roland Tannous
be3934860f strip extra debug statements 2026-02-14 19:23:51 +00:00
Roland Tannous
3ff3def555 replace model unloading and peft loading mechanism for compare feature 2026-02-14 19:18:49 +00:00
Roland Tannous
e7ae901737 del model.peft_config instead of using model.delete_adapter 2026-02-14 17:32:15 +00:00