ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
872 commits 391 branches 58 tags 304 MiB
Commit graph

90 commits

Author SHA1 Message Date
Roland Tannous
a1105d8ef3 wire trust_remote_code from YAML configs to frontend toggles 2026-03-09 10:15:15 +00:00
Manan17
80b704d7b7 Audio_VLM bug fix 2026-03-08 19:14:07 +00:00
Roland Tannous
1435dbaf59 merge nightly into audio branch (mock test) 2026-03-08 10:23:44 +00:00
Roland Tannous
e25705a211 fix: propagate PYTHONPATH to child subprocesses, revert tokenizer patching 2026-03-07 11:28:24 +00:00
Roland Tannous
76c78afb8f fix: patch TokenizersBackend by model name - Qwen3.5→Qwen2Tokenizer, GLM→PreTrainedTokenizer 2026-03-07 10:29:59 +00:00
Roland Tannous
d60cd2843f fix: patch Qwen3.5 broken tokenizer_class TokenizersBackend across all backends 2026-03-07 09:43:25 +00:00
Roland Tannous
bd60562145 fix: bump transformers 5.x pin from 5.1.0 to 5.2.0 for Qwen3.5 support 2026-03-07 09:10:09 +00:00
Roland Tannous
f7a3092cbd fix: correct project root depth in model_config.py vision check 2026-03-07 08:15:29 +00:00
Roland Tannous
4b7ad23b3a feat: broaden Qwen3.5 matching to cover entire family 2026-03-06 16:48:28 +00:00
Roland Tannous
ed1e63c814 feat: add Qwen3.5-35B-A3B and Qwen3-Next to transformers 5.x model list 2026-03-06 10:54:48 +00:00
Roland Tannous
c3bc19494f fix: pin huggingface_hub==1.3.0 in .venv_t5 (satisfies transformers 5.x) 2026-03-06 06:19:28 +00:00
Roland Tannous
6b32af0bdc feat: subprocess-based export, pin huggingface_hub==0.36.0 2026-03-06 06:03:09 +00:00
Roland Tannous
b5cfd0952c fix: use subprocess with transformers 5.x for vision detection 
Models like GLM-4.7-Flash have architectures (glm4_moe_lite) that
AutoConfig in the main process (transformers 4.57.x) can't recognize.
Instead of a raw config.json workaround, run the AutoConfig check in
a subprocess with .venv_t5/ activated — same pattern as training and
inference workers. This is more robust and consistent.
 2026-03-06 04:51:23 +00:00
Roland Tannous
e5c7a18f72 fix: handle unrecognized model architectures in vision detection 
AutoConfig.from_pretrained() fails for models needing transformers 5.x
(e.g. glm4_moe_lite) when running with 4.57.x. Add a raw config.json
fallback that bypasses AutoConfig's architecture registry — fetches
config.json directly from local path or HuggingFace Hub and checks
for vision indicators without needing the architecture to be registered.
 2026-03-06 04:46:51 +00:00
Roland Tannous
1167be2798 refactor: consolidate version switching to .venv_t5, remove .venv_overlay 
All version switching now uses .venv_t5/ (pre-installed by setup.sh).
The old .venv_overlay/ with runtime pip installs is removed.
ensure_transformers_version() (used only by export) now does a
lightweight sys.path swap instead of pip installing at runtime.
 2026-03-06 04:37:06 +00:00
Roland Tannous
9696bd557a fix: exclude bitsandbytes from module purge to prevent duplicate operator registration 2026-03-05 16:40:20 +00:00
Manan17
9909111982 resolved merge conflicts 2026-03-05 07:59:43 +00:00
Roland Tannous
81b4928e99 Merge nightly into feature/transformers-v5-support 2026-03-05 06:49:44 +00:00
Roland Tannous
c171573a8f fix: check for http(s) prefix instead of bare string type for URL detection 2026-03-05 06:10:10 +00:00
Roland Tannous
9ca45826d4 feat: parallel URL image probe with time estimate and progress reporting 
- Add 200-sample parallel probe using ThreadPoolExecutor + safe_num_proc
  to estimate download speed and failure rate before full conversion
- Abort with clear error if >=30% of probe images fail to download
- Show estimated download time in the training overlay modal
- Parallel batch conversion for URL-based datasets (vs sequential for local)
- Add warning field to /check-format response for URL-based image datasets
- Display URL warning in dataset preview dialog (amber banner)
- Thread progress_callback from trainer through format_and_template_dataset
  to convert_to_vlm_format for real-time status updates
 2026-03-04 23:40:38 +00:00
Roland Tannous
f59eaad212 feat: add tqdm progress bar to VLM conversion and download benchmark test 2026-03-04 23:29:43 +00:00
Roland Tannous
50885a7aa3 fix: add early probe to fail fast on datasets with too many broken image URLs 2026-03-04 23:29:43 +00:00
Roland Tannous
fdc23f4a43 fix: use fsspec for URL image downloads with per-sample error handling 2026-03-04 23:29:43 +00:00
Roland Tannous
929c3e9e1e fix: cast URL image columns to HF Image() type in VLM conversion 2026-03-04 23:29:43 +00:00
Roland Tannous
91783c0fb2 Revert "Add index range dataset slicing to Studio training page" 2026-03-05 03:21:07 +04:00
Roland Tannous
e4ec16296e feat: add tqdm progress bar to VLM conversion and download benchmark test 2026-03-04 13:30:27 +00:00
Manan17
a5825f8d44 dynamic detection of audio models and fixing autoencoder issues 2026-03-04 10:44:44 +00:00
Roland Tannous
5ee9479e37 fix: add early probe to fail fast on datasets with too many broken image URLs 2026-03-04 08:05:40 +00:00
Roland Tannous
722744cf04 fix: use fsspec for URL image downloads with per-sample error handling 2026-03-04 07:50:55 +00:00
Roland Tannous
34fb9ec973 fix: cast URL image columns to HF Image() type in VLM conversion 2026-03-04 06:42:37 +00:00
Manan17
f04c684d8a variable changes and some cleanup 2026-03-03 09:35:11 +00:00
Roland Tannous
87f2b2a9db Merge branch 'nightly' into feature/support-for-audio-models 2026-03-02 15:55:25 +04:00
Roland Tannous
7e021886c8 Force num_proc=1 on Windows to avoid slow spawn overhead 2026-03-01 13:05:10 +00:00
Roland Tannous
ff93c97024 fix: support mmproj for local vision GGUF models + fix Windows pipe deadlock 2026-03-01 12:58:38 +00:00
Manan17
c636fd5a42 code cleanup 2026-03-01 08:04:38 +00:00
Manan17
c48437848d revamping up the code and adding inference 2026-03-01 02:30:31 +00:00
Manan17
ab2ac39017 Changes with audio training 2026-03-01 02:27:45 +00:00
Manan17
ac27edde35 merging with nightly 2026-03-01 02:27:45 +00:00
Manan17
168957a87a Aggregating sharded models, showing fit/oom for quantizations 2026-02-27 08:23:15 +00:00
Manan17
2fea4cadd3 Passes metadata to get model size 2026-02-27 07:38:55 +00:00
Roland Tannous
ed18f9b9dd Flatten GGUF subdirs in export and fix metadata lookup in scanner 2026-02-26 11:35:04 +04:00
Roland Tannous
c21cf2ffcf Add GGUF tag for exported models in chat page selector 2026-02-25 19:01:47 +04:00
Roland Tannous
01082b84e5 Merge branch 'nightly' into feat/gguf-llama-cpp-inference 2026-02-25 16:06:03 +04:00
Manan17
6e8e70c987 fixing the chatml None error 2026-02-25 10:23:13 +00:00
Manan17
47fc79df6d My changes for dataset 2026-02-25 08:15:44 +00:00
Manan17
60912e45e6 adding custom mapping according to the chat templates 2026-02-25 07:56:30 +00:00
Roland Tannous
ef1cd3ac98 Use llama-server -hf mode, add GGUF variant selector, fix vision detection 
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
 2026-02-24 19:03:06 +04:00
Roland Tannous
4e88092452 Preflight llama-server check before downloading remote GGUF files 2026-02-24 18:02:43 +04:00
Roland Tannous
a900eb9ad7 Fix GGUF detection for HuggingFace repo IDs (not just local paths) 2026-02-24 17:49:09 +04:00
Roland Tannous
a40ebb1aab Add GGUF model inference via llama-server backend 2026-02-24 17:40:05 +04:00