ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
958 commits 391 branches 58 tags 304 MiB
Commit graph

90 commits

Author SHA1 Message Date
Roland Tannous
4f5c998097 wire trust_remote_code from YAML configs to frontend toggles 2026-03-09 10:15:15 +00:00
Manan17
111caf636f Audio_VLM bug fix 2026-03-08 19:14:07 +00:00
Roland Tannous
7db2c90cc6 merge nightly into audio branch (mock test) 2026-03-08 10:23:44 +00:00
Roland Tannous
c882a3d2f7 fix: propagate PYTHONPATH to child subprocesses, revert tokenizer patching 2026-03-07 11:28:24 +00:00
Roland Tannous
42bd976a2f fix: patch TokenizersBackend by model name - Qwen3.5→Qwen2Tokenizer, GLM→PreTrainedTokenizer 2026-03-07 10:29:59 +00:00
Roland Tannous
44e9b838ae fix: patch Qwen3.5 broken tokenizer_class TokenizersBackend across all backends 2026-03-07 09:43:25 +00:00
Roland Tannous
f101befca7 fix: bump transformers 5.x pin from 5.1.0 to 5.2.0 for Qwen3.5 support 2026-03-07 09:10:09 +00:00
Roland Tannous
48456070a7 fix: correct project root depth in model_config.py vision check 2026-03-07 08:15:29 +00:00
Roland Tannous
2cfcaa8b61 feat: broaden Qwen3.5 matching to cover entire family 2026-03-06 16:48:28 +00:00
Roland Tannous
c31e8a6ed7 feat: add Qwen3.5-35B-A3B and Qwen3-Next to transformers 5.x model list 2026-03-06 10:54:48 +00:00
Roland Tannous
7e59440029 fix: pin huggingface_hub==1.3.0 in .venv_t5 (satisfies transformers 5.x) 2026-03-06 06:19:28 +00:00
Roland Tannous
661ac4be96 feat: subprocess-based export, pin huggingface_hub==0.36.0 2026-03-06 06:03:09 +00:00
Roland Tannous
f15970c02a fix: use subprocess with transformers 5.x for vision detection 
Models like GLM-4.7-Flash have architectures (glm4_moe_lite) that
AutoConfig in the main process (transformers 4.57.x) can't recognize.
Instead of a raw config.json workaround, run the AutoConfig check in
a subprocess with .venv_t5/ activated — same pattern as training and
inference workers. This is more robust and consistent.
 2026-03-06 04:51:23 +00:00
Roland Tannous
67121ce427 fix: handle unrecognized model architectures in vision detection 
AutoConfig.from_pretrained() fails for models needing transformers 5.x
(e.g. glm4_moe_lite) when running with 4.57.x. Add a raw config.json
fallback that bypasses AutoConfig's architecture registry — fetches
config.json directly from local path or HuggingFace Hub and checks
for vision indicators without needing the architecture to be registered.
 2026-03-06 04:46:51 +00:00
Roland Tannous
d55e9abcca refactor: consolidate version switching to .venv_t5, remove .venv_overlay 
All version switching now uses .venv_t5/ (pre-installed by setup.sh).
The old .venv_overlay/ with runtime pip installs is removed.
ensure_transformers_version() (used only by export) now does a
lightweight sys.path swap instead of pip installing at runtime.
 2026-03-06 04:37:06 +00:00
Roland Tannous
794b8fe866 fix: exclude bitsandbytes from module purge to prevent duplicate operator registration 2026-03-05 16:40:20 +00:00
Manan17
8203637d89 resolved merge conflicts 2026-03-05 07:59:43 +00:00
Roland Tannous
f57664e268 Merge nightly into feature/transformers-v5-support 2026-03-05 06:49:44 +00:00
Roland Tannous
a1706c894f fix: check for http(s) prefix instead of bare string type for URL detection 2026-03-05 06:10:10 +00:00
Roland Tannous
e04b9d53d6 feat: parallel URL image probe with time estimate and progress reporting 
- Add 200-sample parallel probe using ThreadPoolExecutor + safe_num_proc
  to estimate download speed and failure rate before full conversion
- Abort with clear error if >=30% of probe images fail to download
- Show estimated download time in the training overlay modal
- Parallel batch conversion for URL-based datasets (vs sequential for local)
- Add warning field to /check-format response for URL-based image datasets
- Display URL warning in dataset preview dialog (amber banner)
- Thread progress_callback from trainer through format_and_template_dataset
  to convert_to_vlm_format for real-time status updates
 2026-03-04 23:40:38 +00:00
Roland Tannous
1f03754c95 feat: add tqdm progress bar to VLM conversion and download benchmark test 2026-03-04 23:29:43 +00:00
Roland Tannous
63f723cc36 fix: add early probe to fail fast on datasets with too many broken image URLs 2026-03-04 23:29:43 +00:00
Roland Tannous
9487d17b94 fix: use fsspec for URL image downloads with per-sample error handling 2026-03-04 23:29:43 +00:00
Roland Tannous
bc244aeb23 fix: cast URL image columns to HF Image() type in VLM conversion 2026-03-04 23:29:43 +00:00
Roland Tannous
9333f99dd3 Revert "Add index range dataset slicing to Studio training page" 2026-03-05 03:21:07 +04:00
Roland Tannous
cc11f066b1 feat: add tqdm progress bar to VLM conversion and download benchmark test 2026-03-04 13:30:27 +00:00
Manan17
ee23628b0c dynamic detection of audio models and fixing autoencoder issues 2026-03-04 10:44:44 +00:00
Roland Tannous
7804a4db2e fix: add early probe to fail fast on datasets with too many broken image URLs 2026-03-04 08:05:40 +00:00
Roland Tannous
9cbd3d44a7 fix: use fsspec for URL image downloads with per-sample error handling 2026-03-04 07:50:55 +00:00
Roland Tannous
3c4bf80cc2 fix: cast URL image columns to HF Image() type in VLM conversion 2026-03-04 06:42:37 +00:00
Manan17
6fd1dd2c0a variable changes and some cleanup 2026-03-03 09:35:11 +00:00
Roland Tannous
da00f5ed1d Merge branch 'nightly' into feature/support-for-audio-models 2026-03-02 15:55:25 +04:00
Roland Tannous
c297d7aa84 Force num_proc=1 on Windows to avoid slow spawn overhead 2026-03-01 13:05:10 +00:00
Roland Tannous
986bef4f99 fix: support mmproj for local vision GGUF models + fix Windows pipe deadlock 2026-03-01 12:58:38 +00:00
Manan17
8cdeb006b6 code cleanup 2026-03-01 08:04:38 +00:00
Manan17
9e89f31bc7 revamping up the code and adding inference 2026-03-01 02:30:31 +00:00
Manan17
90332924de Changes with audio training 2026-03-01 02:27:45 +00:00
Manan17
2c5621dd8c merging with nightly 2026-03-01 02:27:45 +00:00
Manan17
b4311cca82 Aggregating sharded models, showing fit/oom for quantizations 2026-02-27 08:23:15 +00:00
Manan17
4bd5213c05 Passes metadata to get model size 2026-02-27 07:38:55 +00:00
Roland Tannous
c06adc3878 Flatten GGUF subdirs in export and fix metadata lookup in scanner 2026-02-26 11:35:04 +04:00
Roland Tannous
b91b979bf8 Add GGUF tag for exported models in chat page selector 2026-02-25 19:01:47 +04:00
Roland Tannous
efaa0bacfb Merge branch 'nightly' into feat/gguf-llama-cpp-inference 2026-02-25 16:06:03 +04:00
Manan17
5ce88f9aa1 fixing the chatml None error 2026-02-25 10:23:13 +00:00
Manan17
202dd082b7 My changes for dataset 2026-02-25 08:15:44 +00:00
Manan17
611febd2e3 adding custom mapping according to the chat templates 2026-02-25 07:56:30 +00:00
Roland Tannous
3ee4f1359a Use llama-server -hf mode, add GGUF variant selector, fix vision detection 
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
 2026-02-24 19:03:06 +04:00
Roland Tannous
4a82e704aa Preflight llama-server check before downloading remote GGUF files 2026-02-24 18:02:43 +04:00
Roland Tannous
c635d4f49c Fix GGUF detection for HuggingFace repo IDs (not just local paths) 2026-02-24 17:49:09 +04:00
Roland Tannous
2f985ccbb5 Add GGUF model inference via llama-server backend 2026-02-24 17:40:05 +04:00