ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
986 commits 391 branches 58 tags 304 MiB
Commit graph

96 commits

Author SHA1 Message Date
Roland Tannous
20264e973e debug: decode first sample after train_on_completions masking 2026-03-10 14:08:14 +00:00
Roland Tannous
9fd08a3f25 debug: fix dataset access - result is a dict, use dataset['dataset'] 2026-03-10 13:19:31 +00:00
Roland Tannous
a5d9f611e6 debug: improve sample preview with type info and traceback 2026-03-10 12:56:24 +00:00
Roland Tannous
10ba97c34c debug: switch to print() for subprocess visibility 2026-03-10 12:49:01 +00:00
Roland Tannous
19cbf575ac debug: add temporary log statements for dataset preview and VLM instruction 2026-03-10 12:35:55 +00:00
Roland Tannous
afad614bfa feat: add LLM-assisted dataset detection using ephemeral GGUF helper 
Uses Qwen2.5-3B-Instruct Q8_0 via LlamaCppBackend to complement
heuristic-based dataset detection when heuristics are uncertain.

- New llm_assist.py: VLM instruction generation, column classification,
  and user-friendly warning generation for dataset issues
- Pre-cache helper GGUF on FastAPI startup (background thread)
- Reorder training pipeline: dataset processing runs BEFORE model load
  to avoid VRAM contention (detect → dataset → model → train)
- Add pre_detect_and_load_tokenizer() for lightweight detection
- LLM warnings on VLM conversion failures (broken URLs, missing images)
- LLM column classification fallback when heuristics return unknown
- Graceful degradation: all paths unchanged when helper unavailable
 2026-03-10 09:20:45 +00:00
Roland Tannous
22eb0eea29 Revert "Merge pull request #347 from unslothai/feature/studio-storage-roots" 
This reverts commit e9c7b97d23, reversing
changes made to b75cc9b959.
 2026-03-10 01:52:47 +00:00
Shine1i
b08b606b21 feat(studio): studio storage roots path utilities 2026-03-09 23:48:31 +00:00
Roland Tannous
a0f03d3080 Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
Roland Tannous
65d3539bac Merge pull request #342 from unslothai/local-dataset 
dataset upload
 2026-03-09 21:22:23 +04:00
Roland Tannous
a8992279b6 fix: split dataset 80/20 when eval split matches train split 2026-03-09 16:36:44 +00:00
Roland Tannous
a7d78d16be fix: restore eval_enabled early signal for subprocess training 2026-03-09 15:35:49 +00:00
Roland Tannous
d85176ba1a fix: allow eval-only progress events through worker callback filter 2026-03-09 14:39:49 +00:00
Roland Tannous
07ba02d610 include all candidate files when scanning a directory, not just the first 2026-03-09 13:52:45 +00:00
Roland Tannous
dcedc4df56 merge nightly, resolve conflict in use-chat-model-runtime 2026-03-09 13:19:17 +00:00
Roland Tannous
f416b7aa3d training: restore YAML fallback for trust_remote_code (no UI toggle) 2026-03-09 13:10:24 +00:00
Roland Tannous
83b1ff05ef respect trust_remote_code toggle, return helpful error when required 2026-03-09 13:06:55 +00:00
Manan17
b430e23c0e dataset upload 2026-03-09 05:50:18 +00:00
Shine1i
4aa171b079 feat(recipe-studio, datasets): improve dataset handling and update metadata logic 2026-03-09 02:47:32 +01:00
Shine1i
d951e5aef0 merge nightly 2026-03-09 00:32:33 +01:00
samit
6aa50d353f exposed trust_remote_code through the UI 2026-03-08 16:28:56 -07:00
Roland Tannous
c38d24b01d fix: replace is_dataset_multimodal with is_dataset_image/is_dataset_audio in training orchestrator 2026-03-08 19:40:00 +00:00
Roland Tannous
7db2c90cc6 merge nightly into audio branch (mock test) 2026-03-08 10:23:44 +00:00
Manan17
2ce36df03c check fir gated repo 2026-03-07 21:32:50 +00:00
Roland Tannous
9543fa0d1c fix: scope dataloader_num_workers=0 to Windows + transformers 5.x only 2026-03-07 17:55:59 +00:00
Roland Tannous
dcebfe718a fix: prevent training hang on Windows by adding triton-windows support 2026-03-07 17:53:36 +00:00
Roland Tannous
c882a3d2f7 fix: propagate PYTHONPATH to child subprocesses, revert tokenizer patching 2026-03-07 11:28:24 +00:00
Roland Tannous
42bd976a2f fix: patch TokenizersBackend by model name - Qwen3.5→Qwen2Tokenizer, GLM→PreTrainedTokenizer 2026-03-07 10:29:59 +00:00
Roland Tannous
44e9b838ae fix: patch Qwen3.5 broken tokenizer_class TokenizersBackend across all backends 2026-03-07 09:43:25 +00:00
Roland Tannous
f101befca7 fix: bump transformers 5.x pin from 5.1.0 to 5.2.0 for Qwen3.5 support 2026-03-07 09:10:09 +00:00
Roland Tannous
2c4c598832 fix: fail fast if runtime pip install of transformers 5.x fails 2026-03-07 08:40:25 +00:00
Roland Tannous
082a6f876e fix: join prior pump thread before starting new training job 2026-03-07 08:37:03 +00:00
Roland Tannous
7e59440029 fix: pin huggingface_hub==1.3.0 in .venv_t5 (satisfies transformers 5.x) 2026-03-06 06:19:28 +00:00
Roland Tannous
661ac4be96 feat: subprocess-based export, pin huggingface_hub==0.36.0 2026-03-06 06:03:09 +00:00
Roland Tannous
f90af41c5f feat: subprocess-based training for transformers version switching 2026-03-05 15:40:32 +00:00
Shine1i
6fc829e3ef merge: nightly into feature/data-reciper-enchansments 2026-03-05 14:51:08 +01:00
Shine1i
f988202290 refactor(studio): add local data-recipe dataset selection + training wiring 2026-03-05 12:25:51 +01:00
Manan17
5d1a162ddd remove tracked OuteTTS embedded repo reference 2026-03-05 08:44:23 +00:00
Manan17
8203637d89 resolved merge conflicts 2026-03-05 07:59:43 +00:00
Manan17
2fa933640e fix SNAC training crash on variable-length sequences with DataCollatorForSeq2Seq 2026-03-05 07:04:53 +00:00
Roland Tannous
f57664e268 Merge nightly into feature/transformers-v5-support 2026-03-05 06:49:44 +00:00
Roland Tannous
e04b9d53d6 feat: parallel URL image probe with time estimate and progress reporting 
- Add 200-sample parallel probe using ThreadPoolExecutor + safe_num_proc
  to estimate download speed and failure rate before full conversion
- Abort with clear error if >=30% of probe images fail to download
- Show estimated download time in the training overlay modal
- Parallel batch conversion for URL-based datasets (vs sequential for local)
- Add warning field to /check-format response for URL-based image datasets
- Display URL warning in dataset preview dialog (amber banner)
- Thread progress_callback from trainer through format_and_template_dataset
  to convert_to_vlm_format for real-time status updates
 2026-03-04 23:40:38 +00:00
Roland Tannous
b376c54c21 fix: abort training pipeline on dataset conversion failure 2026-03-04 23:29:43 +00:00
Roland Tannous
64889cd5fc feat: add index range dataset slicing to studio training page 
Add Start/End index inputs under Advanced in the dataset card,
allowing users to slice a dataset by row range before training.
Wired end-to-end: frontend store, API payload, backend Pydantic
model, and trainer dataset loading (inclusive on both ends).
 2026-03-04 23:24:09 +00:00
Roland Tannous
9333f99dd3 Revert "Add index range dataset slicing to Studio training page" 2026-03-05 03:21:07 +04:00
Roland Tannous
02b17ec6d9 feat: add index range dataset slicing to studio training page 
Add Start/End index inputs under Advanced in the dataset card,
allowing users to slice a dataset by row range before training.
Wired end-to-end: frontend store, API payload, backend Pydantic
model, and trainer dataset loading (inclusive on both ends).
 2026-03-04 21:48:40 +00:00
Manan17
ee23628b0c dynamic detection of audio models and fixing autoencoder issues 2026-03-04 10:44:44 +00:00
Roland Tannous
c6d82a6fad fix: abort training pipeline on dataset conversion failure 2026-03-04 06:42:48 +00:00
Manan17
6fd1dd2c0a variable changes and some cleanup 2026-03-03 09:35:11 +00:00
Manan17
8cdeb006b6 code cleanup 2026-03-01 08:04:38 +00:00