ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
4,670 commits 391 branches 58 tags 304 MiB
Commit graph

87 commits

Author SHA1 Message Date
Manan Shah
164b5a5b06
 		[Feature] studio: user can upload eval dataset (#4307) 
* user can upload eval dataset, removed bugs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolving merge conflicts

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolving gpt comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
 2026-03-16 11:15:50 +04:00
Roland Tannous
47654cb91c Final cleanup 2026-03-12 18:28:04 +00:00
Roland Tannous
a2baf80511 Update license headers 2026-03-12 17:23:10 +00:00
Shine1i
904e440513 feat(studio): studio storage roots path utilities 2026-03-11 20:19:52 +00:00
Roland Tannous
9dac1bedf9 Merge remote-tracking branch 'origin/nightly' into feature/llm-assist-detection 2026-03-11 16:23:09 +00:00
Roland Tannous
817f2e8dcc feat: integrate structlog, configure workers for prod logging, and migrate print statements 2026-03-11 12:33:16 +00:00
Manan17
5ca623a166 fixing gguf export for gemma with text 2026-03-11 00:58:22 +00:00
Roland Tannous
21ef22a9ff fix: skip streaming when dataset_slice_start > dataset_slice_end 
Prevents training on the wrong row range when start exceeds end by
falling back to full download where existing clamping handles it.
 2026-03-10 20:21:34 +00:00
Roland Tannous
226f251589 fix: guard against negative dataset_slice_end before streaming 
Fall back to full download when dataset_slice_end is negative,
avoiding an empty stream.take(0) that would produce a broken dataset.
 2026-03-10 20:12:42 +00:00
Roland Tannous
970a029108 fix: stream HF dataset when manual slice is specified 
Instead of downloading the full dataset and then slicing, use
streaming mode to only fetch the rows needed (up to slice_end + 1)
when a manual dataset slice is configured.
 2026-03-10 19:50:53 +00:00
Roland Tannous
7f1fd28acd debug: decode first sample after train_on_completions masking 2026-03-10 14:08:14 +00:00
Roland Tannous
f7ca361c5c feat: add LLM-assisted dataset detection using ephemeral GGUF helper 
Uses Qwen2.5-3B-Instruct Q8_0 via LlamaCppBackend to complement
heuristic-based dataset detection when heuristics are uncertain.

- New llm_assist.py: VLM instruction generation, column classification,
  and user-friendly warning generation for dataset issues
- Pre-cache helper GGUF on FastAPI startup (background thread)
- Reorder training pipeline: dataset processing runs BEFORE model load
  to avoid VRAM contention (detect → dataset → model → train)
- Add pre_detect_and_load_tokenizer() for lightweight detection
- LLM warnings on VLM conversion failures (broken URLs, missing images)
- LLM column classification fallback when heuristics return unknown
- Graceful degradation: all paths unchanged when helper unavailable
 2026-03-10 09:20:45 +00:00
Roland Tannous
daa50d0756 Revert "Merge pull request #347 from unslothai/feature/studio-storage-roots" 
This reverts commit 6b43e33ff1, reversing
changes made to 9edadaf21f.
 2026-03-10 01:52:47 +00:00
Shine1i
5301514775 feat(studio): studio storage roots path utilities 2026-03-09 23:48:31 +00:00
Roland Tannous
d882678fe4 Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
Roland Tannous
b6811bc5c4 Merge pull request #342 from unslothai/local-dataset 
dataset upload
 2026-03-09 21:22:23 +04:00
Roland Tannous
41351e1566 fix: split dataset 80/20 when eval split matches train split 2026-03-09 16:36:44 +00:00
Roland Tannous
56412f2362 include all candidate files when scanning a directory, not just the first 2026-03-09 13:52:45 +00:00
Roland Tannous
91dd7fc762 merge nightly, resolve conflict in use-chat-model-runtime 2026-03-09 13:19:17 +00:00
Manan17
a49638c504 dataset upload 2026-03-09 05:50:18 +00:00
Shine1i
3b1663b1e9 feat(recipe-studio, datasets): improve dataset handling and update metadata logic 2026-03-09 02:47:32 +01:00
Shine1i
a2dde15367 merge nightly 2026-03-09 00:32:33 +01:00
samit
86e94b5844 exposed trust_remote_code through the UI 2026-03-08 16:28:56 -07:00
Roland Tannous
1435dbaf59 merge nightly into audio branch (mock test) 2026-03-08 10:23:44 +00:00
Manan17
6487f81113 check fir gated repo 2026-03-07 21:32:50 +00:00
Roland Tannous
8454e6dd2b fix: scope dataloader_num_workers=0 to Windows + transformers 5.x only 2026-03-07 17:55:59 +00:00
Roland Tannous
ef9184c731 fix: prevent training hang on Windows by adding triton-windows support 2026-03-07 17:53:36 +00:00
Roland Tannous
e25705a211 fix: propagate PYTHONPATH to child subprocesses, revert tokenizer patching 2026-03-07 11:28:24 +00:00
Roland Tannous
76c78afb8f fix: patch TokenizersBackend by model name - Qwen3.5→Qwen2Tokenizer, GLM→PreTrainedTokenizer 2026-03-07 10:29:59 +00:00
Roland Tannous
d60cd2843f fix: patch Qwen3.5 broken tokenizer_class TokenizersBackend across all backends 2026-03-07 09:43:25 +00:00
Shine1i
b277308b7e merge: nightly into feature/data-reciper-enchansments 2026-03-05 14:51:08 +01:00
Shine1i
e30fc87187 refactor(studio): add local data-recipe dataset selection + training wiring 2026-03-05 12:25:51 +01:00
Manan17
9909111982 resolved merge conflicts 2026-03-05 07:59:43 +00:00
Manan17
c723f8d4da fix SNAC training crash on variable-length sequences with DataCollatorForSeq2Seq 2026-03-05 07:04:53 +00:00
Roland Tannous
81b4928e99 Merge nightly into feature/transformers-v5-support 2026-03-05 06:49:44 +00:00
Roland Tannous
9ca45826d4 feat: parallel URL image probe with time estimate and progress reporting 
- Add 200-sample parallel probe using ThreadPoolExecutor + safe_num_proc
  to estimate download speed and failure rate before full conversion
- Abort with clear error if >=30% of probe images fail to download
- Show estimated download time in the training overlay modal
- Parallel batch conversion for URL-based datasets (vs sequential for local)
- Add warning field to /check-format response for URL-based image datasets
- Display URL warning in dataset preview dialog (amber banner)
- Thread progress_callback from trainer through format_and_template_dataset
  to convert_to_vlm_format for real-time status updates
 2026-03-04 23:40:38 +00:00
Roland Tannous
2b704221f7 fix: abort training pipeline on dataset conversion failure 2026-03-04 23:29:43 +00:00
Roland Tannous
a80188848d feat: add index range dataset slicing to studio training page 
Add Start/End index inputs under Advanced in the dataset card,
allowing users to slice a dataset by row range before training.
Wired end-to-end: frontend store, API payload, backend Pydantic
model, and trainer dataset loading (inclusive on both ends).
 2026-03-04 23:24:09 +00:00
Roland Tannous
91783c0fb2 Revert "Add index range dataset slicing to Studio training page" 2026-03-05 03:21:07 +04:00
Roland Tannous
11ebea6a4b feat: add index range dataset slicing to studio training page 
Add Start/End index inputs under Advanced in the dataset card,
allowing users to slice a dataset by row range before training.
Wired end-to-end: frontend store, API payload, backend Pydantic
model, and trainer dataset loading (inclusive on both ends).
 2026-03-04 21:48:40 +00:00
Manan17
a5825f8d44 dynamic detection of audio models and fixing autoencoder issues 2026-03-04 10:44:44 +00:00
Roland Tannous
645d7d357a fix: abort training pipeline on dataset conversion failure 2026-03-04 06:42:48 +00:00
Manan17
f04c684d8a variable changes and some cleanup 2026-03-03 09:35:11 +00:00
Manan17
c636fd5a42 code cleanup 2026-03-01 08:04:38 +00:00
Manan17
c48437848d revamping up the code and adding inference 2026-03-01 02:30:31 +00:00
Manan17
ab2ac39017 Changes with audio training 2026-03-01 02:27:45 +00:00
Manan17
ac27edde35 merging with nightly 2026-03-01 02:27:45 +00:00
Roland Tannous
01082b84e5 Merge branch 'nightly' into feat/gguf-llama-cpp-inference 2026-02-25 16:06:03 +04:00
Roland Tannous
6f0b7bc38a fix: use raw github URL for vision.py patch + add VLM processor diagnostic logging 2026-02-25 10:29:05 +00:00
Roland Tannous
2149bc74ee Merge pull request #232 from unslothai/fix/disable-eval-by-default 
# fix/disable eval by default
 2026-02-24 13:35:11 +04:00