Commit graph

17 commits

Author SHA1 Message Date
Roland Tannous
0a81ee38e6 fix: fall back to auto-detection when user VLM mapping fails
Instead of erroring out when custom_format_mapping fails conversion,
clear it and let auto-detection try. Handles stale cached mappings.
2026-03-10 01:42:25 +00:00
Roland Tannous
095a051ee0 feat: add ShareGPT+image VLM format support and improve image column detection
- Detect and convert ShareGPT/ChatML conversations with <image> placeholders
- Add file_name/filename as image column keywords
- Detect image paths and URLs by value (string ending in .jpg/.png/etc)
2026-03-10 01:27:36 +00:00
Roland Tannous
a0f03d3080 Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
Manan17
8203637d89 resolved merge conflicts 2026-03-05 07:59:43 +00:00
Roland Tannous
e04b9d53d6 feat: parallel URL image probe with time estimate and progress reporting
- Add 200-sample parallel probe using ThreadPoolExecutor + safe_num_proc
  to estimate download speed and failure rate before full conversion
- Abort with clear error if >=30% of probe images fail to download
- Show estimated download time in the training overlay modal
- Parallel batch conversion for URL-based datasets (vs sequential for local)
- Add warning field to /check-format response for URL-based image datasets
- Display URL warning in dataset preview dialog (amber banner)
- Thread progress_callback from trainer through format_and_template_dataset
  to convert_to_vlm_format for real-time status updates
2026-03-04 23:40:38 +00:00
Manan17
6fd1dd2c0a variable changes and some cleanup 2026-03-03 09:35:11 +00:00
Manan17
90332924de Changes with audio training 2026-03-01 02:27:45 +00:00
Manan17
5ce88f9aa1 fixing the chatml None error 2026-02-25 10:23:13 +00:00
Manan17
202dd082b7 My changes for dataset 2026-02-25 08:15:44 +00:00
Manan17
611febd2e3 adding custom mapping according to the chat templates 2026-02-25 07:56:30 +00:00
Roland Tannous
4e4fc367b6 fix: auto-detect multimodal datasets in /check-format without requiring is_vlm flag 2026-02-13 17:29:39 +00:00
Roland Tannous
528d2e27f0 authentication refactor - added setup token and token refresh mechanism 2026-02-11 12:09:47 +00:00
Roland Tannous
996b16f9ee Add datasets check-format endpoint 2026-02-03 20:42:25 +00:00
Roland Tannous
4d89dd302c fix custom_format_mapping flow for manual column mapping 2026-02-03 18:42:07 +00:00
Roland Tannous
fa3724b82c Add Flag for Dataset Detection 2026-02-03 18:21:05 +00:00
Roland Tannous
842b2b3186 remove duplicates from dataset_utils.py 2026-02-03 18:03:01 +00:00
Roland Tannous
75d8dcc824 root studio folder 2026-02-02 09:13:49 +00:00
Renamed from backend/utils/datasets/dataset_utils.py (Browse further)