ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
unsloth/studio/backend/utils/datasets
History
Roland Tannous f7ca361c5c feat: add LLM-assisted dataset detection using ephemeral GGUF helper 
Uses Qwen2.5-3B-Instruct Q8_0 via LlamaCppBackend to complement
heuristic-based dataset detection when heuristics are uncertain.

- New llm_assist.py: VLM instruction generation, column classification,
  and user-friendly warning generation for dataset issues
- Pre-cache helper GGUF on FastAPI startup (background thread)
- Reorder training pipeline: dataset processing runs BEFORE model load
  to avoid VRAM contention (detect → dataset → model → train)
- Add pre_detect_and_load_tokenizer() for lightweight detection
- LLM warnings on VLM conversion failures (broken URLs, missing images)
- LLM column classification fallback when heuristics return unknown
- Graceful degradation: all paths unchanged when helper unavailable
2026-03-10 09:20:45 +00:00
..
__init__.py feat: add ShareGPT+image VLM format support and improve image column detection 2026-03-10 01:27:36 +00:00
chat_templates.py Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
data_collators.py Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
dataset_utils.py feat: add LLM-assisted dataset detection using ephemeral GGUF helper 2026-03-10 09:20:45 +00:00
format_conversion.py feat: add LLM-assisted dataset detection using ephemeral GGUF helper 2026-03-10 09:20:45 +00:00
format_detection.py fix: probe image column candidates when multiple exist 2026-03-10 01:38:33 +00:00
llm_assist.py feat: add LLM-assisted dataset detection using ephemeral GGUF helper 2026-03-10 09:20:45 +00:00
model_mappings.py Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
vlm_processing.py feat: add LLM-assisted dataset detection using ephemeral GGUF helper 2026-03-10 09:20:45 +00:00