Prevent negative Train Split Start/End values in the dataset advanced UI and sanitize payload mapping so negative slice values are never sent to the backend.
Made-with: Cursor
Instead of downloading the full dataset and then slicing, use
streaming mode to only fetch the rows needed (up to slice_end + 1)
when a manual dataset slice is configured.
startswith(prefix) could match unrelated split variants whose names
extend the selected file's prefix (e.g. model-Q8_0-v2-00001-of-...).
Now builds an exact regex from the chosen file's base prefix and shard
total so only true siblings are downloaded.
Substring matching (e.g. "Q8_0" in filename) could match superset
variants like "IQ8_0", causing wrong quantizations to be downloaded.
Now uses word-boundary regex for variant matching and discovers split
shards by shared filename prefix rather than treating all variant
matches as shards.
start_training() cherry-picks kwargs into a config dict but was missing
is_embedding, so config.get("is_embedding", False) in worker.py always
returned False and embedding training never ran.
LlamaCppBackend.load_model() only downloaded the first matching GGUF
file. For split models (e.g. 7B Q8_0 with 3 shards), llama-server
needs all shards present. Now collects and downloads all matching files.
Separate pure-audio from audio-VLM logic in runDatasetCheck so pure
audio models are always forced to trainOnCompletions=false regardless
of dataset type, while audio VLMs (gemma3n) only uncheck when the
dataset is audio.
Clear stale isAudioModel in the fallback path when getModelConfig
fails, preventing a previously-selected audio model's flag from
leaking into the next model selection.
Add end-to-end embedding/sentence-transformer training pipeline using
FastSentenceTransformer, SentenceTransformerTrainer, and
MultipleNegativesRankingLoss with BatchSamplers.NO_DUPLICATES.
Backend:
- Add is_embedding_model() detection via HF tags + pipeline_tag
- Add /check-embedding/ API route and EmbeddingCheckResponse
- Extend derive_model_type() to return "embeddings"
- Add _run_embedding_training() in worker.py with progress callbacks,
stop handling, LoRA (task_type=FEATURE_EXTRACTION), and model saving
- Add is_embedding field to TrainingStartRequest and ModelDetails
- Add YAML configs for 5 models: all-MiniLM-L6-v2, bge-m3,
embeddinggemma-300m, gte-modernbert-base, Qwen3-Embedding-0.6B
Frontend:
- Wire isEmbeddingModel flag through store, API types, and mappers
- Force packing=false, train_on_completions=false, warmup_ratio=0.03
- Hide packing and train_on_completions checkboxes for embedding models
- Auto-set modelType to "embeddings" from backend model_type response
Pure audio models (orpheus, sparktts, whisper, sesame-csm) now
always have trainOnCompletions auto-unchecked when selected.
Gemma3n (audio_vlm) only unchecks when the dataset is audio.
- Add is_audio to frontend ModelConfigResponse (backend already returns it)
- Add isAudioModel state to training config store
- Auto-set trainOnCompletions=false for pure audio models on model load
- Auto-set trainOnCompletions=false for audio VLMs when dataset is audio
- Respect manual user override via existing _trainOnCompletionsManuallySet flag
Derive a single model_type string ("text" | "vision" | "audio" | "embeddings")
from existing is_vision and audio_type detection, so the frontend doesn't have
to infer modality from scattered boolean flags.
When multiple image columns are found, probes them (HEAD for URLs,
os.path.exists for paths) and picks the first that works.
Skips probing when top candidate is PIL/dict (score >= 75).
find_image_column now scores candidates by resolvability (PIL > dict > URL > path)
and has a Pass 2 value-based fallback for columns not matching image keywords.
Fixes phiyodr/coco2017 picking file_name (unresolvable) over coco_url (resolvable).
- Detect and convert ShareGPT/ChatML conversations with <image> placeholders
- Add file_name/filename as image column keywords
- Detect image paths and URLs by value (string ending in .jpg/.png/etc)
