Tier 1 check-format was picking images.zip over testmini.parquet,
causing wrong columns (image/label) and broken VLM mapping.
Also log first VLM conversion failure instead of swallowing silently.
- Pass 1: clearer definition of "conversational" vs non-conversational,
constrained dataset_type to specific enum values
- Pass 2: much more explicit worked examples with step-by-step reasoning,
added "skip" role for metadata columns, stronger reminder at end that
all-user is wrong
- Pass 3: returns raw text instead of JSON for cleaner system prompts,
removed system message to give model more freedom
The advisor now only assigns columns to user/assistant roles and
generates a system prompt. Templates (user_template, assistant_template)
are removed entirely — the LLM was frequently putting all columns in
user or copying actual data values into templates.
Column values are now used directly as message content, grouped and
concatenated by role. This is simpler, more robust, and prevents the
class of bugs where the advisor generates bad template content.
The LLM was putting all columns in user_template (e.g. summarization
dataset had both document AND summary as user input). Fixed by:
- Reframed system message: explicitly states user=INPUT, assistant=OUTPUT
- Added 4 concrete correct examples (summarization, NLI, translation, QA)
showing exactly how to split columns
- Added "NEVER put the output/target column in the user template" rule
- Added sanity check: if assistant_template has no column placeholders,
reject the result and fall back to simple classification
Pass 3 now sees the label mapping from Pass 2 (e.g. "0 = does not follow,
1 = follows, 2 = entailed") so the generated system prompt can explain
what each label value means. Also bumped to 2-4 sentences to give room
for the label descriptions.
Pass 1: Classify dataset type (unchanged)
Pass 2: Generate user/assistant templates + label mapping + column roles
(system_prompt removed from this pass to keep it focused)
Pass 3: Generate system prompt (only for non-conversational datasets)
- Dedicated pass with focused prompt that sees the templates from Pass 2
- Skipped entirely for conversational datasets
- Produces specific, task-relevant system prompts
- System prompt is now optional — LLM only generates one when the task
is ambiguous from the data alone (persona, domain, format constraints)
- Sanitize system_prompt extraction (handle literal "null" string)
- Show system prompt, user template, and assistant template in the
advisor notification banner so user can see exactly what was generated
- Templates displayed in monospace with labeled sections
The LLM was bad at scoring its own conversion quality — rejecting good
Pass 2 output (score 5/10 for a perfectly usable conversion). Instead:
- Remove Pass 3 entirely (saves ~0.4s and one inference call)
- Trust Pass 2 output and return it to the user
- Build notification from Pass 1 classification info instead
- User can always adjust mapping via dropdowns if they disagree
- Reject advisor result when Pass 3 scores < 6 or is_acceptable=false,
falls back to simple column classification instead of using bad output
- Improved Pass 2 prompt: explicit rules for label_mapping completeness,
{column_name} vs {column_name_name} for mapped labels, column_roles
must match which template uses them
- Build suggested_mapping from ALL template-referenced columns (not just
first match per role) — fixes hypothesis being dropped from SNLI mapping
- Guard against LLM returning literal string "null" for revised_system_prompt
- Always show AI Assist button when available, even when mapping looks complete
- Handle dict columns (e.g. squad answers) by extracting text instead
of raw repr()
- Handle list columns by joining or extracting single value
- Catch ValueError in .format() calls (stray { } in column data)
- Add missing json import to dataset_utils.py
Non-conversational HF datasets (e.g. stanfordnlp/snli) were naively mapped
column→role, producing poor training results. The AI Assist button now runs
a 3-pass advisor using Qwen 7B that:
1. Fetches the HF dataset card/README to understand the dataset purpose
2. Classifies the dataset type and determines if conversion is needed
3. Generates a system prompt, user/assistant templates with {column}
placeholders, and label mappings (e.g. 0→entailment)
4. Validates the conversion quality (score ≥7/10 required)
Architecture: advisor metadata flows as __-prefixed keys in
custom_format_mapping (e.g. __system_prompt, __user_template,
__assistant_template, __label_mapping). The existing _apply_user_mapping()
detects these keys and routes to template-based conversation construction.
No __ keys = existing simple mode (backwards compatible).
Backend: upgraded llm_assist.py (7B default, multi-pass advisor,
HF card fetching), extended API models, added _apply_template_mapping()
to dataset_utils.py.
Frontend: extended store with advisor state fields, wired AI Assist
to store templates/system prompt, inject __ metadata in training request,
show advisor notification banner in mapping card.
LlamaCppBackend.load_model() and precache_helper_gguf() only downloaded
the first matching GGUF file. For split models (e.g. 7B Q8_0 with 3
shards), llama-server needs all shards present. Now collects and
downloads all matching files.
Move LLM-assisted column mapping from silent /check-format automation
to an explicit "AI Assist" button in the dataset mapping dialog. This
makes the feature transparent and user-controlled.
- Remove llm_classify_columns() from check_dataset_format() (heuristic-only)
- Remove auto-save suggested_mapping from use-training-actions.ts
- Add POST /api/datasets/ai-assist-mapping endpoint (receives preview
samples from frontend, no dataset re-loading needed)
- Add AiAssistMappingRequest/Response models
- Add aiAssistMapping() frontend API function
- Add Sparkles AI Assist button to DatasetMappingCard with loading state
- Wire up handleAiAssist handler in dataset-preview-dialog.tsx
- Frontend auto-saves suggested_mapping into datasetManualMapping when
check-format returns requires_manual_mapping=false, so the mapping
flows to training via custom_format_mapping (no redundant AI calls)
- Backend returns meaningful warning when column detection fails
(LLM-generated or static fallback) for both text and VLM datasets
- /check-format endpoint merges check_dataset_format warnings with
existing URL-based image detection warnings
When multiple image columns are found, probes them (HEAD for URLs,
os.path.exists for paths) and picks the first that works.
Skips probing when top candidate is PIL/dict (score >= 75).
find_image_column now scores candidates by resolvability (PIL > dict > URL > path)
and has a Pass 2 value-based fallback for columns not matching image keywords.
Fixes phiyodr/coco2017 picking file_name (unresolvable) over coco_url (resolvable).
- Detect and convert ShareGPT/ChatML conversations with <image> placeholders
- Add file_name/filename as image column keywords
- Detect image paths and URLs by value (string ending in .jpg/.png/etc)
Datasets like VQAonline store image filenames (e.g. "img.png") without
the directory prefix. Build a basename→repo_path lookup using
list_repo_files, then resolve each file via hf_hub_download.
Tier 1 check-format was picking images.zip over testmini.parquet,
causing wrong columns (image/label) and broken VLM mapping.
Also log first VLM conversion failure instead of swallowing silently.
- Add 200-sample parallel probe using ThreadPoolExecutor + safe_num_proc
to estimate download speed and failure rate before full conversion
- Abort with clear error if >=30% of probe images fail to download
- Show estimated download time in the training overlay modal
- Parallel batch conversion for URL-based datasets (vs sequential for local)
- Add warning field to /check-format response for URL-based image datasets
- Display URL warning in dataset preview dialog (amber banner)
- Thread progress_callback from trainer through format_and_template_dataset
to convert_to_vlm_format for real-time status updates
