Commit graph

33 commits

Author SHA1 Message Date
Wasim Yousef Said
208862218d
feat(studio): training history persistence and past runs viewer (#4501)
* feat(db): add SQLite storage layer for training history

* feat(api): add training history endpoints and response models

* feat(training): integrate DB persistence into training event loop

* feat(ui): add training history views and card grid

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(studio): address review issues in training history persistence

- Strip hf_token/wandb_token from config before SQLite storage
- Add UUID suffix to job_id for collision resistance
- Use isfinite() for 0.0 metric handling throughout
- Respect _should_stop in error event finalization
- Run schema DDL once per process, not per connection
- Close connection on schema init failure
- Guard cleanup_orphaned_runs at startup
- Cap _metric_buffer at 500 entries
- Make FLUSH_THRESHOLD a class constant
- Map 'running' to 'training' phase in historical view
- Derive LR/GradNorm from history arrays in historical view
- Fix nested button with div[role=button] in history cards
- Guard String(value) against null/undefined in config popover
- Clear selectedHistoryRunId on auto tab switch

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(studio): address round-2 review findings across training backend and frontend

Backend (training.py):
- Move state mutation after proc.start() so a failed spawn does not wedge
  the backend with is_training=True
- Create DB run row eagerly after proc.start() so runs appear in history
  during model loading, not after first metric event
- Rewrite _flush_metrics_to_db() with snapshot-before-insert pattern to
  preserve metrics arriving during the write and retain buffer on failure
- Guard eval_loss with float() coercion and math.isfinite(), matching the
  existing grad_norm guard
- Increase pump thread join timeout from 3s to 8s to cover SQLite's
  default 5s lock timeout

Frontend (studio-page.tsx):
- Fix history navigation: check isTrainingRunning instead of
  showTrainingView in onSelectRun so completed runs are not misrouted
- Replace activeTab state + auto-switch useEffect with derived tab to
  eliminate react-hooks/set-state-in-effect lint violation

Frontend (historical-training-view.tsx):
- Add explicit "running" branch to message ternary so running runs no
  longer fall through to "Training errored"
- Derive loading from detail/error state and move cleanup to effect
  return to eliminate react-hooks/set-state-in-effect lint violation

Frontend (progress-section.tsx):
- Derive stopRequested from isTrainingRunning && stopRequestedLocal to
  eliminate react-hooks/set-state-in-effect lint violation and remove
  unused useEffect import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(studio): resolve 3 remaining bugs from round-2 review

1. Stuck on Current Run tab [12/20]: Only force "current-run" tab when
   isTrainingRunning is true, not when stale completed-run data exists.
   After training ends, users can freely navigate to Configure.

2. Incomplete metric sanitization [7/20]: Apply float() coercion and
   isfinite() guards to loss and learning_rate, matching the existing
   pattern used by grad_norm and eval_loss. Prevents TypeError from
   string values and NaN leaks into history arrays.

3. Stop button state leak across runs [10/20]: Add key={runtime.jobId}
   to ProgressSection so React remounts it when a new run starts,
   resetting stopRequestedLocal state.

* fix(studio): deduplicate loss/lr sanitization in training event handler

Reuse _safe_loss/_safe_lr from the progress update block instead of
re-sanitizing the same raw event values for metric history.

* fix(studio): restore loss > 0 guard to prevent eval steps injecting 0.0 into metric histories

Round-2/3 fixes relaxed the history append guard from `loss > 0` to
`loss is not None`, which let eval-only log events (where loss defaults
to 0.0) append fake zeros into loss_history and lr_history. Restore the
`loss > 0` check to match the worker's own has_train_loss gate. The
float() coercion and isfinite() sanitization from round-3 remain intact.

* fix(studio): resolve training history bugs — nullable loss/lr, tab nav, sparkline

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-03-25 00:58:55 -07:00
Manan Shah
164b5a5b06
[Feature] studio: user can upload eval dataset (#4307)
* user can upload eval dataset, removed bugs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolving merge conflicts

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolving gpt comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
2026-03-16 11:15:50 +04:00
Roland Tannous
47654cb91c Final cleanup 2026-03-12 18:28:04 +00:00
Roland Tannous
a2baf80511 Update license headers 2026-03-12 17:23:10 +00:00
Roland Tannous
6f77c63229 refactor: remove project_root passing, use self-resolved paths and ~/.unsloth/studio
- Workers now compute backend_path and venv_t5 locally via Path(__file__)
- Moved .venv_t5 to ~/.unsloth/studio/.venv_t5
- Added ensure_studio_directories() call on server startup
- Expanded CLI studio command into sub-app with setup subcommand
2026-03-11 20:32:18 +00:00
Roland Tannous
817f2e8dcc feat: integrate structlog, configure workers for prod logging, and migrate print statements 2026-03-11 12:33:16 +00:00
Roland Tannous
d6ae910edc fix: propagate is_embedding into worker subprocess config
start_training() cherry-picks kwargs into a config dict but was missing
is_embedding, so config.get("is_embedding", False) in worker.py always
returned False and embedding training never ran.
2026-03-10 19:05:47 +00:00
Manan17
fd7ca8bda8 distinguish cancel and stop for force terminate 2026-03-10 02:35:32 +00:00
Manan17
9be55f0c1b fixing cancel training 2026-03-10 02:20:56 +00:00
Roland Tannous
d882678fe4 Add AGPL-3.0 SPDX headers to all source files 2026-03-09 20:17:45 +00:00
Roland Tannous
2a11e79b8b fix: restore eval_enabled early signal for subprocess training 2026-03-09 15:35:49 +00:00
samit
86e94b5844 exposed trust_remote_code through the UI 2026-03-08 16:28:56 -07:00
Roland Tannous
a1778d6655 fix: replace is_dataset_multimodal with is_dataset_image/is_dataset_audio in training orchestrator 2026-03-08 19:40:00 +00:00
Roland Tannous
f3aeceeb24 fix: join prior pump thread before starting new training job 2026-03-07 08:37:03 +00:00
Roland Tannous
f8bd4303f7 feat: subprocess-based training for transformers version switching 2026-03-05 15:40:32 +00:00
Roland Tannous
a80188848d feat: add index range dataset slicing to studio training page
Add Start/End index inputs under Advanced in the dataset card,
allowing users to slice a dataset by row range before training.
Wired end-to-end: frontend store, API payload, backend Pydantic
model, and trainer dataset loading (inclusive on both ends).
2026-03-04 23:24:09 +00:00
Roland Tannous
91783c0fb2 Revert "Add index range dataset slicing to Studio training page" 2026-03-05 03:21:07 +04:00
Roland Tannous
11ebea6a4b feat: add index range dataset slicing to studio training page
Add Start/End index inputs under Advanced in the dataset card,
allowing users to slice a dataset by row range before training.
Wired end-to-end: frontend store, API payload, backend Pydantic
model, and trainer dataset loading (inclusive on both ends).
2026-03-04 21:48:40 +00:00
Roland Tannous
2be2933846 skip eval split and HF split detection when eval_steps is disabled 2026-02-24 09:26:54 +00:00
Leo Borcherding
cdeed53a97 fix: disable eval by default, set eval_steps to 0.0
- Changed default eval_steps from 0.01 to 0.0 across backend and frontend
- Fixed UI to allow eval_steps=0 (removed min=0.001 constraint)
- Added conditional eval logic with helpful console messages
- Updated tooltip to explain how to disable evaluation
- Tested: confirmed eval disabled by default with eval_steps=0.0
2026-02-23 13:07:47 -06:00
Shine1i
dc0cec772d feat: enhance training stop and reset flow with detailed checks 2026-02-17 23:32:22 +01:00
Shine1i
0be3e6f525 feat: integrate gradient norm tracking in training runtime and metrics
- Enhanced chart logic to filter and visualize finite gradient norm values.
2026-02-17 18:26:59 +01:00
Roland Tannous
ff0aec180a Merge branch 'nightly' into feature/eval-split-auto-detection 2026-02-17 01:11:30 +04:00
Roland Tannous
fa0ca59215 feat: auto-detect model+dataset compatibility to select VLM vs LLM training path 2026-02-16 19:18:49 +00:00
Roland Tannous
5df3a0b250 feat: add eval_enabled flag and format-first-then-split for eval dataset 2026-02-16 14:13:55 +00:00
Roland Tannous
37452d56cf feat: add eval split auto-detection, eval_steps hyperparam, and eval_loss chart integration 2026-02-16 13:38:54 +00:00
Roland Tannous
d0964652af feat: thread dataset subset/split params from API routes through to load_dataset calls 2026-02-16 03:56:22 +00:00
Manan17
6ccbc4edce Fixing stuck training processes 2026-02-15 05:38:06 +00:00
Manan17
97c6a09b84 feat: add cancel or save and stop training 2026-02-15 00:00:22 +00:00
Roland Tannous
4f0fad2156 fix: increase SSE progress timeout to 30min and allow step-0 updates 2026-02-14 05:47:22 +00:00
Roland Tannous
67edebfeb3 feat: wire custom_format_mapping through training pipeline to format_and_template_dataset 2026-02-13 21:07:36 +00:00
Roland Tannous
f52bddc23f refactor: remove gradio dependency from training backend 2026-02-13 09:25:49 +00:00
Roland Tannous
544d6944d1 root studio folder 2026-02-02 09:13:49 +00:00
Renamed from backend/core/training/training.py (Browse further)