2de17c0a96* feat: add checkpoint resume for stopped training runs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix:add resume checkpoint helpers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: use checkpoint parent as resume output dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: save optimizer and scheduler state on stop-and-save Use Trainer._save_checkpoint instead of save_state so resume restores optimizer momentum and LR-schedule position via the checkpoint-NNN/ subdir written by HF's official path. * fix: clean up resume training history and startup progress * fix: preserve resume output dirs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: tighten resume run lookup * fix: remove stale output-dir lookup * fix: preserve startup download progress --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| resume.py | ||
| trainer.py | ||
| training.py | ||
| worker.py | ||