unsloth/studio/backend/core/training
BardiaKoopah 14f679da8a
fix(studio): surface NaN loss honestly instead of laundering to last finite value (#6016)
When training produced a NaN or Inf loss event, the handler filtered the
value to None but never updated progress.loss — clients kept seeing the
last finite value as if everything were fine.

Now: on non-finite loss, clear progress.loss to None and log a one-shot
warning. Training continues (no phase=error, no _should_stop), matching
the expected behavior for a non-fatal numerical event.

Test: tests/test_training_nan_loss_handling.py with 6 cases covering
finite, NaN, +/-Inf, idempotency of the one-shot warning, and recovery
when a finite step follows a non-finite one.
2026-06-11 07:25:24 -07:00
..
__init__.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
resume.py Studio: Add checkpoint resume for stopped training runs (#5255) 2026-05-04 00:34:46 +04:00
trainer.py Studio: training survives a non-writable HF datasets cache (#6148) 2026-06-10 08:22:47 -07:00
training.py fix(studio): surface NaN loss honestly instead of laundering to last finite value (#6016) 2026-06-11 07:25:24 -07:00
worker.py Studio: require an installed ROCm DLL before forcing BNB_ROCM_VERSION; drop shadowing shutil imports in save.py (#6194) 2026-06-11 06:52:38 -07:00