acc604ffaf
torch.cuda.is_bf16_supported() defaults to counting pre-Ampere bf16 EMULATION as supported, so on a T4/V100/RTX 20xx the DiT-training bf16 gates all passed even though the trainer requires native Ampere-or-newer bf16: /diffusion/info advertised the DiT precision modes, /diffusion/start's preflight let the run through and freed resident GPU models, then the trainer child hit the real unsupported bf16 path. The inference device resolver already fixed this (issue #6658) by gating NVIDIA on capability major >= 8; the training path never got it. Add a shared native_bf16_supported() helper (NVIDIA cap major >= 8; ROCm keeps the trustworthy is_bf16_supported()) and use it in the three DiT bf16 sites -- train_precision_modes, bf16_unsupported_reason, and the trainer guard -- so a pre-Ampere card is offered nf4 only and never advertises/evicts-then-fails. Tests now exercise the emulation case (is_bf16_supported True but capability < 8). |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| diffusion_dit_trainer.py | ||
| diffusion_lora_trainer.py | ||
| diffusion_train_common.py | ||
| diffusion_training_service.py | ||
| resume.py | ||
| s3_dataset.py | ||
| trainer.py | ||
| training.py | ||
| worker.py | ||