cc36e4b535
Two errors visible in training logs on Windows ROCm: 1. Server process bitsandbytes crash: "Configured ROCm binary not found at libbitsandbytes_rocm713.dll" The installed BNB wheel ships rocm72.dll (not rocm713.dll). The training worker already sets BNB_ROCM_VERSION=72 via DLL detection but the server process (main.py) imported bitsandbytes before that ran. Fix: add the same DLL-scan + BNB_ROCM_VERSION assignment to main.py inside the existing win32 guard, before any downstream import can pull in bitsandbytes. 2. torch.distributed import failure: "No module named 'torch._C._distributed_c10d'; torch._C is not a package" torch._C is a C extension on Windows ROCm — Python cannot do submodule imports from it, so torch.distributed fails to import before our attribute stubs could ever run. Fix: inject empty ModuleType stubs for _distributed_c10d, _distributed_autograd and _distributed_rpc into sys.modules inside the win32 guard in hardware.py BEFORE importing torch.distributed, so the import succeeds and our attribute stubs take effect. 9 new tests in TestServerStartupRocmFixes; total 212 passed, 2 skipped |
||
|---|---|---|
| .. | ||
| assets | ||
| auth | ||
| core | ||
| loggers | ||
| models | ||
| plugins | ||
| requirements | ||
| routes | ||
| state | ||
| storage | ||
| tests | ||
| utils | ||
| __init__.py | ||
| _platform_compat.py | ||
| colab.py | ||
| main.py | ||
| run.py | ||
| startup_banner.py | ||