cc36e4b535
Two errors visible in training logs on Windows ROCm: 1. Server process bitsandbytes crash: "Configured ROCm binary not found at libbitsandbytes_rocm713.dll" The installed BNB wheel ships rocm72.dll (not rocm713.dll). The training worker already sets BNB_ROCM_VERSION=72 via DLL detection but the server process (main.py) imported bitsandbytes before that ran. Fix: add the same DLL-scan + BNB_ROCM_VERSION assignment to main.py inside the existing win32 guard, before any downstream import can pull in bitsandbytes. 2. torch.distributed import failure: "No module named 'torch._C._distributed_c10d'; torch._C is not a package" torch._C is a C extension on Windows ROCm — Python cannot do submodule imports from it, so torch.distributed fails to import before our attribute stubs could ever run. Fix: inject empty ModuleType stubs for _distributed_c10d, _distributed_autograd and _distributed_rpc into sys.modules inside the win32 guard in hardware.py BEFORE importing torch.distributed, so the import succeeds and our attribute stubs take effect. 9 new tests in TestServerStartupRocmFixes; total 212 passed, 2 skipped |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| amd.py | ||
| hardware.py | ||
| nvidia.py | ||
| VRAM_ESTIMATION.md | ||
| vram_estimation.py | ||