ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix global dequantize buffer dtype mismatch across mixed-precision loads (#4026)

Browse source 
Fix global dequantize buffer dtype mismatch when loading multiple 4-bit models with different dtypes in the same process. Adds dtype check alongside existing None check for WEIGHT_BUFFER in both CUDA/HIP and XPU paths.
This commit is contained in:
金黄色葡萄球君君 2026-03-01 16:15:47 +08:00 • committed by GitHub
commit afd5b687aa
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
unsloth/kernels/utils.py
View file

@ -398,7 +398,7 @@ if DEVICE_TYPE == "xpu" and HAS_XPU_STREAM:
global ABSMAX_BUFFERS
WEIGHT_BUFFER = WEIGHT_BUFFERS[device_index]
ABSMAX_BUFFER = ABSMAX_BUFFERS[device_index]
if WEIGHT_BUFFER is None:
if WEIGHT_BUFFER is None or WEIGHT_BUFFER.dtype != dtype:
WEIGHT_BUFFERS[device_index] = WEIGHT_BUFFER = torch_empty(
size, dtype = dtype, device = device, requires_grad = False
)
@ -508,7 +508,7 @@ elif DEVICE_TYPE in ("cuda", "hip") and HAS_CUDA_STREAM:
global ABSMAX_BUFFERS
WEIGHT_BUFFER = WEIGHT_BUFFERS[device_index]
ABSMAX_BUFFER = ABSMAX_BUFFERS[device_index]
if WEIGHT_BUFFER is None:
if WEIGHT_BUFFER is None or WEIGHT_BUFFER.dtype != dtype:
WEIGHT_BUFFERS[device_index] = WEIGHT_BUFFER = torch_empty(
size, dtype = dtype, device = device, requires_grad = False
)