9780cdcca1* Fix FlashAttention fp32 crash with DoRA (use_dora=True) DoRA upcasts lora_magnitude_vector to fp32 for the optimizer, which promotes the q/k/v_proj output to fp32. FlashAttention only accepts fp16/bf16, so the fp32 q/k/v raised 'FlashAttention only support fp16 and bf16 data type'. Downcast q/k/v to the compute dtype before the flash kernels. Fixes #1013 * Apply kwarg-spacing format hook to DoRA dtype test (pre-commit) * DoRA+FA2: downcast any fp32 among Q/K/V and clamp to a flash-supported dtype * Tighten code comments (no logic change) --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| attention_dispatch.py | ||
| hf_hub.py | ||
| packing.py | ||