diff --git a/tests/utils/ocr_eval.md b/tests/utils/ocr_eval.md index e3e0619f6b..97dbe6dd60 100644 --- a/tests/utils/ocr_eval.md +++ b/tests/utils/ocr_eval.md @@ -87,7 +87,7 @@ for config_name, params in configs.items(): evaluator.print_model_comparison() ``` -### Handwritting Recognition +### Handwriting Recognition ```python # Specialized evaluation for handwriting def evaluate_handwriting_models(models, handwriting_dataset): diff --git a/unsloth/kernels/moe/grouped_gemm/kernels/backward.py b/unsloth/kernels/moe/grouped_gemm/kernels/backward.py index 962e9997b4..ba35ee6993 100644 --- a/unsloth/kernels/moe/grouped_gemm/kernels/backward.py +++ b/unsloth/kernels/moe/grouped_gemm/kernels/backward.py @@ -30,7 +30,7 @@ dX backward kernel `fused_mul` notes: - In the forward pass, if we used the multiplication of topk weights (e.g., in the second grouped GEMM in fused MoE MLP), we need to make a few additional changes: 1) We load topk_weights in natural (token) order. Since we only enable `fuse_mul` when permuting on store (`permute_y`), we multiply grad_output by topk_weights before backpropagating - 2) We need to calculate the gradient of the topk_weights. This gets messy since we need do an additioanl elementwise multiplication in the GEMM main loop and then write out in unpermuted order. For now, we do not fuse this step but calculate as a simple + 2) We need to calculate the gradient of the topk_weights. This gets messy since we need do an additional elementwise multiplication in the GEMM main loop and then write out in unpermuted order. For now, we do not fuse this step but calculate as a simple Invalid combinations: - permute_y and use_tma_load: permuting y on store in forward -> load in permuted order in backward, therefore can't use TMA load (unless Blackwell which supports gather / scatter TMA)