Fix Typos in Documentation and Comments (#2721)
* Update ocr_eval.md * Update backward.py
This commit is contained in:
parent
440bbf5b52
commit
c6e0366e0d
2 changed files with 2 additions and 2 deletions
|
|
@ -87,7 +87,7 @@ for config_name, params in configs.items():
|
|||
evaluator.print_model_comparison()
|
||||
```
|
||||
|
||||
### Handwritting Recognition
|
||||
### Handwriting Recognition
|
||||
```python
|
||||
# Specialized evaluation for handwriting
|
||||
def evaluate_handwriting_models(models, handwriting_dataset):
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ dX backward kernel
|
|||
`fused_mul` notes:
|
||||
- In the forward pass, if we used the multiplication of topk weights (e.g., in the second grouped GEMM in fused MoE MLP), we need to make a few additional changes:
|
||||
1) We load topk_weights in natural (token) order. Since we only enable `fuse_mul` when permuting on store (`permute_y`), we multiply grad_output by topk_weights before backpropagating
|
||||
2) We need to calculate the gradient of the topk_weights. This gets messy since we need do an additioanl elementwise multiplication in the GEMM main loop and then write out in unpermuted order. For now, we do not fuse this step but calculate as a simple
|
||||
2) We need to calculate the gradient of the topk_weights. This gets messy since we need do an additional elementwise multiplication in the GEMM main loop and then write out in unpermuted order. For now, we do not fuse this step but calculate as a simple
|
||||
|
||||
Invalid combinations:
|
||||
- permute_y and use_tma_load: permuting y on store in forward -> load in permuted order in backward, therefore can't use TMA load (unless Blackwell which supports gather / scatter TMA)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue