ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
unsloth/scripts/benchmarks/results/stats
History
Daniel Han 69723ee31c FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 
Summary of sweep (all with CUDA graph capture):

| Batch | flex tps | vLLM tps | flex / vLLM | flex mem |
|------:|---------:|---------:|------------:|---------:|
|    8  |     680  |   1900   |    35.8 %   |  44 GB   |
|   16  |    1626  |   3698   |    44.0 %   |  44 GB   |
|   32  |    3134  |   6318   |    49.6 %   |  44 GB   |
|   64  |    5474  |  10459   |  **52.3 %** |  44 GB   |
|  128  |    5565  |  14996   |    37.1 %   |  81 GB   |
|  256  |    5812  |  21170   |    27.5 %   | 154 GB   |

Canonical GRPO (batch 64 + LoRA rank 32):
- vLLM: 7775 tok/s / 156 GB
- flex: **5616 tok/s / 44 GB** = **72% of vLLM at 3.5x less memory**

Up from 9 % (transformers CB) at the start of this work.

Best FlexKernelOptions after sweep:
  decode: PRESCALE_QK, USE_TMA, BLOCKS_ARE_CONTIGUOUS, num_warps=8, num_stages=3
  prefill: FORCE_USE_FLEX_ATTENTION, PRESCALE_QK, USE_TMA

Biggest single win: `num_warps=8` (+28% at batch 64). Inductor's default
picks 4 on small Triton blocks; 8 is better for our decode shapes.
`BLOCKS_ARE_CONTIGUOUS` adds +10% (safe in our setup because
PageTable.reserve allocates pages sequentially on a fresh batch).
TMA adds 2-3%.

Items documented that broke correctness or didn't help:
- ROWS_GUARANTEED_SAFE=true NaNs the softmax on padded batch slots that
  only attend to reserved page 0 (mask returns False for every kv_idx).
- BACKEND="TRITON_DECODE" from the docs raises
  NameError('TRITON_DECODE is not defined') inside Inductor.
- USE_TMA + torch.compile(call_model_with_flex_kwargs) -> misaligned
  address at runtime (compile breaks TMA alignment assumptions).
- torch.compile(flex_attention, mode="max-autotune") nests cudagraph_trees
  inside our raw CUDA graph -> "Cannot prepare for replay during
  capturing stage". max-autotune-no-cudagraphs works but same throughput
  as default mode.
- compile on call_model_with_flex_kwargs: same as eager walker (CUDA
  graph capture already fuses every op in the walker).
- num_warps=4 / 16 both slower than num_warps=8.

CLI surface added to qwen3_flex_inference.py:
  --decode_kernel_options JSON   (FlexKernelOptions for decode)
  --prefill_kernel_options JSON  (same for prefill)
  --compile_model_forward MODE   (optional torch.compile on the walker)
2026-04-20 23:30:05 +00:00
..
cb_sync_smoke.json Phase 2 vibe (10-step): vllm vs unsloth_fi_false vs cb_paged + Phase 3 fixes 2026-04-20 14:23:51 +00:00
flex_8_tuned.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
flex_16_tuned.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
flex_32_tuned.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
flex_32x512_cudagraph.json Breakthrough: flex_attention + paged KV + CUDA graphs = 35-48% of vLLM 2026-04-20 15:57:41 +00:00
flex_32x512_eager.json Breakthrough: flex_attention + paged KV + CUDA graphs = 35-48% of vLLM 2026-04-20 15:57:41 +00:00
flex_32x512_lora_cudagraph.json Flex+CUDA-graph closes gap to vLLM across batch sizes 2026-04-20 16:09:47 +00:00
flex_32x512_mauto_nocg.json Batch-size sweep: flex@256 vs vLLM@256 + max-autotune check 2026-04-20 16:17:19 +00:00
flex_64_lora_tuned.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
flex_64_tuned.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
flex_64x512_lora_cudagraph.json Flex+CUDA-graph closes gap to vLLM across batch sizes 2026-04-20 16:09:47 +00:00
flex_128_tuned.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
flex_128x512_cudagraph.json Flex+CUDA-graph closes gap to vLLM across batch sizes 2026-04-20 16:09:47 +00:00
flex_256x512.json Batch-size sweep: flex@256 vs vLLM@256 + max-autotune check 2026-04-20 16:17:19 +00:00
grpo_cb_paged_10.json Phase 2 vibe (10-step): vllm vs unsloth_fi_false vs cb_paged + Phase 3 fixes 2026-04-20 14:23:51 +00:00
grpo_cb_paged_10.summary.json Phase 2 vibe (10-step): vllm vs unsloth_fi_false vs cb_paged + Phase 3 fixes 2026-04-20 14:23:51 +00:00
grpo_cb_paged_30.json Phase 2: 30-step equivalence + pairwise diffs vs vLLM 2026-04-20 15:03:32 +00:00
grpo_cb_paged_30.summary.json Phase 2: 30-step equivalence + pairwise diffs vs vLLM 2026-04-20 15:03:32 +00:00
grpo_fi_false_30.json Phase 2: 30-step equivalence + pairwise diffs vs vLLM 2026-04-20 15:03:32 +00:00
grpo_fi_false_30.summary.json Phase 2: 30-step equivalence + pairwise diffs vs vLLM 2026-04-20 15:03:32 +00:00
grpo_unsloth_fi_false_10.json Phase 2 vibe (10-step): vllm vs unsloth_fi_false vs cb_paged + Phase 3 fixes 2026-04-20 14:23:51 +00:00
grpo_unsloth_fi_false_10.summary.json Phase 2 vibe (10-step): vllm vs unsloth_fi_false vs cb_paged + Phase 3 fixes 2026-04-20 14:23:51 +00:00
grpo_vllm_10.json Phase 2 vibe (10-step): vllm vs unsloth_fi_false vs cb_paged + Phase 3 fixes 2026-04-20 14:23:51 +00:00
grpo_vllm_10.summary.json Phase 2 vibe (10-step): vllm vs unsloth_fi_false vs cb_paged + Phase 3 fixes 2026-04-20 14:23:51 +00:00
grpo_vllm_30.json Phase 2: 30-step equivalence + pairwise diffs vs vLLM 2026-04-20 15:03:32 +00:00
grpo_vllm_30.summary.json Phase 2: 30-step equivalence + pairwise diffs vs vLLM 2026-04-20 15:03:32 +00:00
lora_cb_paged_fa4_gen.json Phase 1+3: LoRA rollout benchmarks + CB sync driver + Phase 4 scaffold 2026-04-20 14:01:16 +00:00
lora_cb_sdpa_paged_gen.json Phase 1+3: LoRA rollout benchmarks + CB sync driver + Phase 4 scaffold 2026-04-20 14:01:16 +00:00
lora_unsloth_fi_false_gen.json Phase 1+3: LoRA rollout benchmarks + CB sync driver + Phase 4 scaffold 2026-04-20 14:01:16 +00:00
lora_vllm_gen.json Phase 1+3: LoRA rollout benchmarks + CB sync driver + Phase 4 scaffold 2026-04-20 14:01:16 +00:00
notebook_ref_10.json Add Phase 0+1 GRPO backend comparison scaffolding 2026-04-20 13:54:06 +00:00
vllm_8.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
vllm_16.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
vllm_32.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
vllm_64.json FlexKernelOptions sweep: flex reaches 72% of vLLM at batch 64 + LoRA 2026-04-20 23:30:05 +00:00
vllm_64x512_lora.json Flex+CUDA-graph closes gap to vLLM across batch sizes 2026-04-20 16:09:47 +00:00
vllm_128x512.json Flex+CUDA-graph closes gap to vLLM across batch sizes 2026-04-20 16:09:47 +00:00
vllm_256x512.json Batch-size sweep: flex@256 vs vLLM@256 + max-autotune check 2026-04-20 16:17:19 +00:00