ab37acd5e0
Previous bf16 + LoRA rank 32 runs called `peft_model.merge_and_unload()`, which bakes LoRA into the base weights and destroys the adapter. Every subsequent forward is then plain bf16 with no LoRA-active cost -- one matmul per projection. vLLM's LoRARequest path keeps LoRA dynamic (base matmul + rank-r adapter matmuls + add), which is what GRPO actually needs because the adapter has to be updateable between rollouts and training steps. Adds `--no_merge_lora` flag and runs the honest comparison: | Backend | tok/s best | vs vLLM | |--------------------------------|-----------:|--------:| | vLLM (LoRARequest) | 7775 | 100 % | | flex -- LoRA merged (prior) | 5744 | 74 % | | flex -- LoRA active (no merge) | 2683 | 35 % | The 74 % number in the earlier writeup was only meaningful if you can eat the merge/unmerge cost between rollouts and training steps (which is not free). The real flex-vs-vLLM gap under GRPO semantics is ~35 %, not 72 %. vLLM wins its dynamic-LoRA number via Punica-style fused kernels that avoid the extra matmul roundtrip. flex has no equivalent and runs base + LoRA_A + LoRA_B as three separate matmuls per projection. Writeup updated. |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| enforce_kwargs_spacing.py | ||
| install_gemma4_mlx.sh | ||
| install_qwen3_6_mlx.sh | ||
| run_ruff_format.py | ||