520f548809
Expanded benchmark sweep with the flex_attention + paged-KV path: | Batch | LoRA | vLLM tok/s | flex tok/s | flex / vLLM | |-------|------|-----------:|-----------:|------------:| | 32 | no | 7224 | 2189 | 30 % | | 32 | yes | 4581 | 2334 | 51 % | | 64 | yes | 7775 | 4279 | 55 % | | 128 | no | 14996 | 6501 | 43 % | Before this PR, transformers CB topped out at 9.2 % of vLLM on the reference (batch 32 + LoRA) workload. The flex path reaches 51 % on the same config and 55 % at batch 64. Details in scripts/benchmarks/results/flex_vs_vllm.md plus raw stats for each run. Output coherence verified by sampling the first three completions; see `sample_completions` in the stats JSONs. qwen3_flex_inference.py: added sample_completions + decode_tps_best to the output JSON so the PR writeup can cite both median and steady-state numbers without rerunning. Memory: flex uses 44-81 GB depending on batch, vs vLLM's 156 GB at every configuration. That's half to a fifth of vLLM's footprint. Remaining gap is kernel-level (vLLM uses FlashInfer / TRTLLM kernels tuned for sm_100, flex uses Inductor-generated Triton) plus chunked prefill (flex still does separate prefill passes per new batch). Closing those is out of scope for this PR. |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| enforce_kwargs_spacing.py | ||
| install_gemma4_mlx.sh | ||
| install_qwen3_6_mlx.sh | ||
| run_ruff_format.py | ||