Daniel Han 520f548809 Flex+CUDA-graph closes gap to vLLM across batch sizes ... Expanded benchmark sweep with the flex_attention + paged-KV path: | Batch | LoRA | vLLM tok/s | flex tok/s | flex / vLLM | |-------|------|-----------:|-----------:|------------:| | 32 | no | 7224 | 2189 | 30 % | | 32 | yes | 4581 | 2334 | 51 % | | 64 | yes | 7775 | 4279 | 55 % | | 128 | no | 14996 | 6501 | 43 % | Before this PR, transformers CB topped out at 9.2 % of vLLM on the reference (batch 32 + LoRA) workload. The flex path reaches 51 % on the same config and 55 % at batch 64. Details in scripts/benchmarks/results/flex_vs_vllm.md plus raw stats for each run. Output coherence verified by sampling the first three completions; see `sample_completions` in the stats JSONs. qwen3_flex_inference.py: added sample_completions + decode_tps_best to the output JSON so the PR writeup can cite both median and steady-state numbers without rerunning. Memory: flex uses 44-81 GB depending on batch, vs vLLM's 156 GB at every configuration. That's half to a fifth of vLLM's footprint. Remaining gap is kernel-level (vLLM uses FlashInfer / TRTLLM kernels tuned for sm_100, flex uses Inductor-generated Triton) plus chunked prefill (flex still does separate prefill passes per new batch). Closing those is out of scope for this PR.		2026-04-20 16:09:47 +00:00
..
stats	Flex+CUDA-graph closes gap to vLLM across batch sizes	2026-04-20 16:09:47 +00:00
flex_vs_vllm.md	Flex+CUDA-graph closes gap to vLLM across batch sizes	2026-04-20 16:09:47 +00:00
grpo_equivalence.md	Phase 2: 30-step equivalence + pairwise diffs vs vLLM	2026-04-20 15:03:32 +00:00
lora_rollout_baselines.md	Phase 1+3: LoRA rollout benchmarks + CB sync driver + Phase 4 scaffold	2026-04-20 14:01:16 +00:00
notebook_ref_10.md	Add Phase 0+1 GRPO backend comparison scaffolding	2026-04-20 13:54:06 +00:00