Adds reproducible scripts under scripts/benchmarks/ that compare vLLM
colocated rollouts against the transformers continuous batching API
(model.generate_batch, paged attention) for GRPO training on Qwen3-4B.
Contents:
- unsloth_grpo_common.py: shared dataset, reward functions, and GRPO
hyperparameters so the two backends differ only in the rollout engine.
- qwen3_grpo_vllm.py: baseline training entry using fast_inference=True
and TRL use_vllm=True, vllm_mode=colocate.
- qwen3_grpo_tpaged.py: candidate using a vanilla HF Qwen3 + PEFT LoRA
with TRL use_transformers_paged=True.
- cb_vs_vllm_generation.py: standalone generation microbenchmark.
- README.md: integration notes, reproduction steps, and observed numbers.
On a single B200 with Unsloth Qwen3-4B-Base at LoRA rank 32 bf16,
transformers continuous batching reaches 7-9 percent of vLLM throughput
on this workload. The README documents the integration sharp edges
(top_k=-1, PagedAttentionCache default upper bounds, Unsloth's
Qwen3Attention_fast_forward bypassing the functional attention
interface, TRL importing GuidedDecodingParams from a newer vLLM that no
longer exports it, and UnslothGRPOTrainer expecting for_training /
for_inference hooks on the model).
The scripts are intentionally self-contained so they are easy to rerun
after either upstream change that could close the throughput gap
(flash-attn availability, CUDA graphs in ContinuousBatchingManager,
persistent paged caches across generate_batch calls, or a
paged-compatible Unsloth attention forward).
07939bb025