unsloth/scripts/benchmarks/README.md

Qwen3-4B GRPO rollout engine benchmarks

This directory holds the reproducible scripts behind the experiment documented in the accompanying PR: can Hugging Face transformers' continuous-batching API (model.generate_batch, backed by PagedAttentionCache) serve as a drop-in replacement for vLLM during GRPO rollouts on the Qwen3-4B notebook?

The short answer on a single NVIDIA B200 with Unsloth Qwen3-4B-Base, LoRA rank 32, bf16: transformers continuous batching is functionally correct and integrates with TRL's use_transformers_paged=True path, but end-to-end throughput lands at around 7-9 percent of vLLM colocated. Full numbers and per-step timings are in the PR description.

Files

File	Purpose
unsloth_grpo_common.py	Shared dataset loading, reward functions, and GRPO hyperparameters
qwen3_grpo_vllm.py	vLLM baseline training entry (fast_inference=True, use_vllm=True, vllm_mode="colocate")
qwen3_grpo_tpaged.py	Continuous-batching candidate (fast_inference=False, use_transformers_paged=True, vanilla HF + PEFT)
cb_vs_vllm_generation.py	Standalone generation microbenchmark across both engines

Reproduce

pip install unsloth "transformers>=4.57" "trl>=0.25" peft vllm

# Generation microbenchmark (32 prompts, 512 new tokens each)
CUDA_VISIBLE_DEVICES=2 python scripts/benchmarks/cb_vs_vllm_generation.py \
    --backend vllm --stats_path logs/vllm_gen.json \
    --n_prompts 32 --n_rounds 2 --max_new_tokens 512 \
    --gpu_memory_utilization 0.6

CUDA_VISIBLE_DEVICES=2 python scripts/benchmarks/cb_vs_vllm_generation.py \
    --backend tpaged --stats_path logs/cb_gen.json \
    --n_prompts 32 --n_rounds 2 --max_new_tokens 512

# Full GRPO training (20 steps)
CUDA_VISIBLE_DEVICES=2 python scripts/benchmarks/qwen3_grpo_vllm.py \
    --max_steps 20 --num_generations 2 --per_device_train_batch_size 2 \
    --output_dir outputs/grpo_vllm --stats_path logs/vllm_stats.json \
    --gpu_memory_utilization 0.6

CUDA_VISIBLE_DEVICES=2 python scripts/benchmarks/qwen3_grpo_tpaged.py \
    --max_steps 20 --num_generations 2 --per_device_train_batch_size 2 \
    --output_dir outputs/grpo_tpaged --stats_path logs/tpaged_stats.json \
    --max_batch_tokens 16384 --num_blocks 16384

Known integration notes for transformers continuous batching + TRL + Unsloth

These are the sharp edges you hit going down the continuous-batching path and how qwen3_grpo_tpaged.py handles them:

1. top_k=-1 is not a valid value for transformers. vLLM treats -1 as "disabled", but TopKLogitsWarper raises ValueError: top_k has to be a strictly positive integer. The script rewrites top_k=None on the shared GRPOConfig before handing it to GRPOConfig(use_transformers_paged=True, ...).

2. PagedAttentionCache default upper bounds are extremely conservative. _upper_bound_max_batch_tokens=256 and _upper_bound_num_blocks=4096 choke decode throughput. The script passes generation_kwargs={"max_batch_tokens": 16384, "num_blocks": 16384} which TRL forwards to GenerationConfig, and the CB manager reads them when sizing the paged cache.

3. Unsloth's Qwen3Attention_fast_forward bypasses the functional attention interface. Calling model.generate_batch on an Unsloth-patched Qwen3 model fails inside unsloth.utils.attention_dispatch.run_attention because Unsloth routes through its own dispatcher rather than reading config._attn_implementation. The benchmark script works around this by loading a vanilla HF Qwen3 with PEFT LoRA for the tpaged path. This costs the Unsloth training kernels but keeps the comparison clean. A proper upstream fix is to detect config._attn_implementation ending in _paged and delegate to the stock transformers forward.

4. TRL imports GuidedDecodingParams from vllm.sampling_params. Newer vLLM releases (>= 0.13) have moved or removed that symbol, so trl.trainer.grpo_trainer fails to import on a fresh vLLM install even if you are not using vLLM. qwen3_grpo_tpaged.py installs a minimal shim before importing TRL.

5. UnslothGRPOTrainer calls model.for_training() / for_inference(). Importing unsloth replaces trl.GRPOTrainer with UnslothGRPOTrainer, which assumes the model has these hooks. A vanilla HF model does not, so qwen3_grpo_tpaged.py does not import unsloth at all.

Why continuous batching is slower than vLLM on this workload

• flash_attn is hard to install on this box (CUDA 13.1 detected, torch compiled against CUDA 12.8), so paged|flash_attention_2 falls back to sdpa_paged. vLLM uses FlashInfer and TRTLLM kernels.
• CB re-allocates a fresh PagedAttentionCache on every generate_batch call. For GRPO that is once per step.
• ContinuousBatchingManager does not yet implement CUDA graphs (use_cuda_graph=True raises NotImplementedError). vLLM captures 100+ mixed prefill-decode and decode graphs during warmup.

These are all upstream transformers issues, not Unsloth issues. The scripts in this directory are intentionally simple so they are easy to port into a future upstream fix.