2dd8339946
30-step results: | Backend | Train wall | Median step | Peak mem | % of vLLM | |----------------|-----------|-------------|----------|-----------| | vLLM | 215.9 s | 5.14 s | 159 GB | 100 % | | fi_false | 1165.4 s | 41.30 s | 10.7 GB | 12.4 % | | cb_paged | 1564.5 s | 39.82 s | 61.9 GB | 12.9 % | Pairwise diff vs vLLM (30 steps, compare_grpo_runs.py): | Pair | max |loss| | max |kl| | max |reward| | |---------------------------------|------------|------------|---------------| | vLLM vs unsloth_fi_false | 0.39 | **0.015** | 9.25 (noisy) | | vLLM vs cb_paged | 0.83 | (missing) | 6.25 (noisy) | KL trajectory match between vLLM and unsloth_fi_false is the load-bearing equivalence signal: both stay in [0, 0.015] across all 30 steps, so the policy drift guardrail behaves the same. Reward diffs of ~3-9 are expected because the rollout backends produce different completions even at temperature=0.1 (kernel-level non-determinism). Two caveats documented in results/grpo_equivalence.md: - cb_paged's StatisticsCallback doesn't capture TRL's kl log entry because TRL emits kl on a separate log call that doesn't include loss. - cb_paged's grad_norm (~200-900) is unclipped pre-optimizer, while vLLM's goes through Unsloth's internal max_grad_norm=1.0. Not a correctness bug, just not apples-to-apples until cb_paged sets max_grad_norm in GRPOConfig. Also updates the report with Phase 3 + Phase 4 status (CB sync driver eager works; CUDA graph capture hangs on output_ids slice pending fix; torch.compile on training step incompatible with both unsloth_fi_false and cb_paged). |
||
|---|---|---|
| .. | ||
| stats | ||
| grpo_equivalence.md | ||
| lora_rollout_baselines.md | ||
| notebook_ref_10.md | ||