61c2e5c105
Prior default was `peft_model.merge_and_unload()` which bakes LoRA into
the base and destroys the adapter. Same inference speed, but the adapter
is unrecoverable so you can't train on it for the next rollout -- which
GRPO explicitly needs.
Switch default to `peft_model.merge_adapter()`, which:
- Folds LoRA into `base_layer.weight` non-destructively.
- Keeps `lora_A` / `lora_B` parameters intact.
- Flips a `merged` flag inside each `LoraLayer` so its forward
short-circuits to just `base_layer(x)`, giving identical inference speed
to the destructive merge.
- Is fully reversible via `unmerge_adapter()` (bf16 round-trip error ~6e-5).
Measured end-to-end at batch 64 + LoRA rank 32:
- merge_adapter: 5785 tok/s best (was 5744 with merge_and_unload)
- merge+unmerge cycle: ~48 ms total for the 36-layer 7-target adapter,
which is <1 % of a ~5-7 s rollout -- fully amortizable per iteration.
This is *the* rollout path GRPO should use. vLLM's LoRARequest achieves
the same outcome via double-copy (pristine base + materialized base+LoRA
copy) or Punica-style fused kernels, but from a throughput standpoint
both get you to "near-merged speed with adapter separable for training".
Reframes the writeup: removes the previous panic correction that claimed
flex was 35 % of vLLM. The 35 % row is what you'd get with a naive PEFT
wrapper (3 matmuls per projection) -- a path nobody should actually
use. The real headline is still flex reaches 74 % of vLLM at 3.5 x less
memory under proper LoRA semantics.
`--no_merge_lora` flag preserved for the unmerged-PEFT path; documented as
reference only. 4-bit still uses the unmerged path (bnb merging is
unsupported).
|
||
|---|---|---|
| .. | ||
| benchmarks | ||
| enforce_kwargs_spacing.py | ||
| install_gemma4_mlx.sh | ||
| install_qwen3_6_mlx.sh | ||
| run_ruff_format.py | ||