06a1007c6c
PEFT's merge/unmerge pair is asymmetric at bf16 and leaks ~1 ULP per cycle onto base_layer.weight. Across hundreds of GRPO refreshes the base drifts, so the adapter trains against a moving target. Keep a pristine base_model on GPU and a deep-copied inference_model wrapped by PEFT. Before each rollout, restore the inference copy's LoRA-target base_layer weights in-place from pristine and call merge_adapter fresh. Never call unmerge_adapter. Adds --verify_no_drift which hashes base params before/after N perturb+refresh cycles and asserts bit-identical, and checks that the merged inference state is deterministic after restoring the LoRA. Update flex_vs_vllm.md with the double-copy row and memory cost. |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| enforce_kwargs_spacing.py | ||
| install_gemma4_mlx.sh | ||
| install_qwen3_6_mlx.sh | ||
| run_ruff_format.py | ||