ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

(GRPO) Fix PEFT replacement for TRL >= 1.7.0, add missing compute_aux_loss for TRL >= 1.7.0 (#6904)

Browse source 
* Fix PEFT replacement for TRL >= 1.7.0, add missing compute_aux_loss for TRL 1.7.0

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix GRPO for TRL >= 1.7.0: PEFT ref-adapter removal and return arity

rl.py: for trl >= 1.7.0, scope the PEFT removal regex to the ref-adapter
block only by anchoring the end on ref_param.data.copy_(param.data), so it
no longer also deletes the following gradient-checkpointing
enable_input_require_grads() block. Neutralize TRL 1.7.0's
`if _is_quantized_model:` bf16 cast the same way the existing
is_loaded_in_4bit cast is handled.

rl_replacements.py: initialize _extra_moe_kwargs before use (it was
referenced before assignment whenever compute_aux_loss was passed) and only
request output_router_logits when the aux loss is actually wanted.

rl_replacements.py: _get_per_token_logps_and_entropies now returns a 3-tuple
(logps, entropies, aux_loss) for trl >= 1.7.0 and a 2-tuple for older TRL,
matching how every TRL call site unpacks the result. Without this, TRL 1.7.x
_generate_and_score_completions unpacks 3 values from a 2-tuple and raises
"not enough values to unpack (expected 3, got 2)".

* Return zero aux_loss placeholder and drop inference-mode aux collection

* GRPO TRL >= 1.7.0: reject router aux-loss opt-in at init; drop zero aux placeholder

Unsloth's optimized GRPO forward cannot compute the MoE router auxiliary loss.
Previously an explicit opt-in (router_aux_loss_coef > 0) returned a fabricated
zero, silently training without the requested load-balancing penalty. Now reject
it at trainer init with a clear NotImplementedError, and return None (not zero)
for the aux slot of TRL's 3-tuple. Default stays off (coef 0), so the common
path is unaffected.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* GRPO hidden-states fallback: free ModelOutput before chunked log-softmax

The old/ref logprob fallback binds the full ModelOutput (which holds every
layer's hidden_states when output_hidden_states=True) and kept it alive across
chunked_hidden_states_selective_log_softmax, an avoidable OOM on large models.
Extract logits then del outputs in both the text and VLM branches.

* Version-compat CI: proactively catch TRL GRPO breakage

The existing TRL canary is a static symbol/source grep: it verifies symbols
exist but is blind to structural changes (TRL 1.7.0's 2->3-tuple per-token-logps
return arity and restructured PEFT ref-adapter block, which the fix in this PR
addresses, both slipped past it because the methods still existed).

Two additions:
- test_trl_grpo_pinned_symbols.py: extend TRL_TAGS to 1.5/1.6/1.7 and pin the
  exact source-string contracts the rl.py / rl_replacements.py transforms depend
  on for TRL >= 1.7.0 (PEFT elif ref-adapter block + enable_input_require_grads
  survival, if _is_quantized_model, aux_loss_enabled anchor, compute_aux_loss
  arity). A future TRL change fails on main a few days before the PyPI release.
- test_trl_grpo_fake_run.py + a version-compat-ci job: fake-CUDA run that drives
  the real GRPO/SFT/DPO source-transform patchers against latest + main TRL on a
  CPU-only runner (no training) and asserts the generated Unsloth trainer still
  satisfies the transform contracts. Catches behavioral regressions the grep
  cannot see.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fake-run test: use a normal Version import for the aux gate

* version-compat CI: fix fake-run job gate + torch-absent collection

- Drop the invalid job-level matrix if (matrix is not available in
  jobs.<id>.if -> 'Unrecognized named-value: matrix' fails the whole
  workflow). Use a single job that runs vs TRL latest always and re-runs
  vs TRL main only on schedule/dispatch via a step-level github.event_name
  guard. Validated with actionlint.
- Module-level skip the fake-run test when torch is absent so
  daily-fresh-fetch (pytest-only, collects tests/version_compat/) does not
  crash on the top-level spoof import.

* fake-run test: do not skip on import failure

unsloth/trl are installed in the grpo-fake-run job, so a failing import is the
import-time drift this canary must catch. Keep only the not-installed find_spec
skips; let a real import error fail the test.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* GRPO arity gate: regex downgrade + fail loud + CI coverage

The TRL < 1.7.0 per-token-logps return downgrade was an exact-string replace
anchored on the full return line incl. its comment, so a reformat (e.g.
pre-commit) could silently no-op it and ship a 3-tuple to older TRL. Switch to
a regex tolerant of comment/whitespace drift, and raise if the anchor stops
matching (re.subn count != 1) instead of failing silently. Add a monkeypatched
trl_version unit test asserting both arities, since CI only installs TRL >= 1.7.0
and never exercised the downgrade otherwise.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fake-run: give SFT/DPO a real contract, not just ast-parse

The SFT/DPO fake patch runs only checked the generated trainer parses. Also
assert the shared QLoRA _is_quantized_model bf16 cast is neutralized (TRL 1.7's
spelling, present in both sft_trainer and dpo_trainer), so a structural TRL
change to that block is caught for SFT/DPO too, not just GRPO.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* GRPO PEFT ref-adapter removal: lower gate to the TRL 1.4.0 floor

The elif is_peft_model(model) and args.beta != 0.0: ref-adapter block was
introduced in TRL 1.4.0 and is unchanged through 1.7.x, but the removal was
gated at >= 1.7.0, so for 1.4 <= TRL < 1.7 the transform fell through to the
0.27 branch (which matches the older if is_peft_available()... form) and
silently no-oped: a PEFT + beta != 0 GRPO run then computed the KL reference
from the copied ref adapter instead of the base model. Lower the gate to
1.4.0 and keep the 1.7.0-only router aux-loss fail-fast nested. Widen the
pinned-symbol contract test to run from 1.4.0 so the covered versions are
actually exercised.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
This commit is contained in:
marcandrelarochelle 2026-07-08 07:05:03 -04:00 • committed by GitHub
commit 07c8bbbf5a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 504 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

77
.github/workflows/version-compat-ci.yml vendored
View file

@ -285,6 +285,83 @@ jobs:
tests/vllm_compat/test_extended_module_imports.py \
-v --tb=short
# Fake-CUDA GRPO/SFT/DPO patch run against REAL TRL (latest + main). Unlike
# the static symbol/source greps above, this drives unsloth's actual
# source-transform patchers (models/rl.py + rl_replacements.py) on a CPU-only
# runner under the tests/conftest.py spoof harness -- no GPU, no training.
# Catches structural TRL drift the greps miss (e.g. TRL 1.7.0's 2->3-tuple
# per-token-logps return, restructured PEFT ref-adapter block) by asserting
# the generated Unsloth trainer still satisfies the transform contracts.
grpo-fake-run:
name: GRPO fake-run (latest + main TRL, CPU spoof)
runs-on: ubuntu-latest
timeout-minutes: 18
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
path: unsloth
- name: Clone unsloth-zoo @ main
run: |
for attempt in 1 2 3; do
rm -rf "$RUNNER_TEMP/unsloth-zoo"
if git clone --depth=1 https://github.com/unslothai/unsloth-zoo \
"$RUNNER_TEMP/unsloth-zoo"; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "::error::git clone unsloth-zoo failed after 3 attempts"
exit 1
fi
delay=$((5 * attempt))
echo "::warning::clone failed (attempt $attempt/3), retrying in ${delay}s..."
sleep "$delay"
done
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install CPU torch + ecosystem + TRL latest
run: |
python -m pip install --upgrade pip
pip install --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple \
'torch>=2.4,<2.11' 'torchvision<0.26' 'torchcodec<0.10'
# Ecosystem floors unsloth needs; TRL itself is installed last so it
# can pull the transformers/peft it requires.
pip install \
'transformers>=4.57' 'peft>=0.18.0' 'accelerate>=1.0' 'datasets>=3.4,<5' \
'bitsandbytes>=0.45.5' sentencepiece protobuf safetensors numpy 'pytest>=8' \
'huggingface_hub>=0.34' tqdm packaging psutil triton Pillow
pip install --upgrade trl
pip install --no-deps -e "$RUNNER_TEMP/unsloth-zoo"
pip install --no-deps -e ./unsloth
- name: Fake-run vs TRL latest
env:
UNSLOTH_IS_PRESENT: '1'
UNSLOTH_COMPILE_DISABLE: '1'
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: python
run: |
cd unsloth
python -c "import trl; print('Resolved TRL', trl.__version__)"
PYTHONPATH=. python -m pytest \
tests/version_compat/test_trl_grpo_fake_run.py \
-v --tb=short
# `main` is scheduled/dispatch-only so PR jobs stay fast and a bleeding-edge
# TRL break does not red every PR. github.event_name is valid in a step if.
- name: Fake-run vs TRL main (scheduled / dispatch only)
if: ${{ github.event_name != 'pull_request' }}
env:
UNSLOTH_IS_PRESENT: '1'
UNSLOTH_COMPILE_DISABLE: '1'
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: python
run: |
pip install --upgrade "git+https://github.com/huggingface/trl"
cd unsloth
python -c "import trl; print('Resolved TRL', trl.__version__)"
PYTHONPATH=. python -m pytest \
tests/version_compat/test_trl_grpo_fake_run.py \
-v --tb=short
# Daily-only: same suites but with --strict on importable upstream
# tags. Schedule-only so PR jobs stay fast; cron tolerates a flake.
daily-fresh-fetch: