ci(version-compat): expand TRL coverage + add transformers + PEFT extras
Extend the cross-version compat canary to catch ~80% of upstream drift before a user hits it. Static checks only (GitHub raw fetch + grep), CPU-only, runs PR-time + daily cron. 906 pass, 73 skipped. TRL coverage extended: - TRL_TAGS expanded from 12 to 28 (every stable release >=0.18.2, including the broken 0.19.0, plus main). Anchors: 0.22.2 / 0.27.1 / 1.0.0 marked. - Fix `__version__` parser to handle the TRL 0.22.x pattern (`__version__ = f.read()` from sibling VERSION file). - Fix `has_def` in _fetch.py to allow indented matches so class methods are detected (the original anchored ^def only matched module-scope definitions). - New tests for symbols the audit found we touch but didn't check: is_conversational, sft_trainer module + neftune_post_forward_hook, dpo_trainer module + MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES, trl.trainer.utils.ConstantLengthDataset (gated), trl.models.utils.disable_gradient_checkpointing (gated >=1.0.0), trl.import_utils + _*_available cache pattern, trl.experimental.openenv.utils generators (one of two names), GRPOTrainer required methods (_prepare_inputs, _generate_and_score_completions, compute_loss; per-token-logps legacy/new dispatch), GRPOTrainer source must contain torch.inference_mode + accelerator.unwrap_model fingerprints, KTOTrainer.get_batch_logps (now lives at trl.experimental.kto on TRL 0.27+ — accept either path), SFTTrainer class existence, DPOTrainer methods (informational), chat-template propagation (legacy maybe_apply_chat_template OR successor apply_chat_template + chat_template_kwargs), truncate_with_protected_tokens informational. - Tighten test_unwrap_model_for_generation_either_path to mirror the prod fallback exactly (drop unused trl/extras/profiling.py candidate). - Replace test_trl_generation_vllm_generation_gated symbol set with the actual unsloth dependency (VLLMGeneration class + _init_vllm / sync_weights / generate methods, not VLLMClient/etc). PEFT coverage extended (driven by the 8 PR audit unsloth#5015, #5167, #5036, #4807 + unsloth-zoo#618, #596, #482, #430): - VARIANT_KWARG_KEYS const (peft 0.18+; injected by zoo#430) - ParamWrapper class + members (peft 0.18+; needed by zoo#618) - LoraConfig.target_parameters (peft 0.19+) - LoraModel._create_and_replace (signature pin for unsloth#4807) - transformers_weight_conversion module + build_peft_weight_mapping (unsloth#5167 wraps this) - integrations.dequantize_module_weight (3 callsites) - PeftType.LORA (vllm_utils.py:2520) - ModulesToSaveWrapper (both peft.utils.* paths) - PeftModel.from_pretrained method exists - peft.__version__ parseable Transformers coverage added (driven by the 16-PR audit): - New file test_transformers_pinned_symbols.py with 19 test categories x 12 transformers tags (4.57.6 floor + 5.0..5.8 + main). Anchors: 4.57.6 + 5.5.0. - Trainer surface (compute_loss num_items_in_batch param, training_step grad-accum fingerprints, get_batch_samples num_items contract, inner_training_loop _tr_loss inplace v5) - modeling_utils.checkpoint alias for unsloth-zoo#549 - PushToHubMixin._create_repo presence (unsloth-zoo#393) - integrations.bitsandbytes module + Linear4bit reference - quantizers.should_convert_module signature (zoo#491/#488) - FP8Linear bias/has_bias rename (zoo#572) - processing_utils.Unpack importable (zoo#583/584) - gemma3 Gemma3Attention class + gpt_oss GptOssModel class - auto_factory _LazyAutoMapping private API (unsloth#5155) - configuration_utils PretrainedConfig/PreTrainedConfig alias - tokenization_utils_base.apply_chat_template - modeling_attn_mask_utils symbols - cache_utils Cache + DynamicCache classes - training_args.ParallelMode importable Wire the new transformers job into version-compat-ci.yml (matrix of 5 PR-time symbol jobs + zoo-imports under spoof + daily fresh- fetch cron). Local smoke: 906 pass, 73 skipped (gated optional features) across vLLM + TRL + PEFT + ST + bnb + transformers suites.
This commit is contained in:
Daniel Han
parent
c876296dca
commit
b205ddda65
5 changed files with 1160 additions and 40 deletions
22
.github/workflows/version-compat-ci.yml
vendored
22
.github/workflows/version-compat-ci.yml
vendored
|
|
@ -163,6 +163,28 @@ jobs:
|
|||
tests/version_compat/test_bitsandbytes_pinned_symbols.py \
|
||||
-v --tb=short
|
||||
|
||||
transformers-pinned-symbols:
|
||||
name: transformers pinned-symbol matrix (4.57.6 + 5.x + main)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 12
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
|
||||
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
cache: 'pip'
|
||||
- name: Install pytest only
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install 'pytest>=8'
|
||||
- name: Run transformers compat suite
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
PYTHONPATH=. python -m pytest \
|
||||
tests/version_compat/test_transformers_pinned_symbols.py \
|
||||
-v --tb=short
|
||||
|
||||
# Optional second layer: actually `pip install` ONE representative
|
||||
# version of each package and verify unsloth + unsloth-zoo modules
|
||||
# import on it under the existing CUDA spoof. CPU-only, runs on
|
||||
|
|
|
|||
|
|
@ -45,18 +45,23 @@ def fetch_text(repo: str, ref: str, path: str) -> str | None:
|
|||
|
||||
def has_def(src: str, name: str, kind: str = "any") -> bool:
|
||||
"""Heuristic AST-equivalent grep for `class Name`, `def name`,
|
||||
or `Name = ...` at module scope. We avoid a full ast.parse so a
|
||||
single non-importable line (e.g. `# type: ignore` after an
|
||||
unresolved alias) doesn't false-fail us."""
|
||||
or `Name = ...` — at any indent level. We avoid a full ast.parse
|
||||
so a single non-importable line (e.g. `# type: ignore` after an
|
||||
unresolved alias) doesn't false-fail us. Indented matches are
|
||||
accepted because most class methods we want to verify live four
|
||||
spaces in (and tests should pass for `class.method` definitions
|
||||
just as much as for module-level `def`)."""
|
||||
if kind in ("any", "class") and re.search(
|
||||
rf"^class\s+{re.escape(name)}\b", src, re.MULTILINE
|
||||
rf"^\s*class\s+{re.escape(name)}\b", src, re.MULTILINE
|
||||
):
|
||||
return True
|
||||
if kind in ("any", "func") and re.search(
|
||||
rf"^(?:async\s+)?def\s+{re.escape(name)}\b", src, re.MULTILINE
|
||||
rf"^\s*(?:async\s+)?def\s+{re.escape(name)}\b", src, re.MULTILINE
|
||||
):
|
||||
return True
|
||||
if kind == "any" and re.search(rf"^{re.escape(name)}\s*[:=]", src, re.MULTILINE):
|
||||
if kind == "any" and re.search(
|
||||
rf"^\s*{re.escape(name)}\s*[:=]", src, re.MULTILINE
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
|
|
|||
|
|
@ -27,9 +27,11 @@ test against it.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.version_compat._fetch import fetch_text, has_def
|
||||
from tests.version_compat._fetch import fetch_text, first_match, has_def
|
||||
|
||||
|
||||
# pyproject pin: peft>=0.18.0. Test the floor + each minor since.
|
||||
|
|
@ -173,3 +175,240 @@ def test_peft_lora_bnb_integration(tag: str):
|
|||
f"{tag}: peft.tuners.lora.bnb missing or no Linear4bit/Linear8bitLt "
|
||||
f"class found; unsloth's 4-bit LoRA path silently degrades to fp16"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# Coverage extension (added 2026-05): symbols from the 8-PR audit
|
||||
# unsloth#5015, #5167, #5036, #4807 + unsloth-zoo#618, #596, #482, #430.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 1. peft.tuners.lora.layer.VARIANT_KWARG_KEYS — added in peft 0.18.
|
||||
# unsloth-zoo#430 injects the import into the compiled forward.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_variant_kwarg_keys_const(tag: str):
|
||||
src = fetch_text("huggingface/peft", tag, "src/peft/tuners/lora/layer.py")
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: src/peft/tuners/lora/layer.py missing")
|
||||
if "VARIANT_KWARG_KEYS" not in src:
|
||||
pytest.fail(
|
||||
f"{tag}: peft.tuners.lora.layer.VARIANT_KWARG_KEYS missing; "
|
||||
f"unsloth_zoo/compiler.py:2645 import injection breaks (unsloth-zoo#430)"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 2. peft.tuners.lora.layer.ParamWrapper — peft 0.18 added the class
|
||||
# for MoE 3D-parameter LoRA. Required attrs: parameter_name, lora_A,
|
||||
# forward, get_base_layer. peft 0.19 also added _did_swap_in_out_features.
|
||||
# unsloth-zoo#618 monkey-patches the MoE LoRA extractor.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_param_wrapper_class(tag: str):
|
||||
src = fetch_text("huggingface/peft", tag, "src/peft/tuners/lora/layer.py")
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: layer.py missing")
|
||||
assert has_def(src, "ParamWrapper", "class"), (
|
||||
f"{tag}: peft.tuners.lora.layer.ParamWrapper missing; "
|
||||
f"unsloth_zoo/temporary_patches/qwen3_moe.py:43-130 + "
|
||||
f"moe_utils.py:757 ImportError (unsloth-zoo#618)"
|
||||
)
|
||||
# Required member names — informational only; the class may
|
||||
# legitimately move some to a base class. The bug we want to
|
||||
# catch is full-class-removal.
|
||||
for name in ("parameter_name", "forward", "lora_A", "get_base_layer"):
|
||||
_present = name in src
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 3. peft.tuners.lora.LoraConfig.target_parameters — peft 0.19+. Used
|
||||
# by unsloth-zoo's MoE target-parameter extractor.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_lora_config_target_parameters(tag: str):
|
||||
src = fetch_text("huggingface/peft", tag, "src/peft/tuners/lora/config.py")
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: src/peft/tuners/lora/config.py missing")
|
||||
# Optional on 0.18.x; required from 0.19.0+. Don't fail older
|
||||
# versions; the test is informational below the floor.
|
||||
has_it = "target_parameters" in src
|
||||
if "0.18" in tag and not has_it:
|
||||
pytest.skip(f"{tag}: target_parameters not yet introduced (peft 0.18)")
|
||||
assert has_it, (
|
||||
f"{tag}: LoraConfig.target_parameters missing on peft >=0.19; "
|
||||
f"unsloth-zoo MoE target-parameter extraction breaks"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 4. peft.tuners.lora.model.LoraModel._create_and_replace — unsloth#4807
|
||||
# monkey-patches this for Gemma4ClippableLinear. Signature pin.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_lora_model_create_and_replace(tag: str):
|
||||
src = fetch_text("huggingface/peft", tag, "src/peft/tuners/lora/model.py")
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: src/peft/tuners/lora/model.py missing")
|
||||
assert has_def(src, "LoraModel", "class"), (
|
||||
f"{tag}: class LoraModel missing"
|
||||
)
|
||||
assert has_def(src, "_create_and_replace", "func"), (
|
||||
f"{tag}: LoraModel._create_and_replace missing; "
|
||||
f"unsloth/models/loader.py:1535-1601 monkey-patch breaks (unsloth#4807)"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 5. peft.utils.transformers_weight_conversion.{build_peft_weight_mapping,
|
||||
# WeightConversion} — unsloth#5167 wraps build_peft_weight_mapping
|
||||
# to handle WeightConversion.__init__ kwargs (distributed_operation,
|
||||
# quantization_operation).
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_transformers_weight_conversion_module(tag: str):
|
||||
candidates = [
|
||||
"src/peft/utils/transformers_weight_conversion.py",
|
||||
"src/peft/utils/transformers_weight_conversion/__init__.py",
|
||||
]
|
||||
hit = first_match("huggingface/peft", tag, candidates)
|
||||
if hit is None:
|
||||
pytest.skip(
|
||||
f"{tag}: transformers_weight_conversion not present (legacy peft)"
|
||||
)
|
||||
_, src = hit
|
||||
assert has_def(src, "build_peft_weight_mapping", "func") or "build_peft_weight_mapping" in src, (
|
||||
f"{tag}: build_peft_weight_mapping missing in transformers_weight_conversion; "
|
||||
f"unsloth/import_fixes.py:1375-1456 wrap breaks (unsloth#5167)"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 6. peft.utils.integrations.dequantize_module_weight — used by 3 unsloth/
|
||||
# unsloth-zoo callsites. Function name + module path.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_integrations_dequantize_module_weight(tag: str):
|
||||
candidates = [
|
||||
"src/peft/utils/integrations.py",
|
||||
"src/peft/utils/integrations/__init__.py",
|
||||
]
|
||||
hit = first_match("huggingface/peft", tag, candidates)
|
||||
assert hit is not None, (
|
||||
f"{tag}: src/peft/utils/integrations[.py|/__init__.py] both missing"
|
||||
)
|
||||
_, src = hit
|
||||
assert has_def(src, "dequantize_module_weight", "func") or "dequantize_module_weight" in src, (
|
||||
f"{tag}: peft.utils.integrations.dequantize_module_weight missing; "
|
||||
f"unsloth-zoo vllm_utils.py:2701, unsloth/_utils.py:1550, "
|
||||
f"saving_utils.py:270 ImportError"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 7. peft.PeftType.LORA — used by unsloth-zoo vllm_utils.py:2520-2559.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_type_lora_enum(tag: str):
|
||||
candidates = [
|
||||
"src/peft/utils/peft_types.py",
|
||||
"src/peft/utils/__init__.py",
|
||||
"src/peft/__init__.py",
|
||||
]
|
||||
for p in candidates:
|
||||
src = fetch_text("huggingface/peft", tag, p)
|
||||
if src is None:
|
||||
continue
|
||||
# Either `class PeftType(...)` definition with LORA member, or
|
||||
# re-export from a submodule.
|
||||
if "PeftType" in src and ("LORA" in src or "lora" in src.lower()):
|
||||
return
|
||||
pytest.fail(
|
||||
f"{tag}: peft.PeftType (with LORA member) not in any of {candidates}; "
|
||||
f"unsloth-zoo vllm_utils.py:2520 reference breaks"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 8. peft.utils.ModulesToSaveWrapper — both peft.utils.* and
|
||||
# peft.utils.other.* import paths used.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_modules_to_save_wrapper(tag: str):
|
||||
candidates = [
|
||||
"src/peft/utils/other.py",
|
||||
"src/peft/utils/__init__.py",
|
||||
]
|
||||
found_in = []
|
||||
for p in candidates:
|
||||
src = fetch_text("huggingface/peft", tag, p)
|
||||
if src is None:
|
||||
continue
|
||||
if has_def(src, "ModulesToSaveWrapper", "class"):
|
||||
found_in.append(p)
|
||||
assert found_in, (
|
||||
f"{tag}: ModulesToSaveWrapper not defined in {candidates}; "
|
||||
f"unsloth/training_utils.py:239 + models/llama.py:153 ImportError"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 9. peft.PeftModel.from_pretrained signature pin — unsloth#4807
|
||||
# call site uses (model, name, token, revision, is_trainable,
|
||||
# trust_remote_code).
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_peft_model_from_pretrained_signature(tag: str):
|
||||
src = fetch_text("huggingface/peft", tag, "src/peft/peft_model.py")
|
||||
assert src is not None, f"{tag}: src/peft/peft_model.py missing"
|
||||
# We expect `def from_pretrained` in PeftModel class. Just check
|
||||
# the method name exists; full kwarg list is too brittle.
|
||||
assert has_def(src, "from_pretrained", "func"), (
|
||||
f"{tag}: PeftModel.from_pretrained missing in peft_model.py"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 10. peft.__version__ exported via known mechanism.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", PEFT_TAGS)
|
||||
def test_peft_version_parseable(tag: str):
|
||||
src = fetch_text("huggingface/peft", tag, "src/peft/__init__.py")
|
||||
assert src is not None
|
||||
# Same gates as the TRL test: literal / submodule / metadata / VERSION file.
|
||||
has_literal = bool(re.search(r'^__version__\s*=\s*["\']', src, re.MULTILINE))
|
||||
has_subimport = bool(
|
||||
re.search(r"^from\s+\.version\s+import\s+__version__", src, re.MULTILINE)
|
||||
)
|
||||
has_metadata = bool(
|
||||
re.search(
|
||||
r"^from\s+importlib\.metadata\s+import\s+(?:[\w,\s]+,\s*)?version",
|
||||
src,
|
||||
re.MULTILINE,
|
||||
)
|
||||
and re.search(r"^\s*__version__\s*=\s*version\s*\(", src, re.MULTILINE)
|
||||
)
|
||||
assert has_literal or has_subimport or has_metadata, (
|
||||
f"{tag}: peft.__version__ not exported via any known mechanism"
|
||||
)
|
||||
|
|
|
|||
448
tests/version_compat/test_transformers_pinned_symbols.py
Normal file
448
tests/version_compat/test_transformers_pinned_symbols.py
Normal file
|
|
@ -0,0 +1,448 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team.
|
||||
"""Pinned-symbol + source-pattern compat checks across the
|
||||
transformers PyPI window unsloth + unsloth-zoo target. Catches the
|
||||
classes of breakage we've shipped fixes for in:
|
||||
|
||||
unsloth#3998 notebook compat 4.57.6 + TRL 0.22-0.27
|
||||
unsloth#5036 grad-accum accepts_loss_kwargs vision wrappers
|
||||
unsloth#5155 resolve_model_class fallback against unresolvable AutoModel
|
||||
unsloth#5259 FastSentenceTransformer + ST 5.4 redirect
|
||||
unsloth-zoo#572 forward-compat with transformers 5.x decorators + Qwen2VL
|
||||
unsloth-zoo#571 gemma3, csm, ministral, pixtral 5.3 forward signature
|
||||
unsloth-zoo#549 VRAM regression with transformers 5.2+ checkpoint
|
||||
unsloth-zoo#543 GRPO logging + transformers v5 loss shape mismatch
|
||||
unsloth-zoo#541 got multiple values for argument in compiled forward dispatch
|
||||
unsloth-zoo#495 Qwen3Next/Qwen3.5 MoE + transformers v5 fixes for Gemma
|
||||
unsloth-zoo#491 should_convert_module substring matching
|
||||
unsloth-zoo#488 Gemma3 + Gemma3N transformers 5.x
|
||||
unsloth-zoo#472 ModernBERT, gpt_oss MoE unwrap, SFTTrainer skip_prepare_dataset
|
||||
unsloth-zoo#393 PushToHubMixin._create_repo removed in v5
|
||||
unsloth-zoo#388 generation_config attribute removed for non-gen models in v5
|
||||
unsloth-zoo#583/584 PIL _Ink ImportError (Unpack import guard)
|
||||
unsloth-zoo#159 cross_entropy_replacement_2 num_items_in_batch fallback
|
||||
|
||||
Strategy: GitHub raw-fetch + grep / source-fingerprint. CPU-only, no
|
||||
install. Runs PR-time + daily cron.
|
||||
|
||||
Anchor versions (must work forwards/backwards-compat per project spec):
|
||||
transformers 4.57.6, 5.5.0
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.version_compat._fetch import fetch_text, first_match, has_def
|
||||
|
||||
|
||||
# Stable transformers from 4.57.6 floor onwards + main. The breakage
|
||||
# windows we care about are 4.57.6, then every 5.x minor since 5.0.0.
|
||||
TRANSFORMERS_TAGS = [
|
||||
"v4.57.6", # anchor (must work)
|
||||
"v5.0.0",
|
||||
"v5.1.0",
|
||||
"v5.2.0",
|
||||
"v5.3.0",
|
||||
"v5.4.0",
|
||||
"v5.5.0", # anchor (must work)
|
||||
"v5.5.4",
|
||||
"v5.6.2",
|
||||
"v5.7.0",
|
||||
"v5.8.0",
|
||||
"main",
|
||||
]
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# Trainer surface — the largest failure class. unsloth/models/_utils.py
|
||||
# rewrites Trainer.{__init__, training_step, get_batch_samples, compute_loss}.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_trainer_class_importable_path(tag: str):
|
||||
"""transformers.Trainer must remain at src/transformers/trainer.py
|
||||
or src/transformers/trainer/__init__.py."""
|
||||
candidates = ["src/transformers/trainer.py", "src/transformers/trainer/__init__.py"]
|
||||
hit = first_match("huggingface/transformers", tag, candidates)
|
||||
assert hit is not None, (
|
||||
f"{tag}: src/transformers/trainer[.py|/__init__.py] both missing"
|
||||
)
|
||||
_, src = hit
|
||||
assert has_def(src, "Trainer", "class"), f"{tag}: class Trainer missing"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_trainer_compute_loss_num_items_in_batch_param(tag: str):
|
||||
"""unsloth-zoo#159 + unsloth#4998 + #4616: Trainer.compute_loss
|
||||
must accept num_items_in_batch kwarg. transformers 4.46+ added it."""
|
||||
candidates = ["src/transformers/trainer.py", "src/transformers/trainer/__init__.py"]
|
||||
hit = first_match("huggingface/transformers", tag, candidates)
|
||||
assert hit is not None
|
||||
_, src = hit
|
||||
# Find the compute_loss signature - it's a class method, indented.
|
||||
m = re.search(
|
||||
r"^\s*def compute_loss\(([^)]*)\)", src, re.MULTILINE | re.DOTALL
|
||||
)
|
||||
if m is None:
|
||||
pytest.fail(f"{tag}: Trainer.compute_loss not found in source")
|
||||
assert "num_items_in_batch" in m.group(1), (
|
||||
f"{tag}: Trainer.compute_loss signature missing num_items_in_batch param; "
|
||||
f"unsloth grad-accum patches assume this kwarg present"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_trainer_training_step_grad_accum_pattern(tag: str):
|
||||
"""unsloth#3598 monkey-patches Trainer.training_step source; the
|
||||
rewrite needs four substrings to be present. Drift here = silent
|
||||
no-op = double-scale loss bug."""
|
||||
candidates = ["src/transformers/trainer.py", "src/transformers/trainer/__init__.py"]
|
||||
hit = first_match("huggingface/transformers", tag, candidates)
|
||||
assert hit is not None
|
||||
_, src = hit
|
||||
needed = (
|
||||
"loss *= self.args.gradient_accumulation_steps",
|
||||
"if self.model_accepts_loss_kwargs:",
|
||||
"self.accelerator.backward(loss",
|
||||
)
|
||||
missing = [s for s in needed if s not in src]
|
||||
# Hard-fail only when ALL substrings missing — partial drift is
|
||||
# informational. Note: the third one's exact form may vary slightly.
|
||||
if len(missing) == len(needed):
|
||||
pytest.fail(
|
||||
f"{tag}: Trainer.training_step has none of the grad-accum "
|
||||
f"fingerprints {needed}; unsloth/models/_utils.py:1689-1791 "
|
||||
f"patch silently no-ops -> double-scale loss"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_trainer_get_batch_samples_returns_num_items(tag: str):
|
||||
"""unsloth-zoo loss_utils.py:241 replaces Trainer.get_batch_samples;
|
||||
upstream signature must end `return batch_samples, num_items_in_batch`."""
|
||||
candidates = ["src/transformers/trainer.py", "src/transformers/trainer/__init__.py"]
|
||||
hit = first_match("huggingface/transformers", tag, candidates)
|
||||
assert hit is not None
|
||||
_, src = hit
|
||||
if not has_def(src, "get_batch_samples", "func"):
|
||||
pytest.skip(f"{tag}: get_batch_samples not yet on Trainer")
|
||||
assert "num_items_in_batch" in src, (
|
||||
f"{tag}: Trainer.get_batch_samples / num_items_in_batch contract missing"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_trainer_inner_training_loop_inplace_loss_v5(tag: str):
|
||||
"""unsloth-zoo#543: transformers 5.0+ changed
|
||||
`tr_loss = tr_loss + tr_loss_step` (out-of-place) to
|
||||
`self._tr_loss += tr_loss_step` (in-place). Loss tensor shape
|
||||
requirements differ. Snapshot which form is in source."""
|
||||
candidates = ["src/transformers/trainer.py", "src/transformers/trainer/__init__.py"]
|
||||
hit = first_match("huggingface/transformers", tag, candidates)
|
||||
assert hit is not None
|
||||
_, src = hit
|
||||
has_inplace = "self._tr_loss +=" in src
|
||||
has_outplace = "tr_loss = tr_loss + tr_loss_step" in src
|
||||
# On 4.57.6, only out-of-place. On 5.x, in-place. We just assert
|
||||
# ONE of them is present so a future refactor that drops both is
|
||||
# caught.
|
||||
assert has_inplace or has_outplace, (
|
||||
f"{tag}: Trainer._inner_training_loop has neither "
|
||||
f"`tr_loss = tr_loss + tr_loss_step` nor `self._tr_loss +=`; "
|
||||
f"unsloth-zoo#543 patch breaks"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# modeling_utils — checkpoint, PushToHubMixin, ALL_ATTENTION_FUNCTIONS.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_modeling_utils_exposes_checkpoint(tag: str):
|
||||
"""unsloth-zoo#549: transformers 5.2+ uses `transformers.modeling_utils.checkpoint`
|
||||
(alias for torch.utils.checkpoint.checkpoint). Patch must replace
|
||||
the transformers reference, not just torch's."""
|
||||
src = fetch_text("huggingface/transformers", tag, "src/transformers/modeling_utils.py")
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: modeling_utils.py missing")
|
||||
# Either a direct import or local rebinding.
|
||||
has_import = bool(
|
||||
re.search(
|
||||
r"^from\s+torch\.utils\.checkpoint\s+import\s+checkpoint",
|
||||
src,
|
||||
re.MULTILINE,
|
||||
)
|
||||
or re.search(r"^import\s+torch\.utils\.checkpoint", src, re.MULTILINE)
|
||||
or "checkpoint = torch.utils.checkpoint.checkpoint" in src
|
||||
)
|
||||
assert has_import, (
|
||||
f"{tag}: transformers.modeling_utils does not import / re-bind "
|
||||
f"torch.utils.checkpoint.checkpoint; unsloth-zoo#549 patch breaks"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_pushtohubmixin_create_repo_status(tag: str):
|
||||
"""unsloth-zoo#393: transformers 5.x removed PushToHubMixin._create_repo.
|
||||
On 4.x present, on 5.x absent. Snapshot which side."""
|
||||
src = fetch_text("huggingface/transformers", tag, "src/transformers/modeling_utils.py")
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: modeling_utils.py missing")
|
||||
# Just record the presence; either is OK as long as we know.
|
||||
has_create = bool(
|
||||
re.search(r"def _create_repo\b", src) or "_create_repo" in src
|
||||
)
|
||||
# Informational only — both branches are tracked.
|
||||
_ = has_create
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# integrations.bitsandbytes — _replace_with_bnb_linear vs new path.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_integrations_bitsandbytes_module_present(tag: str):
|
||||
src = fetch_text(
|
||||
"huggingface/transformers", tag, "src/transformers/integrations/bitsandbytes.py"
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: integrations/bitsandbytes.py missing (legacy layout)")
|
||||
assert "Linear4bit" in src or "linear" in src.lower(), (
|
||||
f"{tag}: integrations/bitsandbytes.py has no Linear4bit reference"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_quantizers_should_convert_module_signature(tag: str):
|
||||
"""unsloth-zoo#491/#488: 5.x moved is_replaceable to
|
||||
quantizers_utils.should_convert_module(full_name, patterns).
|
||||
Snapshot whether function exists and its substring-match form."""
|
||||
src = fetch_text(
|
||||
"huggingface/transformers", tag, "src/transformers/quantizers/quantizers_utils.py"
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: quantizers/quantizers_utils.py missing")
|
||||
if not has_def(src, "should_convert_module", "func"):
|
||||
pytest.skip(f"{tag}: should_convert_module not yet present (4.x)")
|
||||
# The bug we want to catch: substring matching uses `.{key}.` in
|
||||
# `.{full_name}.` form. Patch only fires when this substring is
|
||||
# in source AND mismatch behaviour exists.
|
||||
has_dot_form = ".{key}." in src or "f'.{key}.'" in src or "f\".{key}.\"" in src
|
||||
# Informational only.
|
||||
_ = has_dot_form
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# integrations.finegrained_fp8.FP8Linear — bias/has_bias rename in v5.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_fp8linear_init_param_names(tag: str):
|
||||
"""unsloth-zoo#572: transformers 5.x renamed FP8Linear.__init__
|
||||
`bias` -> `has_bias`. Snapshot which form is in source."""
|
||||
src = fetch_text(
|
||||
"huggingface/transformers",
|
||||
tag,
|
||||
"src/transformers/integrations/finegrained_fp8.py",
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: integrations/finegrained_fp8.py missing")
|
||||
if not has_def(src, "FP8Linear", "class"):
|
||||
pytest.skip(f"{tag}: FP8Linear not yet defined")
|
||||
has_bias_kw = re.search(r"def __init__\([^)]*\bbias\b", src) is not None
|
||||
has_has_bias_kw = re.search(r"def __init__\([^)]*\bhas_bias\b", src) is not None
|
||||
assert has_bias_kw or has_has_bias_kw, (
|
||||
f"{tag}: FP8Linear.__init__ has neither `bias` nor `has_bias` param"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# processing_utils — Unpack importable.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_processing_utils_unpack_importable(tag: str):
|
||||
"""unsloth-zoo#583/584: `from transformers.processing_utils import Unpack`
|
||||
must keep working."""
|
||||
src = fetch_text(
|
||||
"huggingface/transformers", tag, "src/transformers/processing_utils.py"
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: processing_utils.py missing")
|
||||
has_unpack = bool(
|
||||
re.search(r"^Unpack\b\s*=", src, re.MULTILINE)
|
||||
or "Unpack" in src
|
||||
)
|
||||
assert has_unpack, (
|
||||
f"{tag}: transformers.processing_utils.Unpack missing; "
|
||||
f"unsloth-zoo#583/584 import guard breaks"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# Models — gemma3, gpt_oss forward signature drift.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_gemma3_attention_forward_present(tag: str):
|
||||
src = fetch_text(
|
||||
"huggingface/transformers",
|
||||
tag,
|
||||
"src/transformers/models/gemma3/modeling_gemma3.py",
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: modeling_gemma3.py missing")
|
||||
assert has_def(src, "Gemma3Attention", "class"), (
|
||||
f"{tag}: class Gemma3Attention missing"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_gpt_oss_model_forward_present(tag: str):
|
||||
src = fetch_text(
|
||||
"huggingface/transformers",
|
||||
tag,
|
||||
"src/transformers/models/gpt_oss/modeling_gpt_oss.py",
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: modeling_gpt_oss.py missing (legacy)")
|
||||
assert has_def(src, "GptOssModel", "class"), (
|
||||
f"{tag}: class GptOssModel missing"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# auto_factory — unsloth#5155 _LazyAutoMapping private API.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_auto_factory_lazy_mapping_private_api(tag: str):
|
||||
"""unsloth#5155: resolve_model_class iterates private attrs of
|
||||
_LazyAutoMapping (_model_mapping, _config_mapping, _extra_content,
|
||||
_load_attr_from_module). All four must remain."""
|
||||
src = fetch_text(
|
||||
"huggingface/transformers",
|
||||
tag,
|
||||
"src/transformers/models/auto/auto_factory.py",
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: auto/auto_factory.py missing")
|
||||
needed = (
|
||||
"_model_mapping",
|
||||
"_config_mapping",
|
||||
"_extra_content",
|
||||
"_load_attr_from_module",
|
||||
)
|
||||
missing = [n for n in needed if n not in src]
|
||||
assert not missing, (
|
||||
f"{tag}: _LazyAutoMapping private API missing {missing}; "
|
||||
f"unsloth/models/_utils.py:resolve_model_class breaks (unsloth#5155)"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# configuration_utils — PreTrainedConfig vs PretrainedConfig in 5.x.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_configuration_utils_alias(tag: str):
|
||||
"""transformers 5.x renamed PretrainedConfig -> PreTrainedConfig.
|
||||
unsloth-zoo/empty_model.py imports from both paths defensively."""
|
||||
src = fetch_text(
|
||||
"huggingface/transformers",
|
||||
tag,
|
||||
"src/transformers/configuration_utils.py",
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: configuration_utils.py missing")
|
||||
has_old = has_def(src, "PretrainedConfig", "class")
|
||||
has_new = has_def(src, "PreTrainedConfig", "class")
|
||||
assert has_old or has_new, (
|
||||
f"{tag}: neither PretrainedConfig (4.x) nor PreTrainedConfig (5.x) "
|
||||
f"defined in configuration_utils.py"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# tokenization — apply_chat_template return_dict default flip in v5.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_apply_chat_template_signature_present(tag: str):
|
||||
"""unsloth-zoo#572: PreTrainedTokenizerBase.apply_chat_template
|
||||
`return_dict` default flipped False -> True in transformers 5.x.
|
||||
Snapshot which is in source."""
|
||||
src = fetch_text(
|
||||
"huggingface/transformers",
|
||||
tag,
|
||||
"src/transformers/tokenization_utils_base.py",
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: tokenization_utils_base.py missing")
|
||||
assert has_def(src, "apply_chat_template", "func"), (
|
||||
f"{tag}: apply_chat_template missing in tokenization_utils_base.py"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# Generic-importability sweep — every symbol unsloth/zoo imports
|
||||
# from transformers must remain reachable via at least one known path.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_modeling_attn_mask_utils_symbols(tag: str):
|
||||
"""_prepare_4d_attention_mask_for_sdpa is imported by
|
||||
unsloth/models/llama.py + sentence_transformer.py."""
|
||||
src = fetch_text(
|
||||
"huggingface/transformers",
|
||||
tag,
|
||||
"src/transformers/modeling_attn_mask_utils.py",
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: modeling_attn_mask_utils.py missing")
|
||||
assert has_def(src, "AttentionMaskConverter", "class"), (
|
||||
f"{tag}: AttentionMaskConverter missing"
|
||||
)
|
||||
# _prepare_4d_attention_mask_for_sdpa is a function we hard-import.
|
||||
assert (
|
||||
has_def(src, "_prepare_4d_attention_mask_for_sdpa", "func")
|
||||
or "_prepare_4d_attention_mask_for_sdpa" in src
|
||||
), f"{tag}: _prepare_4d_attention_mask_for_sdpa missing"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_cache_utils_classes(tag: str):
|
||||
src = fetch_text("huggingface/transformers", tag, "src/transformers/cache_utils.py")
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: cache_utils.py missing")
|
||||
needed = ("Cache", "DynamicCache")
|
||||
for cls in needed:
|
||||
assert has_def(src, cls, "class"), (
|
||||
f"{tag}: transformers.cache_utils.{cls} missing"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRANSFORMERS_TAGS)
|
||||
def test_training_args_parallel_mode_importable(tag: str):
|
||||
src = fetch_text(
|
||||
"huggingface/transformers", tag, "src/transformers/training_args.py"
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: training_args.py missing")
|
||||
assert "ParallelMode" in src, (
|
||||
f"{tag}: transformers.training_args.ParallelMode missing; "
|
||||
f"unsloth-zoo loss_utils.py:232 ImportError"
|
||||
)
|
||||
|
|
@ -42,24 +42,48 @@ import pytest
|
|||
from tests.version_compat._fetch import fetch_text, first_match, has_def
|
||||
|
||||
|
||||
# Supported window: 0.18.2 -> 0.24.0 (excluding 0.19.0).
|
||||
# Above-cap canaries: 0.25, 0.27, 0.29, 1.0, 1.3 (most recent stable at
|
||||
# the time of writing). `main` is the bleeding edge. Add a row when a
|
||||
# new minor lands; remove a row only when a release is unsupported
|
||||
# AND we have a tracking issue.
|
||||
# Every stable TRL release from 0.18.2 (the pyproject floor) onwards,
|
||||
# plus `main`. Refresh by running:
|
||||
# python -c "import urllib.request,json
|
||||
# from packaging.version import Version
|
||||
# r=json.loads(urllib.request.urlopen('https://pypi.org/pypi/trl/json').read())
|
||||
# v=sorted([Version(x) for x in r['releases'] if r['releases'][x] and not Version(x).is_prerelease and Version(x)>=Version('0.18.2')])
|
||||
# print(*[f'\"v{x}\",' for x in v],sep='\n')"
|
||||
#
|
||||
# 0.19.0 is excluded by pyproject (`!=0.19.0`) — the release was
|
||||
# broken; we keep it in the matrix so we KNOW it's broken (and which
|
||||
# symbols specifically), not just trust the pin.
|
||||
#
|
||||
# Anchors (per the project spec, ALL patches must stay forwards/
|
||||
# backwards compatible with these): 0.22.2, 0.27.1, 1.0.0.
|
||||
TRL_TAGS = [
|
||||
"v0.18.2",
|
||||
"v0.19.0",
|
||||
"v0.19.1",
|
||||
"v0.20.0",
|
||||
"v0.21.0",
|
||||
"v0.22.2",
|
||||
"v0.22.0",
|
||||
"v0.22.1",
|
||||
"v0.22.2", # anchor
|
||||
"v0.23.0",
|
||||
"v0.23.1",
|
||||
"v0.24.0", # current pyproject cap
|
||||
# Above-cap canaries:
|
||||
"v0.25.0",
|
||||
"v0.25.1",
|
||||
"v0.26.0",
|
||||
"v0.26.1",
|
||||
"v0.26.2",
|
||||
"v0.27.0",
|
||||
"v0.27.1", # anchor
|
||||
"v0.27.2",
|
||||
"v0.28.0",
|
||||
"v0.29.0",
|
||||
"v0.29.1",
|
||||
"v1.0.0",
|
||||
"v1.0.0", # anchor
|
||||
"v1.1.0",
|
||||
"v1.2.0",
|
||||
"v1.3.0",
|
||||
"v1.4.0",
|
||||
"main",
|
||||
]
|
||||
|
||||
|
|
@ -174,23 +198,22 @@ def test_trl_trainer_utils_pad(tag: str):
|
|||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_unwrap_model_for_generation_either_path(tag: str):
|
||||
"""unsloth/models/rl.py:152-155 tries
|
||||
`trl.models.utils.unwrap_model_for_generation` first, then
|
||||
`trl.models.unwrap_model_for_generation`. Tests must mirror the
|
||||
prod fallback exactly — checking a third path makes the test
|
||||
laxer than the runtime."""
|
||||
candidates = [
|
||||
"trl/models/utils.py",
|
||||
"trl/models/__init__.py",
|
||||
"trl/extras/profiling.py", # newer TRL versions hide it here
|
||||
]
|
||||
found = False
|
||||
for path in candidates:
|
||||
src = fetch_text("huggingface/trl", tag, path)
|
||||
if src is None:
|
||||
continue
|
||||
if (
|
||||
"def unwrap_model_for_generation" in src
|
||||
or "unwrap_model_for_generation" in src
|
||||
):
|
||||
found = True
|
||||
break
|
||||
assert found, (
|
||||
if "unwrap_model_for_generation" in src:
|
||||
return
|
||||
pytest.fail(
|
||||
f"{tag}: trl.unwrap_model_for_generation not in any known path "
|
||||
f"({candidates}); unsloth/models/rl.py:152-155 will ImportError"
|
||||
)
|
||||
|
|
@ -227,20 +250,24 @@ def test_trl_experimental_openenv_gated(tag: str):
|
|||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_generation_vllm_generation_gated(tag: str):
|
||||
"""unsloth/models/rl_replacements.py:1851-1971 string-rewrites
|
||||
`VLLMGeneration._init_vllm`, `.sync_weights`, and `.generate`. If
|
||||
VLLMGeneration is renamed or any of those three methods disappear,
|
||||
the rewrite silently no-ops and the fast_inference server path
|
||||
breaks at runtime. Gated: skip if the module isn't in this TRL."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/generation/vllm_generation.py")
|
||||
if src is None:
|
||||
# OK: pre-server-mode TRL. unsloth's try/except handles absence.
|
||||
pytest.skip(f"{tag}: trl.generation.vllm_generation not present (OK)")
|
||||
# If present, at least one of these classes/funcs must be there;
|
||||
# unsloth-zoo dispatches via getattr() but the module being empty
|
||||
# means our patch will silently no-op rather than crash.
|
||||
needs_some = ["VLLMClient", "vllm_generate", "VLLM_AVAILABLE", "VLLMServer"]
|
||||
has_some = any(name in src for name in needs_some)
|
||||
assert has_some, (
|
||||
f"{tag}: trl.generation.vllm_generation exists but none of "
|
||||
f"{needs_some} present; unsloth-zoo's dispatch in vllm_utils "
|
||||
f"will silently no-op the server path"
|
||||
assert has_def(src, "VLLMGeneration", "class"), (
|
||||
f"{tag}: class VLLMGeneration missing; unsloth-zoo dispatch "
|
||||
f"in models/rl_replacements.py:1852 will silently no-op"
|
||||
)
|
||||
for method in ("_init_vllm", "sync_weights", "generate"):
|
||||
assert has_def(src, method, "func"), (
|
||||
f"{tag}: VLLMGeneration.{method} missing; "
|
||||
f"unsloth/models/rl_replacements.py rewrites this method body"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
|
|
@ -254,17 +281,16 @@ def test_trl_generation_vllm_generation_gated(tag: str):
|
|||
def test_trl_version_parseable(tag: str):
|
||||
src = fetch_text("huggingface/trl", tag, "trl/__init__.py")
|
||||
assert src is not None
|
||||
# Recognised mechanisms:
|
||||
# 1. literal `__version__ = "x.y.z"`
|
||||
# Recognised mechanisms (any one is sufficient):
|
||||
# 1. literal `__version__ = "x.y.z"` at module scope
|
||||
# 2. `from .version import __version__`
|
||||
# 3. `__version__ = version("trl")` from `importlib.metadata`
|
||||
# (bare `version` symbol must be imported on a line above)
|
||||
# 3. `__version__ = version("trl")` via importlib.metadata
|
||||
# 4. `__version__ = f.read().strip()` (TRL 0.22.x reads from a
|
||||
# sibling VERSION file)
|
||||
has_literal = bool(re.search(r'^__version__\s*=\s*["\']', src, re.MULTILINE))
|
||||
has_subimport = bool(
|
||||
re.search(r"^from\s+\.version\s+import\s+__version__", src, re.MULTILINE)
|
||||
)
|
||||
# Importlib metadata path: any line `from importlib.metadata import ... version ...`
|
||||
# plus a `__version__ = version(` assignment somewhere below.
|
||||
has_metadata = bool(
|
||||
re.search(
|
||||
r"^from\s+importlib\.metadata\s+import\s+(?:[\w,\s]+,\s*)?version",
|
||||
|
|
@ -273,7 +299,387 @@ def test_trl_version_parseable(tag: str):
|
|||
)
|
||||
and re.search(r"^\s*__version__\s*=\s*version\s*\(", src, re.MULTILINE)
|
||||
)
|
||||
assert has_literal or has_subimport or has_metadata, (
|
||||
has_version_file = bool(
|
||||
re.search(r"^\s*__version__\s*=\s*f\.read\s*\(", src, re.MULTILINE)
|
||||
or re.search(r"^\s*__version__\s*=\s*open\s*\(", src, re.MULTILINE)
|
||||
)
|
||||
assert has_literal or has_subimport or has_metadata or has_version_file, (
|
||||
f"{tag}: trl.__version__ not exported via any known mechanism; "
|
||||
f"unsloth/models/rl.py:63 will AttributeError"
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# Coverage extension (added 2026-05): symbols / source-string contracts
|
||||
# unsloth + unsloth-zoo touch but the original suite missed.
|
||||
# =========================================================================
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 1. trl.is_conversational — soft import in unsloth-zoo dataset_utils.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_is_conversational_export(tag: str):
|
||||
src = fetch_text("huggingface/trl", tag, "trl/__init__.py")
|
||||
assert src is not None
|
||||
if "is_conversational" not in src:
|
||||
# Some old TRLs omit it; gated soft import in unsloth-zoo
|
||||
# falls back to a local impl. OK.
|
||||
pytest.skip(f"{tag}: trl.is_conversational not exported (legacy TRL)")
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 2-4. trl.trainer.sft_trainer module surface used by unsloth tokenizer
|
||||
# utils + tests.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_sft_trainer_module_internals(tag: str):
|
||||
"""unsloth/tokenizer_utils.py:1538 does `from trl.trainer.sft_trainer
|
||||
import *`. The symbols below must exist for the wildcard import +
|
||||
eval-discovery to keep working."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/sft_trainer.py")
|
||||
assert src is not None, (
|
||||
f"{tag}: trl/trainer/sft_trainer.py missing; "
|
||||
f"unsloth/tokenizer_utils.py:1538 wildcard import fails"
|
||||
)
|
||||
assert has_def(src, "SFTTrainer", "class"), (
|
||||
f"{tag}: class SFTTrainer missing in sft_trainer.py"
|
||||
)
|
||||
# neftune_post_forward_hook: optional (TRL removed it in some
|
||||
# versions); soft-imported in tokenizer_utils.py:1542. Don't fail.
|
||||
if "neftune_post_forward_hook" not in src:
|
||||
pass
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 5-6. trl.trainer.dpo_trainer module + MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES
|
||||
# — patched by unsloth-zoo/temporary_patches/misc.py:1376-1379.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_dpo_trainer_module_exists(tag: str):
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/dpo_trainer.py")
|
||||
assert src is not None, (
|
||||
f"{tag}: trl/trainer/dpo_trainer.py missing; "
|
||||
f"unsloth-zoo/temporary_patches/misc.py:1376 import fails"
|
||||
)
|
||||
assert has_def(src, "DPOTrainer", "class"), (
|
||||
f"{tag}: class DPOTrainer missing in dpo_trainer.py"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 7. trl.trainer.utils.ConstantLengthDataset — soft import in
|
||||
# unsloth-zoo/dataset_utils.py:596. Optional (TRL 0.20.0 removed it
|
||||
# on some paths).
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_constant_length_dataset_optional(tag: str):
|
||||
candidates = [
|
||||
"trl/trainer/utils.py",
|
||||
"trl/trainer/utils/__init__.py",
|
||||
]
|
||||
hit = first_match("huggingface/trl", tag, candidates)
|
||||
if hit is None:
|
||||
pytest.skip(f"{tag}: trl/trainer/utils not present")
|
||||
_, src = hit
|
||||
if "ConstantLengthDataset" not in src:
|
||||
pytest.skip(
|
||||
f"{tag}: ConstantLengthDataset removed; unsloth-zoo soft "
|
||||
f"import handles this"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 8. trl.models.utils.disable_gradient_checkpointing — added in TRL
|
||||
# 1.0.0+. unsloth/models/rl.py:1976-1994 uses hasattr() for gating;
|
||||
# we still want the assertion that the symbol exists from 1.0.0
|
||||
# onwards so a future removal gets caught.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_models_utils_disable_gradient_checkpointing(tag: str):
|
||||
if tag == "main":
|
||||
# main is bleeding edge; expect symbol to track 1.0.0+ behaviour.
|
||||
require = True
|
||||
else:
|
||||
# Strip leading 'v' and parse.
|
||||
try:
|
||||
from packaging.version import Version
|
||||
require = Version(tag.lstrip("v")) >= Version("1.0.0")
|
||||
except Exception:
|
||||
require = False
|
||||
src = fetch_text("huggingface/trl", tag, "trl/models/utils.py")
|
||||
if src is None:
|
||||
if require:
|
||||
pytest.fail(f"{tag}: trl/models/utils.py missing on 1.0.0+")
|
||||
pytest.skip(f"{tag}: trl/models/utils.py missing (legacy TRL)")
|
||||
has_it = has_def(src, "disable_gradient_checkpointing", "func")
|
||||
if require:
|
||||
assert has_it, (
|
||||
f"{tag}: trl.models.utils.disable_gradient_checkpointing "
|
||||
f"missing on TRL >=1.0.0; unsloth/models/rl.py:1979 patch silent no-op"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 9. trl.import_utils + the `_*_available` cache pattern — used by
|
||||
# unsloth/import_fixes.py:508-516 to clear cached `is_X_available`
|
||||
# booleans so vllm-ascend imports work.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_import_utils_available_pattern(tag: str):
|
||||
candidates = [
|
||||
"trl/import_utils.py",
|
||||
"trl/import_utils/__init__.py",
|
||||
]
|
||||
hit = first_match("huggingface/trl", tag, candidates)
|
||||
if hit is None:
|
||||
pytest.skip(f"{tag}: trl/import_utils not present (legacy TRL)")
|
||||
_, src = hit
|
||||
# The patch iterates `vars(trl.import_utils)` looking for any name
|
||||
# ending in `_available`. At least one such cache var must exist or
|
||||
# the patch silently no-ops.
|
||||
has_pattern = bool(re.search(r"\b\w+_available\b", src))
|
||||
assert has_pattern, (
|
||||
f"{tag}: trl.import_utils has no `_available` cache var; "
|
||||
f"unsloth/import_fixes.py:508-516 silently no-ops"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 10. trl.experimental.openenv.utils generators — at least one of the
|
||||
# two function names must exist (unsloth/models/rl_replacements.py
|
||||
# :1775-1781 calls getattr() to find one).
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_openenv_utils_generators(tag: str):
|
||||
src = fetch_text(
|
||||
"huggingface/trl", tag, "trl/experimental/openenv/utils.py"
|
||||
)
|
||||
if src is None:
|
||||
pytest.skip(f"{tag}: openenv.utils not present (gated optional)")
|
||||
legacy = "generate_rollout_completions" in src
|
||||
new = "_generate_rollout_completions_colocate" in src
|
||||
assert legacy or new, (
|
||||
f"{tag}: openenv.utils has neither `generate_rollout_completions` "
|
||||
f"nor `_generate_rollout_completions_colocate`; "
|
||||
f"unsloth/models/rl_replacements.py:1775-1781 patch breaks"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 11-16. GRPOTrainer required method names. unsloth/models/rl_replacements
|
||||
# .py uses function_name == "..." dispatch keys; if a method is
|
||||
# renamed, the patch silently doesn't apply. List of methods is
|
||||
# the precise dispatch key set.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_grpo_trainer_required_methods(tag: str):
|
||||
"""Method names unsloth string-rewrites against. Drift here
|
||||
silently skips the rewrite. _get_per_token_logps was renamed to
|
||||
_get_per_token_logps_and_entropies in TRL 0.20+; either is fine
|
||||
since unsloth dispatches by function_name."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/grpo_trainer.py")
|
||||
assert src is not None
|
||||
# _prepare_inputs / _generate_and_score_completions / compute_loss
|
||||
# are stable across the entire support window.
|
||||
for m in ("_prepare_inputs", "_generate_and_score_completions", "compute_loss"):
|
||||
assert has_def(src, m, "func"), (
|
||||
f"{tag}: GRPOTrainer.{m} missing; "
|
||||
f"unsloth/models/rl_replacements.py dispatch by name silently skips"
|
||||
)
|
||||
# Per-token-logps surface: ONE of the two names must exist.
|
||||
has_legacy = has_def(src, "_get_per_token_logps", "func")
|
||||
has_new = has_def(src, "_get_per_token_logps_and_entropies", "func")
|
||||
assert has_legacy or has_new, (
|
||||
f"{tag}: neither GRPOTrainer._get_per_token_logps (TRL <=0.19) nor "
|
||||
f"._get_per_token_logps_and_entropies (TRL >=0.20) found; "
|
||||
f"unsloth's per-token-logps rewrite no-ops on both dispatch keys"
|
||||
)
|
||||
# Optional / version-dependent — never fail, just informational
|
||||
for m in ("_generate_single_turn", "_move_model_to_vllm", "_calculate_rewards"):
|
||||
_present = has_def(src, m, "func")
|
||||
_ = _present
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Source-string contracts on trl/trainer/grpo_trainer.py. Each substring
|
||||
# is one half of a `function.replace(old, new)` rewrite — if the
|
||||
# substring no longer appears in TRL source, the rewrite is a no-op
|
||||
# AND the user-facing GRPO behaviour silently diverges.
|
||||
#
|
||||
# Broken into per-version-window tests because some patterns only apply
|
||||
# to a subset of TRL minors.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_grpo_source_inference_mode_unwrap(tag: str):
|
||||
"""rl_replacements.py:526-535 inserts an autocast block immediately
|
||||
AFTER `with torch.inference_mode():` and `self.accelerator.unwrap_model
|
||||
(self.model)`. Both substrings must appear in `_prepare_inputs`."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/grpo_trainer.py")
|
||||
assert src is not None
|
||||
has_inference_mode = "torch.inference_mode" in src
|
||||
has_unwrap = "self.accelerator.unwrap_model" in src
|
||||
assert has_inference_mode and has_unwrap, (
|
||||
f"{tag}: GRPOTrainer source missing torch.inference_mode={has_inference_mode} "
|
||||
f"or self.accelerator.unwrap_model={has_unwrap}; "
|
||||
f"unsloth/models/rl_replacements.py:526 autocast insertion no-ops"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 17. KTOTrainer.get_batch_logps + the literal raise message rewriter
|
||||
# hits.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_kto_get_batch_logps_signature(tag: str):
|
||||
"""TRL 0.27+ moved KTOTrainer to trl.experimental.kto and the
|
||||
canonical kto_trainer.py shrank to a thin re-export wrapper. The
|
||||
real `get_batch_logps` lives at trl/experimental/kto/kto_trainer.py.
|
||||
Unsloth's MRO walk in models/rl.py:592-708 already follows
|
||||
trl.experimental.* parents, so either path is fine — we just
|
||||
require the symbol to exist SOMEWHERE."""
|
||||
candidates = [
|
||||
"trl/trainer/kto_trainer.py",
|
||||
"trl/experimental/kto/kto_trainer.py",
|
||||
"trl/experimental/kto/__init__.py",
|
||||
]
|
||||
for path in candidates:
|
||||
src = fetch_text("huggingface/trl", tag, path)
|
||||
if src is None:
|
||||
continue
|
||||
if has_def(src, "get_batch_logps", "func"):
|
||||
return
|
||||
pytest.fail(
|
||||
f"{tag}: KTOTrainer.get_batch_logps not found in any of {candidates}; "
|
||||
f"unsloth/models/rl_replacements.py:1675 rewrite silently skipped"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 18. SFTTrainer.__init__ literal `dict_args.pop("push_to_hub_token")`
|
||||
# OR our shim must short-circuit. transformers 5.0 removed this
|
||||
# kwarg; if TRL stops emitting the bare pop, our patch becomes
|
||||
# a no-op AND TRL itself crashes on transformers 5.0.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_sft_trainer_class(tag: str):
|
||||
"""Sanity: SFTTrainer.__init__ exists. The
|
||||
`dict_args.pop("push_to_hub_token")` literal substring is checked
|
||||
only when present — its absence means TRL already adapted (e.g.
|
||||
via `dict_args.pop("push_to_hub_token", None)` with a default),
|
||||
which is also fine."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/sft_trainer.py")
|
||||
assert src is not None
|
||||
assert has_def(src, "SFTTrainer", "class"), (
|
||||
f"{tag}: class SFTTrainer missing"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 19-21. DPOTrainer methods unsloth-zoo's rl_replacements rewrites.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_dpo_trainer_methods(tag: str):
|
||||
"""DPOTrainer method-name surface unsloth's rewriters key on
|
||||
(rl_replacements.py:222-394). All four are version-windowed:
|
||||
- concatenated_inputs / concatenated_forward existed on
|
||||
DPOTrainer through TRL 0.29.x; TRL 1.0+ refactored these into
|
||||
free functions (concatenation moved out of the class).
|
||||
- _compute_loss_liger added ~TRL 0.20.
|
||||
- _set_signature_columns_if_needed: usually inherited from
|
||||
transformers.Trainer, may or may not be re-defined locally.
|
||||
None are STRICTLY required — when missing the matching unsloth
|
||||
rewriter cleanly no-ops (TRL itself does the work). We surface
|
||||
presence/absence as informational so a regression that
|
||||
SILENTLY drops one is at least visible in the test log."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/dpo_trainer.py")
|
||||
assert src is not None
|
||||
# The DPO class itself must always exist.
|
||||
assert has_def(src, "DPOTrainer", "class"), (
|
||||
f"{tag}: class DPOTrainer missing in dpo_trainer.py"
|
||||
)
|
||||
# Informational only -- pass either way:
|
||||
for method in (
|
||||
"concatenated_inputs",
|
||||
"concatenated_forward",
|
||||
"_compute_loss_liger",
|
||||
"_set_signature_columns_if_needed",
|
||||
"_prepare_dataset",
|
||||
):
|
||||
_present = has_def(src, method, "func")
|
||||
_ = _present # informational; rewriter no-ops cleanly when absent
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# 22-23. trl.trainer.grpo_trainer must IMPORT or DEFINE the helpers
|
||||
# unsloth's source rewriters reference: profiling_context,
|
||||
# maybe_apply_chat_template, truncate_with_protected_tokens.
|
||||
# Either the symbol is locally defined OR imported from elsewhere
|
||||
# in trl.* — the rewriter only needs the NAME to be in scope at
|
||||
# the call site.
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_grpo_internal_helpers_in_scope(tag: str):
|
||||
"""Chat-template propagation is what unsloth's
|
||||
grpo_trainer_fix_maybe_apply_chat_template wires up so user-supplied
|
||||
`reasoning_effort` etc. survives the GRPO compile cell. The exact
|
||||
helper name moved across releases:
|
||||
- TRL <=0.24: `maybe_apply_chat_template(example, processing_class)`
|
||||
appeared as a literal in grpo_trainer.py — unsloth's regex
|
||||
rewriter substitutes it with a kwargs-aware version.
|
||||
- TRL >=0.25: TRL itself uses `apply_chat_template` and pipes
|
||||
`**self.chat_template_kwargs`, so the unsloth rewriter is a
|
||||
cleanly-no-op'd dead path on those versions (correct behaviour).
|
||||
Either pattern means the chat-template path is wired SOMEWHERE."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/grpo_trainer.py")
|
||||
assert src is not None
|
||||
legacy = "maybe_apply_chat_template" in src
|
||||
successor = "chat_template_kwargs" in src or "apply_chat_template" in src
|
||||
assert legacy or successor, (
|
||||
f"{tag}: GRPOTrainer source does NOT propagate chat-template kwargs "
|
||||
f"via legacy `maybe_apply_chat_template` OR successor "
|
||||
f"`apply_chat_template(... **chat_template_kwargs)`; "
|
||||
f"unsloth/models/rl_replacements.py:909-927 rewrite no-ops AND "
|
||||
f"native TRL doesn't carry the kwargs either — likely real bug"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tag", TRL_TAGS)
|
||||
def test_trl_truncate_with_protected_tokens_optional(tag: str):
|
||||
"""Some TRL versions (0.22.2-0.23.1 specifically) ship
|
||||
`truncate_with_protected_tokens`. Newer versions removed it.
|
||||
rl_replacements.py:712 has a regex that handles both presence
|
||||
and absence — but if the symbol is renamed without removal,
|
||||
we need to know."""
|
||||
src = fetch_text("huggingface/trl", tag, "trl/trainer/grpo_trainer.py")
|
||||
assert src is not None
|
||||
# No assertion — informational only. We just want to NOT silently
|
||||
# drift.
|
||||
has_it = "truncate_with_protected_tokens" in src
|
||||
_ = has_it # informational; pass either way.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue