ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ci: add cross-version compat canary for vLLM, TRL, PEFT, ST, bnb

Browse source 
Catches upstream API drift early — before a PyPI release breaks user
workloads. For each tracked package + version, fetch the relevant
source files from raw.githubusercontent.com and grep for the symbols
unsloth + unsloth-zoo monkey-patch, subclass, or eval-import. No pip
install required, CPU-only, runs PR-time + daily cron.

Files:
- tests/vllm_compat/test_vllm_pinned_symbols.py
    extend VLLM_TAGS from {0.9.0..0.15.0} to include
    {0.16.0, 0.17.1, 0.18.1, 0.19.1, 0.20.1, main}.
- tests/version_compat/_fetch.py
    shared fetch + grep helpers (fetch_text / has_def / first_match).
- tests/version_compat/test_trl_grpo_pinned_symbols.py
    12 TRL tags (0.18.2 -> v1.3.0 + main) covering the supported
    window (pyproject pin trl>=0.18.2,!=0.19.0,<=0.24.0) plus
    above-cap canaries. Asserts:
      * top-level GRPOTrainer / GRPOConfig / SFTTrainer / SFTConfig
        re-exports (used by `from trl import X`)
      * trl.trainer.grpo_trainer.GRPOTrainer class
      * trl.trainer.grpo_config.GRPOConfig (or grpo_trainer.py fallback)
      * DataCollatorForPreference reachable from EITHER dpo_trainer or
        utils (rl_replacements.py:318 string-emits the dpo_trainer path)
      * trl.trainer.utils.pad (rl_replacements.py:326)
      * unwrap_model_for_generation in any known submodule
        (rl.py:152-155 try/except handles both)
      * trl.experimental.openenv (gated; rl_replacements.py:1765-1770)
      * trl.generation.vllm_generation (gated; rl_replacements.py:1846)
      * trl.__version__ exported via literal / submodule / metadata
- tests/version_compat/test_peft_pinned_symbols.py
    5 PEFT tags (0.18.0 -> 0.19.1 + main). Asserts:
      * top-level LoraConfig / get_peft_model / PeftModel
      * peft.tuners.lora.LoraConfig at canonical path
      * get_peft_model in mapping.py / mapping_func.py
        (peft 0.18 split this out)
      * peft.tuners.lora.LoraLayer
      * peft.tuners.lora.bnb (Linear4bit / Linear8bitLt)
- tests/version_compat/test_sentence_transformers_pinned_symbols.py
    6 ST tags (5.0.0 -> 5.4.1 + main). Handles BOTH layouts:
      legacy (< 5.4): sentence_transformers/models[.py|/__init__.py]
      modular (>= 5.4): classes under
        sentence_transformers/base/modules/*
        sentence_transformers/sentence_transformer/modules/*
      Plus verifies the deprecated-import shim
      (`setup_deprecated_module_imports`) is wired in __init__.py
      so `from sentence_transformers.models import Pooling` keeps
      working for unsloth/models/sentence_transformer.py.
- tests/version_compat/test_bitsandbytes_pinned_symbols.py
    4 bnb tags (0.45.5 -> 0.49.2 + main; skip the broken 0.46.0 /
    0.48.0 listed in pyproject !=). Asserts:
      * bnb.functional.{dequantize_4bit, quantize_4bit}
      * bnb.nn.{Linear4bit, Params4bit}
- .github/workflows/version-compat-ci.yml
    7 jobs:
      * vllm-pinned-symbols  (existing tests/vllm_compat/, now wired)
      * trl-grpo-pinned-symbols
      * peft-pinned-symbols
      * st-pinned-symbols
      * bitsandbytes-pinned-symbols
      * zoo-imports-under-spoof  (real pip install + CUDA spoof,
        unsloth_zoo.{rl_replacements, empty_model, vllm_utils,
        vllm_lora_*} import smoke)
      * daily-fresh-fetch (cron-only superset)
    Triggers: pull_request (paths), daily 06:43 UTC, workflow_dispatch.
    Authenticated GitHub raw fetches (GITHUB_TOKEN) for the 5000 req/h
    quota.

Smoke-tested locally: 226 pass, 15 skipped (gated optional features).
This commit is contained in:
Daniel Han 2026-05-08 11:58:11 +00:00
commit 6ec6ca39ef
8 changed files with 1088 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

237
.github/workflows/version-compat-ci.yml vendored Normal file
View file

@ -0,0 +1,237 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
#
# Cross-version compat canary for the four upstream packages whose
# release cadence regularly breaks unsloth + unsloth-zoo:
#
# 1. vLLM (LoRA worker manager, BnB loader, cumem allocator)
# 2. TRL / GRPO (trainer source rewriters in unsloth.models.rl*)
# 3. PEFT (LoraConfig, get_peft_model, LoraLayer, bnb integration)
# 4. sentence-transformers (Transformer/Pooling/Normalize, Trainer)
# 5. bitsandbytes (Linear4bit, dequantize_4bit)
#
# Strategy: GitHub raw-fetch + symbol grep against every tracked
# version (no pip install, CPU-only). When upstream renames a symbol
# we depend on, the matching test fails BEFORE a user hits it. The
# `main` branch entries give us a few-day lead on PyPI releases.
#
# Cross-references:
# tests/vllm_compat/test_vllm_pinned_symbols.py (vLLM symbols)
# tests/version_compat/test_trl_grpo_pinned_symbols.py
# tests/version_compat/test_peft_pinned_symbols.py
# tests/version_compat/test_sentence_transformers_pinned_symbols.py
# tests/version_compat/test_bitsandbytes_pinned_symbols.py
name: Version Compat CI
on:
pull_request:
paths:
- 'unsloth/models/rl.py'
- 'unsloth/models/rl_replacements.py'
- 'unsloth/models/sentence_transformer.py'
- 'tests/vllm_compat/**'
- 'tests/version_compat/**'
- 'pyproject.toml'
- '.github/workflows/version-compat-ci.yml'
schedule:
# Daily 06:43 UTC. Catches upstream PyPI releases roughly within
# 24 h. Off the :00 / :30 fleet-collision spots.
- cron: '43 6 * * *'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
vllm-pinned-symbols:
name: vLLM pinned-symbol matrix (≥ 0.9.0 + main)
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install pytest only
# The test fetches from raw.githubusercontent.com and greps
# source. No pip install of vllm / torch / transformers is
# needed — that's the whole point of this canary.
run: |
python -m pip install --upgrade pip
pip install 'pytest>=8'
- name: Run vllm-compat suite
env:
# Authenticated requests get a 5000-req/h quota on raw
# fetches; unauthenticated is 60/h and trips on the matrix.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
python -m pytest tests/vllm_compat/test_vllm_pinned_symbols.py -v --tb=short
trl-grpo-pinned-symbols:
name: TRL / GRPO pinned-symbol matrix
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install pytest only
run: |
python -m pip install --upgrade pip
pip install 'pytest>=8'
- name: Run trl-compat suite
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# PYTHONPATH=. so `from tests.version_compat._fetch import …`
# works without an editable install of unsloth itself.
PYTHONPATH=. python -m pytest \
tests/version_compat/test_trl_grpo_pinned_symbols.py \
-v --tb=short
peft-pinned-symbols:
name: PEFT pinned-symbol matrix (pyproject window + main)
runs-on: ubuntu-latest
timeout-minutes: 8
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install pytest only
run: |
python -m pip install --upgrade pip
pip install 'pytest>=8'
- name: Run peft-compat suite
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PYTHONPATH=. python -m pytest \
tests/version_compat/test_peft_pinned_symbols.py \
-v --tb=short
st-pinned-symbols:
name: sentence-transformers pinned-symbol matrix
runs-on: ubuntu-latest
timeout-minutes: 8
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install pytest only
run: |
python -m pip install --upgrade pip
pip install 'pytest>=8'
- name: Run sentence-transformers compat suite
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PYTHONPATH=. python -m pytest \
tests/version_compat/test_sentence_transformers_pinned_symbols.py \
-v --tb=short
bitsandbytes-pinned-symbols:
name: bitsandbytes pinned-symbol matrix
runs-on: ubuntu-latest
timeout-minutes: 8
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install pytest only
run: |
python -m pip install --upgrade pip
pip install 'pytest>=8'
- name: Run bitsandbytes compat suite
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PYTHONPATH=. python -m pytest \
tests/version_compat/test_bitsandbytes_pinned_symbols.py \
-v --tb=short
# Optional second layer: actually `pip install` ONE representative
# version of each package and verify unsloth + unsloth-zoo modules
# import on it under the existing CUDA spoof. CPU-only, runs on
# ubuntu-latest. Catches the small set of breakages that the static
# symbol check misses (e.g. import-time side effects).
zoo-imports-under-spoof:
name: unsloth_zoo vllm/grpo/peft/st modules import under CUDA spoof
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with: { path: unsloth }
- name: Clone unsloth-zoo @ main
run: |
git clone --depth=1 https://github.com/unslothai/unsloth-zoo \
"$RUNNER_TEMP/unsloth-zoo"
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install CPU torch + supported pkg pins
run: |
python -m pip install --upgrade pip
# CPU torch (vllm/peft/st all depend on it).
pip install --index-url https://download.pytorch.org/whl/cpu \
'torch>=2.4,<2.11' 'torchvision<0.26'
# Ladder of supported floor versions per pyproject.toml.
pip install \
'transformers>=4.56,<5.6' 'trl>=0.22,<0.26' \
'peft>=0.18.0' 'sentence-transformers>=5.0' \
'accelerate>=1.0' 'datasets>=3.4,<5' \
'bitsandbytes>=0.45.5' \
sentencepiece protobuf safetensors numpy 'pytest>=8' \
'huggingface_hub>=0.34' tqdm packaging psutil triton Pillow
# Editable-install both repos so the test imports the
# checkouts (not whatever stale PyPI version pip resolved).
pip install --no-deps -e "$RUNNER_TEMP/unsloth-zoo"
pip install --no-deps -e ./unsloth
- name: Run vllm_compat zoo-imports tests under spoof
env:
UNSLOTH_IS_PRESENT: '1'
UNSLOTH_COMPILE_DISABLE: '1'
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: python
run: |
cd unsloth
PYTHONPATH=. python -m pytest \
tests/vllm_compat/test_unsloth_zoo_imports.py \
-v --tb=short
# Daily-only: same suites but with --strict on importable upstream
# tags. Schedule-only so PR jobs stay fast; cron tolerates a flake.
daily-fresh-fetch:
name: daily fresh-fetch sweep (cron only)
if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Install pytest
run: pip install 'pytest>=8'
- name: Run all version-compat suites in one process (no cache)
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PYTHONPATH=. python -m pytest \
tests/vllm_compat/test_vllm_pinned_symbols.py \
tests/version_compat/ \
-v --tb=short

0
tests/version_compat/__init__.py Normal file
View file

72
tests/version_compat/_fetch.py Normal file
View file

@ -0,0 +1,72 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team.
"""Shared helpers for the version-compat suites: fetch a file from
GitHub raw at a specific tag/branch, and grep for class / def / module
symbols without ast.parse so a single non-importable line doesn't
false-fail us. Mirrors tests/vllm_compat/test_vllm_pinned_symbols.py.
Used by:
- tests/version_compat/test_trl_grpo_pinned_symbols.py
- tests/version_compat/test_peft_pinned_symbols.py
- tests/version_compat/test_sentence_transformers_pinned_symbols.py
- tests/version_compat/test_bitsandbytes_pinned_symbols.py
"""
from __future__ import annotations
import os
import re
import urllib.error
import urllib.request
import pytest
def fetch_text(repo: str, ref: str, path: str) -> str | None:
"""Fetch a file from GitHub raw. None on 404 (the path was renamed
or removed in this version, which is informational and the caller
decides whether that's fatal). Skips the test on transient network
errors so we don't make CI flaky."""
url = f"https://raw.githubusercontent.com/{repo}/{ref}/{path}"
req = urllib.request.Request(url)
token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN")
if token:
req.add_header("Authorization", f"Bearer {token}")
try:
with urllib.request.urlopen(req, timeout = 15) as r:
return r.read().decode("utf-8", errors = "replace")
except urllib.error.HTTPError as e:
if e.code == 404:
return None
pytest.skip(f"GitHub fetch failed ({e.code}) for {url}")
except (urllib.error.URLError, TimeoutError) as e:
pytest.skip(f"GitHub fetch failed ({e}) for {url}")
def has_def(src: str, name: str, kind: str = "any") -> bool:
"""Heuristic AST-equivalent grep for `class Name`, `def name`,
or `Name = ...` at module scope. We avoid a full ast.parse so a
single non-importable line (e.g. `# type: ignore` after an
unresolved alias) doesn't false-fail us."""
if kind in ("any", "class") and re.search(
rf"^class\s+{re.escape(name)}\b", src, re.MULTILINE
):
return True
if kind in ("any", "func") and re.search(
rf"^(?:async\s+)?def\s+{re.escape(name)}\b", src, re.MULTILINE
):
return True
if kind == "any" and re.search(rf"^{re.escape(name)}\s*[:=]", src, re.MULTILINE):
return True
return False
def first_match(repo: str, ref: str, paths: list[str]) -> tuple[str, str] | None:
"""Try a list of candidate paths; return (path, src) for the first
one that exists, or None if none do. Useful when upstream split or
moved a module across versions."""
for p in paths:
src = fetch_text(repo, ref, p)
if src is not None:
return (p, src)
return None

92
tests/version_compat/test_bitsandbytes_pinned_symbols.py Normal file
View file

@ -0,0 +1,92 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team.
"""Pinned-symbol compat check across bitsandbytes PyPI minor versions
unsloth + unsloth-zoo target. Catches API drift like:
- bnb 0.46.0 release was broken (in pyproject.toml as `!=0.46.0`).
Don't test against it.
- bnb 0.48.0 release was broken (also `!=0.48.0`). Same.
- bnb 0.45 series introduced fp4 + nf4 paged optimisers; unsloth-zoo
expects bnb.functional.dequantize_4bit + bnb.nn.Linear4bit /
Params4bit to remain stable from this point onward.
- vLLM bitsandbytes-loader patches in unsloth_zoo/vllm_utils.py:
apply_bnb_4bit (line 237), is_layer_skipped_bnb (line 281),
BitsAndBytesLinearMethod._apply_4bit_weight (line 282) — these
live in vllm.* but they call into bnb's public surface.
Strategy: GitHub raw fetch + symbol grep. CPU-only, no install.
"""
from __future__ import annotations
import pytest
from tests.version_compat._fetch import fetch_text, has_def, first_match
# pyproject pin: bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0
# Test floor + each safe minor since.
BNB_TAGS = [
"0.45.5",
"0.47.0", # skip 0.46.0 (broken)
"0.49.2", # skip 0.48.0 (broken)
"main",
]
# -------------------------------------------------------------------------
# bnb.functional: dequantize_4bit / quantize_4bit are the public 4-bit
# surface unsloth's compiled kernels and unsloth-zoo's vllm_utils
# bnb-loader patches all call into.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", BNB_TAGS)
def test_bnb_functional_4bit(tag: str):
candidates = [
"bitsandbytes/functional.py",
"bitsandbytes/functional/__init__.py",
]
hit = first_match(
"bitsandbytes-foundation/bitsandbytes", tag, candidates
)
assert hit is not None, (
f"{tag}: bitsandbytes/functional[.py|/__init__.py] both missing"
)
_, src = hit
needed = ("dequantize_4bit", "quantize_4bit")
missing = [n for n in needed if not has_def(src, n, "func") and n not in src]
assert not missing, (
f"{tag}: bnb.functional missing {missing}; "
f"unsloth-zoo dequant kernels rely on these"
)
# -------------------------------------------------------------------------
# bnb.nn.Linear4bit / Params4bit: the two classes peft and unsloth
# isinstance-check against. Renaming either silently breaks 4-bit LoRA.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", BNB_TAGS)
def test_bnb_nn_linear4bit_classes(tag: str):
candidates = [
"bitsandbytes/nn/modules.py",
"bitsandbytes/nn/__init__.py",
]
found_linear = False
found_params = False
for p in candidates:
src = fetch_text("bitsandbytes-foundation/bitsandbytes", tag, p)
if src is None:
continue
if has_def(src, "Linear4bit", "class") or "Linear4bit" in src:
found_linear = True
if has_def(src, "Params4bit", "class") or "Params4bit" in src:
found_params = True
if found_linear and found_params:
return
pytest.fail(
f"{tag}: Linear4bit={found_linear} Params4bit={found_params} "
f"in {candidates}; unsloth + peft 4-bit isinstance checks fail"
)

179
tests/version_compat/test_peft_pinned_symbols.py Normal file
View file

@ -0,0 +1,179 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team.
"""Pinned-symbol compat check across PEFT PyPI minor versions
unsloth + unsloth-zoo target. Catches API drift like:
- peft 0.18 finalised the LoraConfig public surface (+ MoE-aware
target_modules); unsloth uses target_modules + r + lora_alpha +
lora_dropout + bias.
- peft 0.19 introduced the LoraConfig.target_parameters extension;
unsloth-zoo's MoE LoRA extractor in saving_utils.py reads it via
getattr() so missing on older versions is OK but the attribute
shape must remain stable on >= 0.19.
- peft.tuners.lora package layout: LoraLayer / LoraConfig / Linear4bit
re-exports must keep working under both `from peft import X` and
`from peft.tuners.lora import X`.
Strategy: for each tracked PEFT tag, fetch source from
github.com/huggingface/peft (no pip install needed) and assert that
every symbol unsloth + unsloth-zoo's PEFT touchpoints depend on is
present.
Versioning policy: cover the supported window declared in
unsloth/pyproject.toml (`peft>=0.18.0,!=0.11.0`) plus `main`. The
`!=0.11.0` exclusion is for the historical broken release; we don't
test against it.
"""
from __future__ import annotations
import pytest
from tests.version_compat._fetch import fetch_text, has_def
# pyproject pin: peft>=0.18.0. Test the floor + each minor since.
# `main` catches breakage before a release lands.
PEFT_TAGS = [
"v0.18.0",
"v0.18.1",
"v0.19.0",
"v0.19.1",
"main",
]
# -------------------------------------------------------------------------
# Top-level public re-exports. unsloth/models/sentence_transformer.py:1948
# does `from peft import LoraConfig, get_peft_model as peft_get_peft_model`.
# unsloth_zoo's saving_utils + lora extractors hit `peft.PeftModel`.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", PEFT_TAGS)
def test_peft_top_level_exports(tag: str):
src = fetch_text("huggingface/peft", tag, "src/peft/__init__.py")
assert src is not None, f"{tag}: src/peft/__init__.py missing"
needed = (
"LoraConfig",
"get_peft_model",
"PeftModel",
)
missing = [n for n in needed if n not in src]
assert not missing, (
f"{tag}: peft top-level missing {missing}; "
f"unsloth.models.sentence_transformer:1948 + unsloth-zoo saving_utils "
f"will ImportError"
)
# -------------------------------------------------------------------------
# LoraConfig at the canonical sub-module path: peft.tuners.lora.LoraConfig
# (or peft.tuners.lora.config.LoraConfig). unsloth-zoo's LoraConfig
# normaliser inspects it via getattr() and dataclass field
# introspection.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", PEFT_TAGS)
def test_peft_lora_config_class(tag: str):
candidates = [
"src/peft/tuners/lora/config.py",
"src/peft/tuners/lora/__init__.py",
"src/peft/tuners/lora.py",
]
found_in = []
for p in candidates:
src = fetch_text("huggingface/peft", tag, p)
if src is not None and has_def(src, "LoraConfig", "class"):
found_in.append(p)
assert found_in, (
f"{tag}: peft.tuners.lora.LoraConfig not in any of {candidates}"
)
# -------------------------------------------------------------------------
# get_peft_model: top-level helper used by sentence_transformer.py:2043.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", PEFT_TAGS)
def test_get_peft_model_function(tag: str):
"""`def get_peft_model(...)` may live in mapping.py (older
layout) or mapping_func.py (peft 0.18+ split). Either is fine."""
candidates = [
"src/peft/mapping.py",
"src/peft/mapping_func.py",
"src/peft/__init__.py",
"src/peft/peft_model.py",
]
for p in candidates:
src = fetch_text("huggingface/peft", tag, p)
if src is not None and has_def(src, "get_peft_model", "func"):
return
pytest.fail(
f"{tag}: def get_peft_model(...) not found in any of {candidates}"
)
# -------------------------------------------------------------------------
# LoraLayer base class: unsloth-zoo's MoE LoRA extractor walks subclasses
# of peft.tuners.lora.LoraLayer to find quantised LoRA modules. If the
# class is renamed or moved, the walk silently returns 0 modules (the
# pytest tests mentioned in the audit report exercise exactly this).
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", PEFT_TAGS)
def test_peft_lora_layer_class(tag: str):
candidates = [
"src/peft/tuners/lora/layer.py",
"src/peft/tuners/lora/__init__.py",
"src/peft/tuners/lora.py",
]
for p in candidates:
src = fetch_text("huggingface/peft", tag, p)
if src is not None and has_def(src, "LoraLayer", "class"):
return
pytest.fail(
f"{tag}: class LoraLayer not in any of {candidates} — "
f"unsloth-zoo MoE LoRA extractor relies on isinstance checks "
f"against this class"
)
# -------------------------------------------------------------------------
# bnb-aware LoRA: peft.tuners.lora.bnb is the integration point with
# bitsandbytes. unsloth + unsloth-zoo dispatch to this when the user
# loads a 4-bit base. Missing this module -> 4bit LoRA silently falls
# back to fp16 LoRA (silently bigger memory footprint).
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", PEFT_TAGS)
def test_peft_lora_bnb_integration(tag: str):
candidates = [
"src/peft/tuners/lora/bnb.py",
"src/peft/tuners/lora/_bnb.py",
]
for p in candidates:
src = fetch_text("huggingface/peft", tag, p)
if src is None:
continue
# The Linear4bit subclass naming is the contract -- either name
# is fine, but at least one bnb-flavoured Linear must exist.
has_4bit = any(
cls in src
for cls in (
"class Linear4bit",
"class Linear8bitLt",
"class _Linear4bit",
"class _Linear8bitLt",
)
)
if has_4bit:
return
pytest.fail(
f"{tag}: peft.tuners.lora.bnb missing or no Linear4bit/Linear8bitLt "
f"class found; unsloth's 4-bit LoRA path silently degrades to fp16"
)

219
tests/version_compat/test_sentence_transformers_pinned_symbols.py Normal file
View file

@ -0,0 +1,219 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team.
"""Pinned-symbol compat check across sentence-transformers PyPI minor
versions. unsloth has a custom integration in
unsloth/models/sentence_transformer.py that:
- Imports SentenceTransformer / SentenceTransformerTrainer at the
top of the public surface (lines 1467, 1798, 1947, 2154).
- Walks `sentence_transformers.models` for Transformer / Pooling /
Normalize (lines 1016, 1206, 1467).
- Calls `sentence_transformers.util.import_from_string` and
`load_dir_path` (lines 1177, 1205).
- Tolerates two alternate base-class paths
(sentence_transformers.base.modules.transformer.Transformer vs
sentence_transformers.models.transformer.Transformer; lines
1169-1171) — at least ONE must resolve.
Strategy: GitHub raw fetch + symbol grep (no pip install, runs CPU-only
on every PR + daily cron). Versioning policy: ST is unpinned in
unsloth/pyproject.toml; cover the most recent minors (5.x line) plus
`main`.
"""
from __future__ import annotations
import re
import pytest
from tests.version_compat._fetch import fetch_text, first_match, has_def
# Policy: unsloth/pyproject.toml does NOT pin sentence-transformers. We
# track the last few minors plus main. Add a row when a new minor lands.
ST_TAGS = [
"v5.0.0",
"v5.1.2",
"v5.2.3",
"v5.3.0",
"v5.4.1",
"master",
]
# -------------------------------------------------------------------------
# Top-level public surface: SentenceTransformer + SentenceTransformerTrainer
# must be importable as `from sentence_transformers import X`.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", ST_TAGS)
def test_st_top_level_exports(tag: str):
src = fetch_text("UKPLab/sentence-transformers", tag, "sentence_transformers/__init__.py")
assert src is not None, (
f"{tag}: sentence_transformers/__init__.py missing"
)
needed = ("SentenceTransformer", "SentenceTransformerTrainer")
missing = [n for n in needed if n not in src]
assert not missing, (
f"{tag}: sentence_transformers top-level missing {missing}; "
f"unsloth.models.sentence_transformer:1467,2154 will ImportError"
)
# -------------------------------------------------------------------------
# Sub-modules: Transformer / Pooling / Normalize. unsloth walks
# `sentence_transformers.models` to introspect these (line 1016, 1206).
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", ST_TAGS)
def test_st_models_re_exports(tag: str):
"""Transformer / Pooling / Normalize must be reachable through
`sentence_transformers.models`. ST 5.4 reorganised the package
(no more top-level `models/` dir; modules live under
`sentence_transformer/` and `base/modules/`), but the public
re-export at `sentence_transformers/__init__.py` still has to
surface these three so user code (and unsloth/models/sentence_transformer.py:1016,1206,1467)
can `from sentence_transformers.models import Transformer` (or
equivalently `from sentence_transformers import models`)."""
# Layout 1 (legacy &lt; 5.4): sentence_transformers/models[.py|/__init__.py].
# Layout 2 (&gt;= 5.4): top-level __init__.py re-exports the symbols
# plus the modules live under base/modules and sentence_transformer/.
legacy_candidates = [
"sentence_transformers/models/__init__.py",
"sentence_transformers/models.py",
]
legacy_hit = first_match("UKPLab/sentence-transformers", tag, legacy_candidates)
needed = ("Transformer", "Pooling", "Normalize")
if legacy_hit is not None:
_path, src = legacy_hit
missing = [n for n in needed if n not in src]
assert not missing, (
f"{tag}: legacy sentence_transformers/models layout missing "
f"{missing}; unsloth.models.sentence_transformer:1016,1206,1467 "
f"ImportError"
)
return
# ST 5.4+ modular layout: classes moved under
# - sentence_transformers/base/modules/transformer.py (Transformer)
# - sentence_transformers/sentence_transformer/modules/pooling.py (Pooling)
# - sentence_transformers/sentence_transformer/modules/normalize.py (Normalize)
# Backward compatibility for `from sentence_transformers.models
# import X` is set up at import time via
# `sentence_transformers.util.deprecated_import.setup_deprecated_module_imports`
# called from sentence_transformers/__init__.py.
expected_paths = {
"Transformer": [
"sentence_transformers/base/modules/transformer.py",
"sentence_transformers/sentence_transformer/Transformer.py",
"sentence_transformers/sentence_transformer/transformer.py",
],
"Pooling": [
"sentence_transformers/sentence_transformer/modules/pooling.py",
"sentence_transformers/sentence_transformer/Pooling.py",
],
"Normalize": [
"sentence_transformers/sentence_transformer/modules/normalize.py",
"sentence_transformers/sentence_transformer/Normalize.py",
],
}
for cls, paths in expected_paths.items():
for p in paths:
src = fetch_text("UKPLab/sentence-transformers", tag, p)
if src and has_def(src, cls, "class"):
break
else:
pytest.fail(
f"{tag}: ST 5.4+ layout: class {cls} not found in any of {paths}"
)
# The backward-compat shim must be wired up so user code doing
# `from sentence_transformers.models import Pooling` keeps working.
top = fetch_text(
"UKPLab/sentence-transformers", tag, "sentence_transformers/__init__.py"
)
assert top is not None, f"{tag}: sentence_transformers/__init__.py missing"
has_shim = bool(
re.search(r"setup_deprecated_module_imports\s*\(", top)
or "import_from_string" in top # fallback signal
)
assert has_shim, (
f"{tag}: ST 5.4+ layout: deprecated-module shim NOT wired in "
f"sentence_transformers/__init__.py; `from "
f"sentence_transformers.models import Pooling` will ImportError "
f"on real install"
)
# -------------------------------------------------------------------------
# Transformer base class: unsloth checks two alternate paths at
# sentence_transformer.py:1169-1171. At least ONE must resolve.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", ST_TAGS)
def test_st_transformer_base_class_either_path(tag: str):
candidates = [
"sentence_transformers/models/Transformer.py",
"sentence_transformers/models/transformer.py",
"sentence_transformers/models/transformer/__init__.py",
"sentence_transformers/base/modules/transformer.py",
]
for p in candidates:
src = fetch_text("UKPLab/sentence-transformers", tag, p)
if src is not None and has_def(src, "Transformer", "class"):
return
pytest.fail(
f"{tag}: class Transformer not in any of {candidates} — "
f"unsloth's three-path probe in sentence_transformer.py:1169-1171 "
f"will ImportError on every fallback"
)
# -------------------------------------------------------------------------
# sentence_transformers.util: import_from_string + load_dir_path are the
# two helpers unsloth.models.sentence_transformer:1177,1205 calls.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", ST_TAGS)
def test_st_util_helpers(tag: str):
"""`sentence_transformers.util.{import_from_string, load_dir_path}` —
used by unsloth.models.sentence_transformer:1177,1205. ST 5.4+ moved
util into a package; we accept either layout. We also accept the
function being defined in any submodule of the util package, since
`from sentence_transformers.util import import_from_string` works
when util/__init__.py re-exports."""
candidates = [
"sentence_transformers/util.py",
"sentence_transformers/util/__init__.py",
]
hit = first_match("UKPLab/sentence-transformers", tag, candidates)
assert hit is not None, (
f"{tag}: sentence_transformers/util[.py|/__init__.py] both missing"
)
_path, src = hit
for fn in ("import_from_string", "load_dir_path"):
defined_here = has_def(src, fn, "func")
reexported = bool(re.search(rf"\b{re.escape(fn)}\b", src))
if not (defined_here or reexported):
# Try common subfiles for the modular layout.
subpaths = [
"sentence_transformers/util/import_utils.py",
"sentence_transformers/util/file_utils.py",
"sentence_transformers/util/_helpers.py",
"sentence_transformers/util/_utils.py",
]
found = False
for sp in subpaths:
sub = fetch_text("UKPLab/sentence-transformers", tag, sp)
if sub and (has_def(sub, fn, "func") or fn in sub):
found = True
break
assert found, (
f"{tag}: sentence_transformers.util.{fn} not found in "
f"util[.py|/__init__.py] or any of {subpaths}"
)

281
tests/version_compat/test_trl_grpo_pinned_symbols.py Normal file
View file

@ -0,0 +1,281 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team.
"""Pinned-symbol compat check across all TRL PyPI minor versions
unsloth + unsloth-zoo target. Catches API drift like:
- trl 0.18 split DataCollatorForPreference into trl.trainer.dpo_trainer
(was trl.trainer.utils). unsloth.models.rl_replacements:318 imports
the post-split path; if a new TRL release moves it again, the
GRPOTrainer.compile cell crashes with ImportError.
- trl 0.20 introduced trl.experimental.openenv as a *gated* module;
unsloth.models.rl_replacements:1765-1770 catches ImportError, but
the gate must remain importable when present.
- trl 0.22 introduced trl.generation.vllm_generation for the
server-mode fast_inference path; unsloth.models.rl_replacements
:1846-1848 catches ImportError, but the module must exist on
versions where unsloth-zoo's vllm_utils dispatches to it.
- trl unwrap_model_for_generation moved from trl.models to
trl.models.utils across releases (unsloth/models/rl.py:152-155
handles both with try/except).
- trl GRPOTrainer / GRPOConfig must remain top-level exports for
`from trl import GRPOTrainer` to work in user code, which is what
`_patch_trl_rl_trainers("grpo_trainer")` discovers.
Strategy: for each tracked TRL tag, fetch the relevant source files
straight from github.com/huggingface/trl (no pip install required) and
assert that every symbol unsloth/unsloth-zoo's RL surface depends on
is present.
Versioning policy: cover the supported window declared in
pyproject.toml (`trl>=0.18.2,!=0.19.0,<=0.24.0`) PLUS several recent
releases ABOVE the cap, so we get early warning when TRL ships
something incompatible and the maintainer can extend the cap or add a
patch BEFORE a user hits it.
"""
from __future__ import annotations
import re
import pytest
from tests.version_compat._fetch import fetch_text, first_match, has_def
# Supported window: 0.18.2 -> 0.24.0 (excluding 0.19.0).
# Above-cap canaries: 0.25, 0.27, 0.29, 1.0, 1.3 (most recent stable at
# the time of writing). `main` is the bleeding edge. Add a row when a
# new minor lands; remove a row only when a release is unsupported
# AND we have a tracking issue.
TRL_TAGS = [
"v0.18.2",
"v0.20.0",
"v0.21.0",
"v0.22.2",
"v0.23.0",
"v0.24.0", # current pyproject cap
# Above-cap canaries:
"v0.25.1",
"v0.27.2",
"v0.29.1",
"v1.0.0",
"v1.3.0",
"main",
]
# -------------------------------------------------------------------------
# HARD-import top-level: from trl import X must keep working for these.
# unsloth/trainer.py + unsloth/models/rl.py rebind these by name.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_trl_top_level_grpo_sft(tag: str):
"""`from trl import GRPOTrainer, GRPOConfig, SFTTrainer, SFTConfig`
must keep resolving at the package root."""
src = fetch_text("huggingface/trl", tag, "trl/__init__.py")
assert src is not None, f"trl/__init__.py missing in {tag}"
for name in ("GRPOTrainer", "GRPOConfig", "SFTTrainer", "SFTConfig"):
assert name in src, (
f"{tag}: `from trl import {name}` will fail; "
f"unsloth/trainer.py + unsloth/models/rl.py rely on this re-export"
)
# -------------------------------------------------------------------------
# trl.trainer.grpo_trainer.GRPOTrainer -- the canonical class. unsloth's
# RL patcher discovers it via `eval(f"trl.trainer.{trainer_file}.{name}")`
# in unsloth/models/rl.py:548-594.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_grpo_trainer_class_canonical_path(tag: str):
src = fetch_text("huggingface/trl", tag, "trl/trainer/grpo_trainer.py")
assert src is not None, (
f"{tag}: trl/trainer/grpo_trainer.py missing — "
f"unsloth.models.rl._patch_trl_rl_trainers('grpo_trainer') breaks"
)
assert has_def(src, "GRPOTrainer", "class"), (
f"{tag}: trl.trainer.grpo_trainer.GRPOTrainer not defined as a class"
)
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_grpo_config_class_canonical_path(tag: str):
"""unsloth/models/rl.py:579-618 looks for the *Config sibling of the
Trainer class via heuristic discovery; the canonical one is in
grpo_config.py."""
candidates = ["trl/trainer/grpo_config.py", "trl/trainer/grpo_trainer.py"]
hit = first_match("huggingface/trl", tag, candidates)
assert hit is not None, f"{tag}: neither grpo_config.py nor grpo_trainer.py found"
_, src = hit
assert has_def(src, "GRPOConfig", "class"), (
f"{tag}: GRPOConfig class missing in {[p for p, _ in [hit]]}; "
f"unsloth's *Config heuristic in models/rl.py:579-618 will fail"
)
# -------------------------------------------------------------------------
# DataCollatorForPreference: unsloth.models.rl_replacements:318 hard-imports
# from trl.trainer.dpo_trainer. Some old TRL versions had it in
# trl.trainer.utils; modern ones moved to trl.trainer.dpo_trainer.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_data_collator_for_preference_resolvable(tag: str):
"""Either the new path (trl.trainer.dpo_trainer) or the old path
(trl.trainer.utils) must define DataCollatorForPreference. unsloth's
string-emitted import in rl_replacements.py:318 uses dpo_trainer;
if neither path resolves, we have a gap."""
new_path = fetch_text("huggingface/trl", tag, "trl/trainer/dpo_trainer.py")
old_path = fetch_text("huggingface/trl", tag, "trl/trainer/utils.py")
have = []
if new_path is not None and "DataCollatorForPreference" in new_path:
have.append("trl.trainer.dpo_trainer")
if old_path is not None and "DataCollatorForPreference" in old_path:
have.append("trl.trainer.utils")
assert have, (
f"{tag}: DataCollatorForPreference defined in NEITHER "
f"trl/trainer/dpo_trainer.py NOR trl/trainer/utils.py — "
f"unsloth/models/rl_replacements.py:318 will ImportError on real install"
)
# -------------------------------------------------------------------------
# trl.trainer.utils.pad: emitted into the GRPO compile cell as
# _unsloth_trl_pad (rl_replacements.py:326).
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_trl_trainer_utils_pad(tag: str):
src = fetch_text("huggingface/trl", tag, "trl/trainer/utils.py")
if src is None:
# Some TRL versions split utils into a package; check the
# alternative location.
src = fetch_text("huggingface/trl", tag, "trl/trainer/utils/__init__.py")
assert src is not None, f"{tag}: trl/trainer/utils[.py|/__init__.py] both missing"
assert has_def(src, "pad", "func") or "def pad(" in src, (
f"{tag}: trl.trainer.utils.pad missing — "
f"unsloth/models/rl_replacements.py:326 emits `from trl.trainer.utils "
f"import pad as _unsloth_trl_pad` into the GRPO compile cell"
)
# -------------------------------------------------------------------------
# trl.models.unwrap_model_for_generation -- moved between submodules
# across releases. unsloth/models/rl.py:152-155 handles both paths.
# Assert at least one resolves on every tag.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_unwrap_model_for_generation_either_path(tag: str):
candidates = [
"trl/models/utils.py",
"trl/models/__init__.py",
"trl/extras/profiling.py", # newer TRL versions hide it here
]
found = False
for path in candidates:
src = fetch_text("huggingface/trl", tag, path)
if src is None:
continue
if (
"def unwrap_model_for_generation" in src
or "unwrap_model_for_generation" in src
):
found = True
break
assert found, (
f"{tag}: trl.unwrap_model_for_generation not in any known path "
f"({candidates}); unsloth/models/rl.py:152-155 will ImportError"
)
# -------------------------------------------------------------------------
# trl.experimental.openenv: gated import (rl_replacements.py:1765-1770
# wraps in try/except). When present, must export the symbols unsloth
# patches.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_trl_experimental_openenv_gated(tag: str):
src = fetch_text("huggingface/trl", tag, "trl/experimental/openenv/__init__.py")
if src is None:
# OK: feature not in this release; unsloth's try/except handles it.
pytest.skip(f"{tag}: trl.experimental.openenv not present (OK)")
# Module exists -> at minimum, `utils` submodule must be importable
# because unsloth patches via `import trl.experimental.openenv.utils`.
utils_src = fetch_text(
"huggingface/trl", tag, "trl/experimental/openenv/utils.py"
)
assert utils_src is not None, (
f"{tag}: trl.experimental.openenv exists but utils.py missing; "
f"unsloth/models/rl_replacements.py:1765 imports openenv.utils explicitly"
)
# -------------------------------------------------------------------------
# trl.generation.vllm_generation: gated import for the fast_inference
# server mode (rl_replacements.py:1846-1848). When present, must define
# at least one symbol unsloth patches against.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_trl_generation_vllm_generation_gated(tag: str):
src = fetch_text("huggingface/trl", tag, "trl/generation/vllm_generation.py")
if src is None:
# OK: pre-server-mode TRL. unsloth's try/except handles absence.
pytest.skip(f"{tag}: trl.generation.vllm_generation not present (OK)")
# If present, at least one of these classes/funcs must be there;
# unsloth-zoo dispatches via getattr() but the module being empty
# means our patch will silently no-op rather than crash.
needs_some = ["VLLMClient", "vllm_generate", "VLLM_AVAILABLE", "VLLMServer"]
has_some = any(name in src for name in needs_some)
assert has_some, (
f"{tag}: trl.generation.vllm_generation exists but none of "
f"{needs_some} present; unsloth-zoo's dispatch in vllm_utils "
f"will silently no-op the server path"
)
# -------------------------------------------------------------------------
# Sanity: TRL's __version__ string is parseable. unsloth/models/rl.py:63
# does `from trl import __version__ as trl_version_raw` and string-
# matches on it.
# -------------------------------------------------------------------------
@pytest.mark.parametrize("tag", TRL_TAGS)
def test_trl_version_parseable(tag: str):
src = fetch_text("huggingface/trl", tag, "trl/__init__.py")
assert src is not None
# Recognised mechanisms:
# 1. literal `__version__ = "x.y.z"`
# 2. `from .version import __version__`
# 3. `__version__ = version("trl")` from `importlib.metadata`
# (bare `version` symbol must be imported on a line above)
has_literal = bool(re.search(r'^__version__\s*=\s*["\']', src, re.MULTILINE))
has_subimport = bool(
re.search(r"^from\s+\.version\s+import\s+__version__", src, re.MULTILINE)
)
# Importlib metadata path: any line `from importlib.metadata import ... version ...`
# plus a `__version__ = version(` assignment somewhere below.
has_metadata = bool(
re.search(
r"^from\s+importlib\.metadata\s+import\s+(?:[\w,\s]+,\s*)?version",
src,
re.MULTILINE,
)
and re.search(r'^\s*__version__\s*=\s*version\s*\(', src, re.MULTILINE)
)
assert has_literal or has_subimport or has_metadata, (
f"{tag}: trl.__version__ not exported via any known mechanism; "
f"unsloth/models/rl.py:63 will AttributeError"
)

8
tests/vllm_compat/test_vllm_pinned_symbols.py
View file

@ -65,6 +65,14 @@ VLLM_TAGS = [
"v0.13.0",
"v0.14.0",
"v0.15.0",
"v0.16.0",
"v0.17.1",
"v0.18.1",
"v0.19.1",
"v0.20.1",
# `main` catches symbol drift that hasn't shipped to PyPI yet,
# giving us a few-day lead on a release that would break us.
"main",
]