tests: public-api surface drift detector (companion to test_import_fixes_drift.py) (#5428)

* tests: ship public-api surface drift detector + wire into Core matrix

Companion to tests/test_import_fixes_drift.py (PR #5414): that file
catches drift in THIRD-PARTY libs (transformers / trl / triton / peft /
vllm / torchcodec / xformers); this file catches drift in unsloth's
OWN public-surface API -- the top-9 classmethods + symbols that
unslothai/notebooks calls at ~2000 cumulative sites.

Closes the gap where a refactor on this repo (e.g. renaming
FastLanguageModel.from_pretrained -> .load) would pass unsloth CI
green and surface only on the next unslothai/notebooks CI run, or
worse, on a user's Colab crash report.

Coverage (call-site counts measured against unslothai/notebooks main):
  test_fast_language_model_class_present
  test_fast_language_model_from_pretrained_kwargs        506 sites
  test_fast_language_model_get_peft_model_kwargs         304 sites
  test_fast_language_model_for_inference_callable        370 sites
  test_fast_vision_model_class_and_methods         (4 methods)
  test_fast_vision_model_get_peft_model_vision_kwargs    (4 kwargs)
  test_fast_model_class_and_methods                (2 methods)
  test_fast_model_from_pretrained_kwargs                 103 sites
  test_is_bf16_supported_or_alias_callable        48 + 8 sites

Each test asserts the healthy public shape via inspect.signature; on
regression fires pytest.fail("DRIFT DETECTED: ...") -- never
pytest.skip -- so the Core matrix cell goes red. Mirrors the same
skeleton used by tests/test_import_fixes_drift.py.

Wired as a new step in consolidated-tests-ci.yml right after the
import_fixes drift step, inside every Core matrix cell.

Local verification on transformers 4.57.6 + unsloth main:
  pytest tests/test_public_api_surface.py -v
  -> 9 passed in 0.02s

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-05-14 19:56:21 -07:00 committed by GitHub
commit ab21dc25b4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 227 additions and 0 deletions

View file

@ -304,6 +304,17 @@ jobs:
run: |
python -m pytest -v --tb=short tests/test_import_fixes_drift.py
- name: public-api surface drift detectors (9 tests, HARD GATE)
# Companion to test_import_fixes_drift.py: that file catches
# third-party drift; this one catches drift in unsloth's OWN
# public surface (FastLanguageModel / FastVisionModel /
# FastModel + their classmethods + is_bf16_supported). A
# rename here would silently break the unslothai/notebooks tree
# one PR cycle later -- this gate catches it BEFORE the
# breakage reaches users.
run: |
python -m pytest -v --tb=short tests/test_public_api_surface.py
- name: unsloth Bucket-A — CPU tests not in Repo tests (CPU)
# 16 tests across 5 files. They live inside tests/saving/ and
# tests/utils/, both of which Repo tests (CPU) excludes via --ignore