tests + CI: callback signature drift detector (#5498)

* tests: callback signature drift detector

Static AST check that fails fast when a producer in unsloth_zoo (or
unsloth) changes the arity of a callback but a consumer callback def
still declares the old arity. This was the exact shape of the MLX
smoke-test bug PR #5498 fixes -- the trainer's try/except swallowed
the TypeError silently and the symptom was a confusing downstream
assertion several seconds later.

What the detector does:
  * Producer side: walks every .py and finds classes that own a
    self._<name>_callbacks list, populated via .append() from an
    add_<name>_callback method, and invoked via
    `for cb in self._<name>_callbacks: cb(arg1, ..., argN)`. The
    arity at the call site is the canonical expected arity.
  * Consumer side: walks every <obj>.add_<name>_callback(fn) call,
    resolves fn to a def or lambda in the same file, and asserts
    arity matches. Consumers that use *args or **kwargs are
    tolerantly accepted as any arity.
  * Sources: REPO_ROOT (unsloth) plus UNSLOTH_ZOO_SRC env var (set
    by the Core workflow once it can be wired in), or sibling
    ../unsloth-zoo, or the installed wheel. Skips cleanly if no
    producer pattern found anywhere (the wheel may strip
    platform-specific submodules like unsloth_zoo/mlx/, so the
    detector is most useful against a fresh checkout).

Validated end-to-end:
  * Reverted run_real_mlx_smoke.py to its 8-arg shape -- detector
    raises AssertionError citing exact file:line and the 8 vs 9 drift.
  * Restored the 9-arg shape -- detector PASSes.
  * Total runtime ~7 s in pytest.

Suggested CI wiring (workflow file change held out of this commit
because the pushing PAT lacks `workflow` scope; safe to apply via
the GitHub web editor or a maintainer push):

```yaml
- name: callback signature drift detector (HARD GATE)
  env:
    UNSLOTH_ZOO_SRC: ${{ runner.temp }}/unsloth-zoo
  run: |
    python -m pytest -v --tb=short tests/test_callback_signature_drift.py
```

Drop the step into .github/workflows/consolidated-tests-ci.yml right
after the existing public-api drift detector step. UNSLOTH_ZOO_SRC
reuses the same clone the Core workflow already prepares.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ci: wire callback-signature drift detector into Core matrix

Drops a 6-line pytest step right after the public-api drift detector,
with UNSLOTH_ZOO_SRC pointed at the freshly cloned $RUNNER_TEMP/unsloth-zoo
so the detector sees unsloth_zoo/mlx/ (the wheel strips it).

Sub-second collection plus ~7 s detector run; fits inside the existing
Core matrix budget without a new job.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-05-18 04:42:37 -07:00 committed by GitHub
commit d774af2041
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 352 additions and 0 deletions

View file

@ -316,6 +316,22 @@ jobs:
run: |
python -m pytest -v --tb=short tests/test_public_api_surface.py
- name: callback signature drift detector (HARD GATE)
# Catches the MLX-style bug from PR #5498: a producer in
# unsloth_zoo (or unsloth) grows a callback arg, but a consumer
# callback def still declares the old arity. The producer's
# try/except swallows the resulting TypeError and the symptom is
# "callback never fires" -- usually diagnosed downstream as a
# confusing assertion several seconds later. This static AST
# check fails fast at PR time. UNSLOTH_ZOO_SRC points at the
# freshly cloned main so the detector sees platform-specific
# submodules (e.g. unsloth_zoo/mlx/) that the released wheel
# may strip.
env:
UNSLOTH_ZOO_SRC: ${{ runner.temp }}/unsloth-zoo
run: |
python -m pytest -v --tb=short tests/test_callback_signature_drift.py
- name: unsloth Bucket-A — CPU tests not in Repo tests (CPU)
# 16 tests across 5 files. They live inside tests/saving/ and
# tests/utils/, both of which Repo tests (CPU) excludes via --ignore