tests + CI: callback signature drift detector (#5498)
* tests: callback signature drift detector Static AST check that fails fast when a producer in unsloth_zoo (or unsloth) changes the arity of a callback but a consumer callback def still declares the old arity. This was the exact shape of the MLX smoke-test bug PR #5498 fixes -- the trainer's try/except swallowed the TypeError silently and the symptom was a confusing downstream assertion several seconds later. What the detector does: * Producer side: walks every .py and finds classes that own a self._<name>_callbacks list, populated via .append() from an add_<name>_callback method, and invoked via `for cb in self._<name>_callbacks: cb(arg1, ..., argN)`. The arity at the call site is the canonical expected arity. * Consumer side: walks every <obj>.add_<name>_callback(fn) call, resolves fn to a def or lambda in the same file, and asserts arity matches. Consumers that use *args or **kwargs are tolerantly accepted as any arity. * Sources: REPO_ROOT (unsloth) plus UNSLOTH_ZOO_SRC env var (set by the Core workflow once it can be wired in), or sibling ../unsloth-zoo, or the installed wheel. Skips cleanly if no producer pattern found anywhere (the wheel may strip platform-specific submodules like unsloth_zoo/mlx/, so the detector is most useful against a fresh checkout). Validated end-to-end: * Reverted run_real_mlx_smoke.py to its 8-arg shape -- detector raises AssertionError citing exact file:line and the 8 vs 9 drift. * Restored the 9-arg shape -- detector PASSes. * Total runtime ~7 s in pytest. Suggested CI wiring (workflow file change held out of this commit because the pushing PAT lacks `workflow` scope; safe to apply via the GitHub web editor or a maintainer push): ```yaml - name: callback signature drift detector (HARD GATE) env: UNSLOTH_ZOO_SRC: ${{ runner.temp }}/unsloth-zoo run: | python -m pytest -v --tb=short tests/test_callback_signature_drift.py ``` Drop the step into .github/workflows/consolidated-tests-ci.yml right after the existing public-api drift detector step. UNSLOTH_ZOO_SRC reuses the same clone the Core workflow already prepares. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ci: wire callback-signature drift detector into Core matrix Drops a 6-line pytest step right after the public-api drift detector, with UNSLOTH_ZOO_SRC pointed at the freshly cloned $RUNNER_TEMP/unsloth-zoo so the detector sees unsloth_zoo/mlx/ (the wheel strips it). Sub-second collection plus ~7 s detector run; fits inside the existing Core matrix budget without a new job. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
parent
525b3b4a43
commit
d774af2041
2 changed files with 352 additions and 0 deletions
16
.github/workflows/consolidated-tests-ci.yml
vendored
16
.github/workflows/consolidated-tests-ci.yml
vendored
|
|
@ -316,6 +316,22 @@ jobs:
|
|||
run: |
|
||||
python -m pytest -v --tb=short tests/test_public_api_surface.py
|
||||
|
||||
- name: callback signature drift detector (HARD GATE)
|
||||
# Catches the MLX-style bug from PR #5498: a producer in
|
||||
# unsloth_zoo (or unsloth) grows a callback arg, but a consumer
|
||||
# callback def still declares the old arity. The producer's
|
||||
# try/except swallows the resulting TypeError and the symptom is
|
||||
# "callback never fires" -- usually diagnosed downstream as a
|
||||
# confusing assertion several seconds later. This static AST
|
||||
# check fails fast at PR time. UNSLOTH_ZOO_SRC points at the
|
||||
# freshly cloned main so the detector sees platform-specific
|
||||
# submodules (e.g. unsloth_zoo/mlx/) that the released wheel
|
||||
# may strip.
|
||||
env:
|
||||
UNSLOTH_ZOO_SRC: ${{ runner.temp }}/unsloth-zoo
|
||||
run: |
|
||||
python -m pytest -v --tb=short tests/test_callback_signature_drift.py
|
||||
|
||||
- name: unsloth Bucket-A — CPU tests not in Repo tests (CPU)
|
||||
# 16 tests across 5 files. They live inside tests/saving/ and
|
||||
# tests/utils/, both of which Repo tests (CPU) excludes via --ignore
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue