CI(consolidated): fix spoof recursion + per-step continue-on-error + drop static-check upgrades
Run #8 (matrix) failures: - Cells 2 & 3: RecursionError in patch_tiled_mlp shim. Root cause: tests/_zoo_aggressive_cuda_spoof.py routed torch.cuda.manual_seed and manual_seed_all back through torch.manual_seed, but torch.manual_seed internally calls torch.cuda.manual_seed_all -> infinite recursion. Fix: no-op the cuda seed APIs (callers already paid the CPU-RNG cost via torch.manual_seed; CUDA-side seeding has no meaning on a GPU-less runner). Same fix for cuda.set_rng_state / get_rng_state and initial_seed / seed / seed_all. Locally re-validated tiled MLP shim: diff = 0.000e+00, no recursion. - Cell 1: unsloth_zoo's test_every_patched_moe_experts_class_has_lora_extractor fails on transformers==4.57.6 because the MoE class surface unsloth_zoo patches is newer. That's the real drift signal the matrix is supposed to surface; the bug is upstream, not in CI. Keeping it as-is. Per-step `continue-on-error: true` added on every test step so a cell running into one failure (like cell 1's MoE test) still runs the remaining steps (test_apply_fused_lm_head, static checks, runtime patch ledger, tiled MLP, llama-cli smoke). The job-level continue-on-error remains. Drop `pip install --upgrade 'transformers>=4.51,<5.5'` and `'trl>=0.13,<1'` in the static-check steps -- those upgrades would override the matrix-selected versions and defeat the matrix's purpose. The static checks now use whatever versions the runtime-deps step installed for that cell.
This commit is contained in:
Daniel Han
parent
e519ac56ac
commit
d1d89519a9
2 changed files with 50 additions and 54 deletions
22
.github/workflows/consolidated-tests-ci.yml
vendored
22
.github/workflows/consolidated-tests-ci.yml
vendored
|
|
@ -232,6 +232,7 @@ jobs:
|
|||
# the job out fast if a transformers/torch resolution went sideways.
|
||||
# Inherits PYTHONPATH / UNSLOTH_COMPILE_DISABLE / PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION
|
||||
# from the job-level env block.
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
python -m pytest --collect-only -q \
|
||||
|
|
@ -248,6 +249,7 @@ jobs:
|
|||
# because their sibling files need real GPUs / real HF weights.
|
||||
# The five files below are pure-Python + AST/protobuf/regex tests
|
||||
# that run cleanly on CPU. Env inherited from the job block.
|
||||
continue-on-error: true
|
||||
run: |
|
||||
python -m pytest -q --tb=short \
|
||||
tests/saving/test_save_shell_injection.py \
|
||||
|
|
@ -266,6 +268,7 @@ jobs:
|
|||
# cases below auto-skip on a GPU-less runner; deselect them
|
||||
# explicitly so the no-CUDA outcome is "deselected", not "skipped",
|
||||
# making intent visible in the report. Env inherited from job block.
|
||||
continue-on-error: true
|
||||
working-directory: ${{ runner.temp }}/unsloth-zoo
|
||||
run: |
|
||||
python -m pytest -q --tb=short tests/ \
|
||||
|
|
@ -283,6 +286,7 @@ jobs:
|
|||
# spoof is required because unsloth_zoo/temporary_patches/gpt_oss.py
|
||||
# at module load reads torch.cuda.memory.mem_get_info(0), which
|
||||
# bare `is_available = True` doesn't cover. Env inherited.
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
cat > tests/_zoo_apply_fused_lm_head_shim.py <<'PY'
|
||||
|
|
@ -307,12 +311,13 @@ jobs:
|
|||
# they reference still exists in the installed `trl`. Catches API
|
||||
# drift (renamed / removed TRL classes) without running training.
|
||||
# Pre-fetches latest pip transformers in case TRL pinned an older one.
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
# Refresh transformers + trl to whatever pip resolves as latest within
|
||||
# the constraint range so this step verifies against the upstream the
|
||||
# PR will actually install in production.
|
||||
pip install --upgrade 'transformers>=4.51,<5.5' 'trl>=0.13,<1'
|
||||
# Use the matrix-resolved transformers + trl versions already
|
||||
# installed by the runtime-deps step (don't upgrade here; that
|
||||
# would defeat the matrix's purpose of testing against the
|
||||
# specific (transformers, trl) combination the cell selected).
|
||||
python <<'PY'
|
||||
import ast, importlib, pathlib, sys
|
||||
paths = [pathlib.Path("unsloth/trainer.py"),
|
||||
|
|
@ -338,11 +343,12 @@ jobs:
|
|||
- name: Static checks — unsloth_zoo/tiled_mlp.py against latest pip transformers
|
||||
# AST parse + transformers symbol-resolution. The user flagged tiled
|
||||
# MLP patching as the path that breaks first when transformers ships
|
||||
# an MLP class rename; this step is the canary.
|
||||
# an MLP class rename; this step is the canary against whatever
|
||||
# transformers version the matrix cell selected.
|
||||
continue-on-error: true
|
||||
working-directory: ${{ runner.temp }}/unsloth-zoo
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
pip install --upgrade 'transformers>=4.51,<5.5'
|
||||
python <<'PY'
|
||||
import ast, importlib, pathlib, sys
|
||||
p = pathlib.Path("unsloth_zoo/tiled_mlp.py")
|
||||
|
|
@ -367,6 +373,7 @@ jobs:
|
|||
PY
|
||||
|
||||
- name: Static checks — unsloth_zoo/hf_utils.py syntax + import-graph
|
||||
continue-on-error: true
|
||||
working-directory: ${{ runner.temp }}/unsloth-zoo
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
|
|
@ -391,6 +398,7 @@ jobs:
|
|||
# NameError: name 'fast_lora_forward' is not defined). The shim
|
||||
# reports the full ledger but only fails when one of the two
|
||||
# `required` helpers is absent.
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
cat > tests/_runtime_patch_check_shim.py <<'PY'
|
||||
|
|
@ -477,6 +485,7 @@ jobs:
|
|||
# Same shim pattern: pytest picks up tests/conftest.py before importing
|
||||
# unsloth_zoo.tiled_mlp, so the GPU-spoof harness covers
|
||||
# unsloth_zoo.temporary_patches.gpt_oss's mem_get_info call.
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
cat > tests/_tiled_mlp_check_shim.py <<'PY'
|
||||
|
|
@ -526,6 +535,7 @@ jobs:
|
|||
rm -f tests/_tiled_mlp_check_shim.py
|
||||
|
||||
- name: llama.cpp install + `llama-cli --help` smoke
|
||||
continue-on-error: true
|
||||
# The user asked to confirm llama.cpp installs and the CLI runs.
|
||||
# Studio uses prebuilt llama.cpp binaries via studio/install_llama_prebuilt.py;
|
||||
# we mirror that flow here at a smaller scale: pull the upstream prebuilt
|
||||
|
|
|
|||
|
|
@ -88,39 +88,43 @@ def apply() -> None:
|
|||
torch.cuda.nvtx = nvtx_stub # type: ignore[attr-defined]
|
||||
|
||||
# ----- random API ----------------------------------------------------
|
||||
torch.cuda.manual_seed = lambda seed: torch.manual_seed(seed) # type: ignore[assignment]
|
||||
torch.cuda.manual_seed_all = lambda seed: torch.manual_seed(seed) # type: ignore[assignment]
|
||||
torch.cuda.get_rng_state = lambda *a, **k: torch.get_rng_state() # type: ignore[assignment]
|
||||
torch.cuda.set_rng_state = lambda state, *a, **k: torch.set_rng_state(state) # type: ignore[assignment]
|
||||
torch.cuda.get_rng_state_all = lambda *a, **k: [torch.get_rng_state()] # type: ignore[attr-defined]
|
||||
torch.cuda.set_rng_state_all = (
|
||||
lambda states, *a, **k: torch.set_rng_state(states[0]) if states else None
|
||||
) # type: ignore[attr-defined]
|
||||
# CRITICAL: torch.manual_seed() internally calls torch.cuda.manual_seed_all(),
|
||||
# so routing the cuda seed APIs back through torch.manual_seed would
|
||||
# infinite-recurse (observed as RecursionError in run #8 cells 2/3 of the
|
||||
# consolidated CI matrix). No-op them: callers that explicitly seed CUDA
|
||||
# have already paid the cost of seeding CPU via torch.manual_seed; the
|
||||
# CUDA-side seeding has no meaning on a GPU-less runner.
|
||||
torch.cuda.manual_seed = lambda *a, **k: None # type: ignore[assignment]
|
||||
torch.cuda.manual_seed_all = lambda *a, **k: None # type: ignore[assignment]
|
||||
# rng_state APIs: return a CPU-shaped placeholder and accept anything for
|
||||
# set; do NOT route through torch.set_rng_state / get_rng_state -- those
|
||||
# operate on the CPU RNG directly and are independent of the cuda surface.
|
||||
import torch as _t
|
||||
_empty_rng_state = _t.empty(0, dtype=_t.uint8)
|
||||
torch.cuda.get_rng_state = lambda *a, **k: _empty_rng_state.clone() # type: ignore[assignment]
|
||||
torch.cuda.set_rng_state = lambda *a, **k: None # type: ignore[assignment]
|
||||
torch.cuda.get_rng_state_all = lambda *a, **k: [_empty_rng_state.clone()] # type: ignore[attr-defined]
|
||||
torch.cuda.set_rng_state_all = lambda *a, **k: None # type: ignore[attr-defined]
|
||||
torch.cuda.initial_seed = lambda *a, **k: 0 # type: ignore[assignment]
|
||||
torch.cuda.seed = lambda *a, **k: None # type: ignore[assignment]
|
||||
torch.cuda.seed_all = lambda *a, **k: None # type: ignore[assignment]
|
||||
|
||||
# ----- Stream / Event no-op classes -----------------------------------
|
||||
class _NoopStream:
|
||||
def __init__(self, *a, **k): ...
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
def __enter__(self): return self
|
||||
def __exit__(self, *a): return False
|
||||
def synchronize(self, *a, **k): ...
|
||||
def wait_stream(self, *a, **k): ...
|
||||
def query(self):
|
||||
return True
|
||||
def query(self): return True
|
||||
|
||||
class _NoopEvent:
|
||||
def __init__(self, *a, **k): ...
|
||||
def record(self, *a, **k): ...
|
||||
def wait(self, *a, **k): ...
|
||||
def query(self):
|
||||
return True
|
||||
|
||||
def query(self): return True
|
||||
def synchronize(self, *a, **k): ...
|
||||
def elapsed_time(self, *a, **k):
|
||||
return 0.0
|
||||
def elapsed_time(self, *a, **k): return 0.0
|
||||
|
||||
torch.cuda.Stream = _NoopStream # type: ignore[assignment]
|
||||
torch.cuda.Event = _NoopEvent # type: ignore[assignment]
|
||||
|
|
@ -132,21 +136,15 @@ def apply() -> None:
|
|||
# `torch.empty(..., pin_memory=True)` and friends raise on a CPU-only
|
||||
# build. Strip the kwarg — pin_memory has no meaning here.
|
||||
for _name in (
|
||||
"empty",
|
||||
"zeros",
|
||||
"ones",
|
||||
"empty_like",
|
||||
"zeros_like",
|
||||
"ones_like",
|
||||
"rand",
|
||||
"randn",
|
||||
"randint",
|
||||
"empty", "zeros", "ones",
|
||||
"empty_like", "zeros_like", "ones_like",
|
||||
"rand", "randn", "randint",
|
||||
):
|
||||
_orig = getattr(torch, _name, None)
|
||||
if _orig is None:
|
||||
continue
|
||||
|
||||
def _wrap(*args: Any, _orig = _orig, **kwargs: Any):
|
||||
def _wrap(*args: Any, _orig=_orig, **kwargs: Any):
|
||||
kwargs.pop("pin_memory", None)
|
||||
return _orig(*args, **kwargs)
|
||||
|
||||
|
|
@ -166,28 +164,16 @@ def apply() -> None:
|
|||
import torch.cuda.amp # type: ignore
|
||||
except Exception:
|
||||
cuda_amp = types.ModuleType("torch.cuda.amp")
|
||||
|
||||
class _StubScaler:
|
||||
def __init__(self, *a, **k): ...
|
||||
def scale(self, x):
|
||||
return x
|
||||
|
||||
def step(self, opt):
|
||||
opt.step()
|
||||
|
||||
def scale(self, x): return x
|
||||
def step(self, opt): opt.step()
|
||||
def update(self, *a, **k): ...
|
||||
def unscale_(self, *a, **k): ...
|
||||
def get_scale(self):
|
||||
return 1.0
|
||||
|
||||
def is_enabled(self):
|
||||
return False
|
||||
|
||||
def state_dict(self):
|
||||
return {}
|
||||
|
||||
def get_scale(self): return 1.0
|
||||
def is_enabled(self): return False
|
||||
def state_dict(self): return {}
|
||||
def load_state_dict(self, *a, **k): ...
|
||||
|
||||
cuda_amp.GradScaler = _StubScaler # type: ignore[attr-defined]
|
||||
sys.modules.setdefault("torch.cuda.amp", cuda_amp)
|
||||
torch.cuda.amp = cuda_amp # type: ignore[attr-defined]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue