docker: keep transformers sidecar selection inside what the baked vLLM can import
The image runs unslothai/notebooks unchanged by refusing a notebook's
transformers pin and activating a baked sidecar on sys.path instead. Selection
was a pure ceiling (smallest baked version >= the request) and ignored that vLLM
is version-locked to transformers, so two of the four baked sidecars could not be
imported by the baked vLLM 0.26.0 at all:
4.57.6 ImportError: Support for Transformers v4 is deprecated and was removed
in vLLM v0.24.0
5.3.0 ImportError: cannot import name 'ALLOWED_LAYER_TYPES' from
transformers.configuration_utils
Those two are exactly the ones the common pins select. 241 notebooks pin
4.48/4.52.3/4.55.4/4.56.1/4.56.2/4.57.x and land on the 4.57.6 sidecar, 13 pin
5.2.0/5.3.0 and land on the 5.3.0 sidecar. All 254 died at
`from unsloth import FastModel`, before the first model cell. Pointing
UNSLOTH_TF_SIDECAR_ROOT at an empty directory and changing nothing else turned
Gemma3 (270M) and Gemma3 (1B) GRPO into clean 22/22 and 25/25 passes.
Put a floor in front of the ceiling. Which versions clear the floor is measured,
not hardcoded: the build imports vllm.transformers_utils.config under every
candidate sidecar, deletes the ones that raise, and records the lowest survivor.
That is the vLLM module which reads the transformers API, it reproduces both
failures, and it imports without a GPU, which matters because the build host has
none. A request below the floor is clamped up to the lowest eligible sidecar,
the closest version to the notebook's pin this image can actually run; a request
above every sidecar still falls through to the baked transformers.
Measured on the rebuilt image: sidecars 5.5.0 and 5.10.2 survive, floor 5.5.0,
tf-sidecars drops from 250M to 123M, and all 13 distinct transformers pins found
across the 433 shipped notebooks now reach `from unsloth import FastModel`.
Gemma3 (270M) runs end to end exactly as shipped, 22 of 22 cells, loss 4.09 down
to 0.85 over 10 steps.
This commit is contained in:
Daniel Han
parent
a05c58b6bb
commit
faf1821fcb
3 changed files with 355 additions and 14 deletions
|
|
@ -252,9 +252,28 @@ RUN set -eux \
|
|||
# unsloth_nb_compat.py). Each sidecar is transformers==X + matched
|
||||
# huggingface_hub/tokenizers/safetensors, --no-deps into its own --target under
|
||||
# ${VENV}/tf-sidecars. Prepending one to sys.path swaps transformers without
|
||||
# touching the cu128 base. Versions mirror Studio's tiers (4.57.6 +
|
||||
# 5.3.0/5.5.0/5.10.2). ~300MB after the strip below. Fail-soft per arch/wheel.
|
||||
# touching the cu128 base. Candidate versions mirror Studio's tiers (4.57.6 +
|
||||
# 5.3.0/5.5.0/5.10.2). Fail-soft per arch/wheel.
|
||||
#
|
||||
# Every candidate is then VERIFIED against the baked vLLM and dropped if it does
|
||||
# not survive, because vLLM is version-locked to transformers and a sidecar it
|
||||
# cannot import does not give the notebook an older transformers -- it gives it
|
||||
# an ImportError at `import unsloth`, before the first model cell. Measured on
|
||||
# this image (vLLM 0.26.0): 4.57.6 raises "Support for Transformers v4 ... was
|
||||
# removed in vLLM v0.24.0" and 5.3.0 raises "cannot import name
|
||||
# 'ALLOWED_LAYER_TYPES'", between them breaking 254 of the 433 shipped notebooks,
|
||||
# whose transformers pins select exactly those two. 5.5.0 and 5.10.2 pass.
|
||||
#
|
||||
# vllm.transformers_utils.config is the gate because it is the vLLM module that
|
||||
# reads the transformers API, it reproduces BOTH failures, and it imports without
|
||||
# a GPU (the build host has none, so `import unsloth` cannot be used here).
|
||||
# Deriving the kept set instead of hardcoding it means a later vLLM bump that
|
||||
# widens or narrows the supported range re-tunes the image by itself. The lowest
|
||||
# survivor is recorded as the selection FLOOR read by unsloth_nb_compat.
|
||||
RUN set -eux \
|
||||
&& if ${VENV}/bin/python -c "import vllm" >/dev/null 2>&1; then HAVE_VLLM=1; else HAVE_VLLM=0; fi \
|
||||
&& echo ">> sidecar verification: baked vLLM importable=${HAVE_VLLM}" \
|
||||
&& KEPT="" \
|
||||
&& for TFV in 4.57.6 5.3.0 5.5.0 5.10.2; do \
|
||||
SCRATCH="$(mktemp -d)"; \
|
||||
if ! ${VENV}/bin/uv pip install --python ${VENV}/bin/python \
|
||||
|
|
@ -271,8 +290,24 @@ RUN set -eux \
|
|||
${HFV:+"huggingface_hub==${HFV}"} \
|
||||
${TKV:+"tokenizers==${TKV}"} \
|
||||
${SFV:+"safetensors==${SFV}"}; \
|
||||
echo ">> sidecar transformers==${TFV} (hf_hub=${HFV} tokenizers=${TKV} safetensors=${SFV})"; \
|
||||
if [ "$HAVE_VLLM" = "1" ] && ! PYTHONPATH="$DEST" ${VENV}/bin/python \
|
||||
-c "import vllm.transformers_utils.config" >/dev/null 2>&1; then \
|
||||
echo ">> sidecar transformers==${TFV} DROPPED -- the baked vLLM cannot import under it:"; \
|
||||
PYTHONPATH="$DEST" ${VENV}/bin/python \
|
||||
-c "import vllm.transformers_utils.config" 2>&1 | tail -2 || true; \
|
||||
rm -rf "$DEST"; \
|
||||
continue; \
|
||||
fi; \
|
||||
KEPT="${KEPT} ${TFV}"; \
|
||||
echo ">> sidecar transformers==${TFV} kept (hf_hub=${HFV} tokenizers=${TKV} safetensors=${SFV})"; \
|
||||
done \
|
||||
&& if [ -z "$KEPT" ]; then \
|
||||
echo ">> FATAL: no transformers sidecar survived vLLM verification"; exit 1; \
|
||||
fi \
|
||||
&& if [ "$HAVE_VLLM" = "1" ]; then \
|
||||
printf '%s\n' $KEPT | sort -V | head -1 > ${VENV}/tf-sidecars/.vllm_min_transformers; \
|
||||
fi \
|
||||
&& echo ">> sidecars kept:${KEPT} floor=$(cat ${VENV}/tf-sidecars/.vllm_min_transformers 2>/dev/null || echo '(none)')" \
|
||||
&& { du -sh ${VENV}/tf-sidecars || true; }
|
||||
|
||||
# Informational pin record (NOT byte-reproducible: pip freeze omits wheel hashes
|
||||
|
|
|
|||
|
|
@ -14,8 +14,14 @@ keep the base venv intact and ship coherent transformers "sidecars" -- each is a
|
|||
`pip install --target <dir> --no-deps transformers==X` plus the matched
|
||||
huggingface_hub/tokenizers/safetensors. To use version X we just prepend its
|
||||
sidecar dir to sys.path BEFORE transformers is imported; the rest of the stack
|
||||
(torch, vllm, unsloth, peft, trl) comes from the base venv unchanged. Verified:
|
||||
base unsloth loads + generates under both a 4.57.6 and a 5.5.0 sidecar on B200.
|
||||
(torch, vllm, unsloth, peft, trl) comes from the base venv unchanged.
|
||||
|
||||
That "rest of the stack" is the catch, and it is why selection has a FLOOR as
|
||||
well as a ceiling (see sidecar_for): vLLM is version-locked to transformers, so a
|
||||
sidecar older than what the baked vLLM accepts does not give the notebook an
|
||||
older transformers, it gives it an ImportError at `import unsloth`. The image
|
||||
therefore only ships sidecars whose vLLM import has been verified at build time,
|
||||
and records the lowest of them as the floor.
|
||||
|
||||
Two activation paths:
|
||||
* driven/headless: `unsloth-run <notebook>` sets PYTHONPATH at kernel launch.
|
||||
|
|
@ -31,6 +37,20 @@ SIDECAR_ROOT = os.environ.get("UNSLOTH_TF_SIDECAR_ROOT", "/opt/unsloth-venv/tf-s
|
|||
# The pip/uv shim writes the transformers version a notebook asked for here.
|
||||
MARKER = os.environ.get("UNSLOTH_NB_TF_MARKER", "/tmp/unsloth_nb/requested_transformers")
|
||||
|
||||
# Lowest transformers the image's baked vLLM can import. A sidecar below this is
|
||||
# not "an older transformers", it is a BROKEN image: `import unsloth` dies before
|
||||
# the first model cell. Written by the Dockerfile's sidecar verification step
|
||||
# (which imports vllm.transformers_utils.config under every candidate and drops
|
||||
# the ones that raise), so it tracks whatever vLLM the image actually bakes
|
||||
# instead of a literal that rots on the next bump. Measured on vLLM 0.26.0:
|
||||
#
|
||||
# transformers 4.57.6 FAIL "Support for Transformers v4 ... removed in vLLM v0.24.0"
|
||||
# transformers 5.3.0 FAIL "cannot import name 'ALLOWED_LAYER_TYPES'"
|
||||
# transformers 5.5.0 OK
|
||||
# transformers 5.10.2 OK
|
||||
# transformers 5.14.1 OK (the baked one, no sidecar)
|
||||
FLOOR_FILE = os.path.join(SIDECAR_ROOT, ".vllm_min_transformers")
|
||||
|
||||
|
||||
def _logging_enabled() -> bool:
|
||||
"""Sidecar activation is silent by default; users found the per-cell
|
||||
|
|
@ -70,6 +90,51 @@ def _baked():
|
|||
return out
|
||||
|
||||
|
||||
def min_version():
|
||||
"""Lowest transformers this image's vLLM can import, or None if unrecorded.
|
||||
|
||||
UNSLOTH_TF_SIDECAR_MIN overrides, so a hand-mounted sidecar root can declare
|
||||
its own floor. Returns None when neither is set, which keeps the pre-floor
|
||||
behaviour for any environment that never ran the build-time verification."""
|
||||
v = os.environ.get("UNSLOTH_TF_SIDECAR_MIN", "").strip()
|
||||
if v:
|
||||
return v
|
||||
try:
|
||||
with open(FLOOR_FILE) as f:
|
||||
return f.read().strip() or None
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def _eligible():
|
||||
"""Baked sidecars the floor allows, as a sorted [(Version, version_str, dir)].
|
||||
|
||||
Returns None when the versions cannot be parsed (no packaging available)."""
|
||||
baked = _baked()
|
||||
if not baked:
|
||||
return []
|
||||
try:
|
||||
from packaging.version import Version
|
||||
except Exception:
|
||||
return None
|
||||
floor = min_version()
|
||||
try:
|
||||
low = Version(floor) if floor else None
|
||||
except Exception:
|
||||
low = None
|
||||
rows = []
|
||||
for v, d in baked.items():
|
||||
try:
|
||||
ver = Version(v)
|
||||
except Exception:
|
||||
continue
|
||||
if low is not None and ver < low:
|
||||
continue # vLLM cannot import it; activating it only breaks the run
|
||||
rows.append((ver, v, d))
|
||||
rows.sort()
|
||||
return rows
|
||||
|
||||
|
||||
def tier_for_model(model_name: str):
|
||||
"""Best-effort minimum transformers version for a model id (or None)."""
|
||||
if not model_name:
|
||||
|
|
@ -85,21 +150,42 @@ def tier_for_model(model_name: str):
|
|||
def sidecar_for(version: str):
|
||||
"""Map a requested/needed transformers version to a baked sidecar dir.
|
||||
|
||||
Uses ceiling semantics: the smallest baked version >= the request, because a
|
||||
model added in version X needs *at least* X. If the request is newer than
|
||||
every baked sidecar, return None -> use the base venv (the newest 5.x)."""
|
||||
baked = _baked()
|
||||
if not baked or not version:
|
||||
FLOOR then CEILING, in that order:
|
||||
|
||||
* floor -- a sidecar the baked vLLM cannot import is never eligible, no
|
||||
matter what the notebook pinned. Selecting one used to break `import
|
||||
unsloth` in 254 of the 433 shipped notebooks, because the two common pin
|
||||
families (4.5x -> the 4.57.6 sidecar, 5.2/5.3 -> the 5.3.0 sidecar) both
|
||||
landed on a sidecar vLLM 0.26.0 refuses. A request below the floor is
|
||||
clamped UP to the lowest eligible sidecar: that is the closest version to
|
||||
what the notebook asked for that this image can actually run.
|
||||
* ceiling -- among the eligible sidecars pick the smallest >= the request,
|
||||
because a model added in version X needs *at least* X.
|
||||
|
||||
A request newer than every eligible sidecar returns None -> use the base venv
|
||||
(the newest 5.x), which is always vLLM-compatible."""
|
||||
if not version:
|
||||
return None
|
||||
if version in baked:
|
||||
return baked[version]
|
||||
rows = _eligible()
|
||||
if rows is None: # no packaging: only an exact, still-eligible match is safe
|
||||
baked = _baked()
|
||||
d = baked.get(version)
|
||||
floor = min_version()
|
||||
return d if (d and (not floor or version == floor)) else None
|
||||
if not rows:
|
||||
return None
|
||||
for _ver, v, d in rows:
|
||||
if v == version:
|
||||
return d
|
||||
try:
|
||||
from packaging.version import Version
|
||||
want = Version(version)
|
||||
except Exception:
|
||||
return None
|
||||
ge = sorted((Version(v), d) for v, d in baked.items() if Version(v) >= want)
|
||||
return ge[0][1] if ge else None
|
||||
for ver, _v, d in rows:
|
||||
if ver >= want:
|
||||
return d
|
||||
return None
|
||||
|
||||
|
||||
def requested_version():
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue