ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docker: keep transformers sidecar selection inside what the baked vLLM can import

Browse source 
The image runs unslothai/notebooks unchanged by refusing a notebook's
transformers pin and activating a baked sidecar on sys.path instead. Selection
was a pure ceiling (smallest baked version >= the request) and ignored that vLLM
is version-locked to transformers, so two of the four baked sidecars could not be
imported by the baked vLLM 0.26.0 at all:

  4.57.6  ImportError: Support for Transformers v4 is deprecated and was removed
          in vLLM v0.24.0
  5.3.0   ImportError: cannot import name 'ALLOWED_LAYER_TYPES' from
          transformers.configuration_utils

Those two are exactly the ones the common pins select. 241 notebooks pin
4.48/4.52.3/4.55.4/4.56.1/4.56.2/4.57.x and land on the 4.57.6 sidecar, 13 pin
5.2.0/5.3.0 and land on the 5.3.0 sidecar. All 254 died at
`from unsloth import FastModel`, before the first model cell. Pointing
UNSLOTH_TF_SIDECAR_ROOT at an empty directory and changing nothing else turned
Gemma3 (270M) and Gemma3 (1B) GRPO into clean 22/22 and 25/25 passes.

Put a floor in front of the ceiling. Which versions clear the floor is measured,
not hardcoded: the build imports vllm.transformers_utils.config under every
candidate sidecar, deletes the ones that raise, and records the lowest survivor.
That is the vLLM module which reads the transformers API, it reproduces both
failures, and it imports without a GPU, which matters because the build host has
none. A request below the floor is clamped up to the lowest eligible sidecar,
the closest version to the notebook's pin this image can actually run; a request
above every sidecar still falls through to the baked transformers.

Measured on the rebuilt image: sidecars 5.5.0 and 5.10.2 survive, floor 5.5.0,
tf-sidecars drops from 250M to 123M, and all 13 distinct transformers pins found
across the 433 shipped notebooks now reach `from unsloth import FastModel`.
Gemma3 (270M) runs end to end exactly as shipped, 22 of 22 cells, loss 4.09 down
to 0.85 over 10 steps.
This commit is contained in:
Daniel Han 2026-07-26 17:28:08 +00:00
commit faf1821fcb
3 changed files with 355 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

41
docker/Dockerfile
View file

@ -252,9 +252,28 @@ RUN set -eux \
# unsloth_nb_compat.py). Each sidecar is transformers==X + matched
# huggingface_hub/tokenizers/safetensors, --no-deps into its own --target under
# ${VENV}/tf-sidecars. Prepending one to sys.path swaps transformers without
# touching the cu128 base. Versions mirror Studio's tiers (4.57.6 +
# 5.3.0/5.5.0/5.10.2). ~300MB after the strip below. Fail-soft per arch/wheel.
# touching the cu128 base. Candidate versions mirror Studio's tiers (4.57.6 +
# 5.3.0/5.5.0/5.10.2). Fail-soft per arch/wheel.
#
# Every candidate is then VERIFIED against the baked vLLM and dropped if it does
# not survive, because vLLM is version-locked to transformers and a sidecar it
# cannot import does not give the notebook an older transformers -- it gives it
# an ImportError at `import unsloth`, before the first model cell. Measured on
# this image (vLLM 0.26.0): 4.57.6 raises "Support for Transformers v4 ... was
# removed in vLLM v0.24.0" and 5.3.0 raises "cannot import name
# 'ALLOWED_LAYER_TYPES'", between them breaking 254 of the 433 shipped notebooks,
# whose transformers pins select exactly those two. 5.5.0 and 5.10.2 pass.
#
# vllm.transformers_utils.config is the gate because it is the vLLM module that
# reads the transformers API, it reproduces BOTH failures, and it imports without
# a GPU (the build host has none, so `import unsloth` cannot be used here).
# Deriving the kept set instead of hardcoding it means a later vLLM bump that
# widens or narrows the supported range re-tunes the image by itself. The lowest
# survivor is recorded as the selection FLOOR read by unsloth_nb_compat.
RUN set -eux \
&& if ${VENV}/bin/python -c "import vllm" >/dev/null 2>&1; then HAVE_VLLM=1; else HAVE_VLLM=0; fi \
&& echo ">> sidecar verification: baked vLLM importable=${HAVE_VLLM}" \
&& KEPT="" \
&& for TFV in 4.57.6 5.3.0 5.5.0 5.10.2; do \
SCRATCH="$(mktemp -d)"; \
if ! ${VENV}/bin/uv pip install --python ${VENV}/bin/python \
@ -271,8 +290,24 @@ RUN set -eux \
${HFV:+"huggingface_hub==${HFV}"} \
${TKV:+"tokenizers==${TKV}"} \
${SFV:+"safetensors==${SFV}"}; \
echo ">> sidecar transformers==${TFV} (hf_hub=${HFV} tokenizers=${TKV} safetensors=${SFV})"; \
if [ "$HAVE_VLLM" = "1" ] && ! PYTHONPATH="$DEST" ${VENV}/bin/python \
-c "import vllm.transformers_utils.config" >/dev/null 2>&1; then \
echo ">> sidecar transformers==${TFV} DROPPED -- the baked vLLM cannot import under it:"; \
PYTHONPATH="$DEST" ${VENV}/bin/python \
-c "import vllm.transformers_utils.config" 2>&1 | tail -2 || true; \
rm -rf "$DEST"; \
continue; \
fi; \
KEPT="${KEPT} ${TFV}"; \
echo ">> sidecar transformers==${TFV} kept (hf_hub=${HFV} tokenizers=${TKV} safetensors=${SFV})"; \
done \
&& if [ -z "$KEPT" ]; then \
echo ">> FATAL: no transformers sidecar survived vLLM verification"; exit 1; \
fi \
&& if [ "$HAVE_VLLM" = "1" ]; then \
printf '%s\n' $KEPT | sort -V | head -1 > ${VENV}/tf-sidecars/.vllm_min_transformers; \
fi \
&& echo ">> sidecars kept:${KEPT} floor=$(cat ${VENV}/tf-sidecars/.vllm_min_transformers 2>/dev/null || echo '(none)')" \
&& { du -sh ${VENV}/tf-sidecars || true; }
# Informational pin record (NOT byte-reproducible: pip freeze omits wheel hashes

108
docker/unsloth_nb_compat.py
View file

@ -14,8 +14,14 @@ keep the base venv intact and ship coherent transformers "sidecars" -- each is a
`pip install --target <dir> --no-deps transformers==X` plus the matched
huggingface_hub/tokenizers/safetensors. To use version X we just prepend its
sidecar dir to sys.path BEFORE transformers is imported; the rest of the stack
(torch, vllm, unsloth, peft, trl) comes from the base venv unchanged. Verified:
base unsloth loads + generates under both a 4.57.6 and a 5.5.0 sidecar on B200.
(torch, vllm, unsloth, peft, trl) comes from the base venv unchanged.
That "rest of the stack" is the catch, and it is why selection has a FLOOR as
well as a ceiling (see sidecar_for): vLLM is version-locked to transformers, so a
sidecar older than what the baked vLLM accepts does not give the notebook an
older transformers, it gives it an ImportError at `import unsloth`. The image
therefore only ships sidecars whose vLLM import has been verified at build time,
and records the lowest of them as the floor.
Two activation paths:
* driven/headless: `unsloth-run <notebook>` sets PYTHONPATH at kernel launch.
@ -31,6 +37,20 @@ SIDECAR_ROOT = os.environ.get("UNSLOTH_TF_SIDECAR_ROOT", "/opt/unsloth-venv/tf-s
# The pip/uv shim writes the transformers version a notebook asked for here.
MARKER = os.environ.get("UNSLOTH_NB_TF_MARKER", "/tmp/unsloth_nb/requested_transformers")
# Lowest transformers the image's baked vLLM can import. A sidecar below this is
# not "an older transformers", it is a BROKEN image: `import unsloth` dies before
# the first model cell. Written by the Dockerfile's sidecar verification step
# (which imports vllm.transformers_utils.config under every candidate and drops
# the ones that raise), so it tracks whatever vLLM the image actually bakes
# instead of a literal that rots on the next bump. Measured on vLLM 0.26.0:
#
# transformers 4.57.6 FAIL "Support for Transformers v4 ... removed in vLLM v0.24.0"
# transformers 5.3.0 FAIL "cannot import name 'ALLOWED_LAYER_TYPES'"
# transformers 5.5.0 OK
# transformers 5.10.2 OK
# transformers 5.14.1 OK (the baked one, no sidecar)
FLOOR_FILE = os.path.join(SIDECAR_ROOT, ".vllm_min_transformers")
def _logging_enabled() -> bool:
"""Sidecar activation is silent by default; users found the per-cell
@ -70,6 +90,51 @@ def _baked():
return out
def min_version():
"""Lowest transformers this image's vLLM can import, or None if unrecorded.
UNSLOTH_TF_SIDECAR_MIN overrides, so a hand-mounted sidecar root can declare
its own floor. Returns None when neither is set, which keeps the pre-floor
behaviour for any environment that never ran the build-time verification."""
v = os.environ.get("UNSLOTH_TF_SIDECAR_MIN", "").strip()
if v:
return v
try:
with open(FLOOR_FILE) as f:
return f.read().strip() or None
except OSError:
return None
def _eligible():
"""Baked sidecars the floor allows, as a sorted [(Version, version_str, dir)].
Returns None when the versions cannot be parsed (no packaging available)."""
baked = _baked()
if not baked:
return []
try:
from packaging.version import Version
except Exception:
return None
floor = min_version()
try:
low = Version(floor) if floor else None
except Exception:
low = None
rows = []
for v, d in baked.items():
try:
ver = Version(v)
except Exception:
continue
if low is not None and ver < low:
continue # vLLM cannot import it; activating it only breaks the run
rows.append((ver, v, d))
rows.sort()
return rows
def tier_for_model(model_name: str):
"""Best-effort minimum transformers version for a model id (or None)."""
if not model_name:
@ -85,21 +150,42 @@ def tier_for_model(model_name: str):
def sidecar_for(version: str):
"""Map a requested/needed transformers version to a baked sidecar dir.
Uses ceiling semantics: the smallest baked version >= the request, because a
model added in version X needs *at least* X. If the request is newer than
every baked sidecar, return None -> use the base venv (the newest 5.x)."""
baked = _baked()
if not baked or not version:
FLOOR then CEILING, in that order:
* floor -- a sidecar the baked vLLM cannot import is never eligible, no
matter what the notebook pinned. Selecting one used to break `import
unsloth` in 254 of the 433 shipped notebooks, because the two common pin
families (4.5x -> the 4.57.6 sidecar, 5.2/5.3 -> the 5.3.0 sidecar) both
landed on a sidecar vLLM 0.26.0 refuses. A request below the floor is
clamped UP to the lowest eligible sidecar: that is the closest version to
what the notebook asked for that this image can actually run.
* ceiling -- among the eligible sidecars pick the smallest >= the request,
because a model added in version X needs *at least* X.
A request newer than every eligible sidecar returns None -> use the base venv
(the newest 5.x), which is always vLLM-compatible."""
if not version:
return None
if version in baked:
return baked[version]
rows = _eligible()
if rows is None: # no packaging: only an exact, still-eligible match is safe
baked = _baked()
d = baked.get(version)
floor = min_version()
return d if (d and (not floor or version == floor)) else None
if not rows:
return None
for _ver, v, d in rows:
if v == version:
return d
try:
from packaging.version import Version
want = Version(version)
except Exception:
return None
ge = sorted((Version(v), d) for v, d in baked.items() if Version(v) >= want)
return ge[0][1] if ge else None
for ver, _v, d in rows:
if ver >= want:
return d
return None
def requested_version():

220
tests/python/test_docker_tf_sidecar_vllm_floor.py Normal file
View file

@ -0,0 +1,220 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-Present the Unsloth team. See /studio/LICENSE.AGPL-3.0
"""Regression guard for transformers-sidecar selection in the Unsloth Docker image.
The image runs unslothai/notebooks unchanged by refusing a notebook's
`transformers==X` install and activating a baked "sidecar" (transformers X plus
its matched huggingface_hub/tokenizers/safetensors) on sys.path instead. The
selection was a pure CEILING -- smallest baked version >= the request -- which
ignored that vLLM is version-locked to transformers. Two of the four baked
sidecars could not be imported by the baked vLLM 0.26.0 at all, and they were
exactly the two the common pins selected:
sidecar 4.57.6 ImportError: Support for Transformers v4 is deprecated and
was removed in vLLM v0.24.0
<- pins 4.48 / 4.52.3 / 4.55.4 / 4.56.1 / 4.56.2 / 4.57.x
= 241 of the 433 shipped notebooks
sidecar 5.3.0 ImportError: cannot import name 'ALLOWED_LAYER_TYPES' from
transformers.configuration_utils
<- pins 5.2.0 / 5.3.0 = 13 more notebooks
254 of 433 notebooks therefore died at `from unsloth import FastModel`, before
the first model cell. Pointing UNSLOTH_TF_SIDECAR_ROOT at an empty directory,
changing nothing else, turned two of them into clean 22/22 and 25/25 passes.
The fix is a FLOOR in front of the ceiling. Which versions are above the floor is
not hardcoded: the Dockerfile imports vllm.transformers_utils.config under every
candidate sidecar (the vLLM module that reads the transformers API -- it
reproduces both failures and needs no GPU, which matters because the build host
has none), deletes the ones that raise, and records the lowest survivor. A
request below the floor is clamped UP to the lowest eligible sidecar, which is
the closest thing to the notebook's pin the image can actually run.
Static: parses the Dockerfile and drives unsloth_nb_compat against a synthetic
sidecar root. No docker, no GPU, no network.
"""
from __future__ import annotations
import importlib.util
import re
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
DOCKERFILE = REPO_ROOT / "docker" / "Dockerfile"
COMPAT_PATH = REPO_ROOT / "docker" / "unsloth_nb_compat.py"
# Every distinct transformers pin across the 433 shipped notebooks, and the
# sidecar each must resolve to once 4.57.6 and 5.3.0 are gone.
SHIPPED_PINS = [
"4.48", "4.52.3", "4.55.4", "4.56.1", "4.56.2",
"4.57.0", "4.57.1", "4.57.3", "5.2.0", "5.3.0",
"5.5.0", "5.10.1", "5.11.0",
]
@pytest.fixture(scope = "module")
def dockerfile() -> str:
assert DOCKERFILE.is_file(), f"missing {DOCKERFILE}"
return DOCKERFILE.read_text()
@pytest.fixture(scope = "module")
def sidecar_block(dockerfile: str) -> str:
start = dockerfile.index("tf-sidecars/t_$(echo")
block = dockerfile[dockerfile.rindex("RUN set -eux", 0, start) :]
return block[: block.index("\n\n")]
def _load_compat(root, floor = None):
"""Import a fresh unsloth_nb_compat bound to a synthetic sidecar root."""
import os
prev_root = os.environ.get("UNSLOTH_TF_SIDECAR_ROOT")
prev_min = os.environ.get("UNSLOTH_TF_SIDECAR_MIN")
os.environ["UNSLOTH_TF_SIDECAR_ROOT"] = str(root)
os.environ.pop("UNSLOTH_TF_SIDECAR_MIN", None)
try:
spec = importlib.util.spec_from_file_location("unsloth_nb_compat_under_test", COMPAT_PATH)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
finally:
if prev_root is None:
os.environ.pop("UNSLOTH_TF_SIDECAR_ROOT", None)
else:
os.environ["UNSLOTH_TF_SIDECAR_ROOT"] = prev_root
if prev_min is not None:
os.environ["UNSLOTH_TF_SIDECAR_MIN"] = prev_min
return mod
@pytest.fixture()
def fixed_root(tmp_path):
"""The sidecar root the fixed Dockerfile produces: only verified sidecars,
plus the recorded floor."""
for name in ("t_5_5_0", "t_5_10_2"):
(tmp_path / name).mkdir()
(tmp_path / ".vllm_min_transformers").write_text("5.5.0\n")
return tmp_path
@pytest.fixture()
def stale_root(tmp_path):
"""A root that still carries the incompatible sidecars (a bind-mounted or
pre-fix directory). The recorded floor must keep them unselectable."""
for name in ("t_4_57_6", "t_5_3_0", "t_5_5_0", "t_5_10_2"):
(tmp_path / name).mkdir()
(tmp_path / ".vllm_min_transformers").write_text("5.5.0\n")
return tmp_path
# --------------------------------------------------------------------------
# The build must decide eligibility by measurement, not by a literal.
# --------------------------------------------------------------------------
def test_build_verifies_every_sidecar_against_the_baked_vllm(sidecar_block: str):
assert "import vllm.transformers_utils.config" in sidecar_block, (
"each baked sidecar must be proven importable by the baked vLLM; this is "
"the module that reads the transformers API and it reproduces both the "
"v4 refusal and the ALLOWED_LAYER_TYPES break"
)
def test_build_verification_needs_no_gpu(sidecar_block: str):
# `import unsloth` raises NotImplementedError("cannot find any torch
# accelerator") on the build host, so it can never be the gate.
assert "import unsloth" not in sidecar_block, (
"the sidecar gate must not import unsloth: the build host has no GPU"
)
def test_an_unverifiable_sidecar_is_deleted_not_shipped(sidecar_block: str):
assert re.search(r'DROPPED', sidecar_block), "a failed candidate must be reported"
assert re.search(r'rm -rf "\$DEST"', sidecar_block), (
"a sidecar the baked vLLM cannot import must be removed, not shipped: it "
"can never be selected safely and it costs image size"
)
def test_build_records_the_selection_floor(sidecar_block: str):
assert ".vllm_min_transformers" in sidecar_block, (
"the lowest verified version must be recorded for unsloth_nb_compat"
)
assert "sort -V | head -1" in sidecar_block, "the floor is the LOWEST survivor"
def test_build_fails_when_no_sidecar_survives(sidecar_block: str):
assert "exit 1" in sidecar_block, (
"an empty sidecar set means the whole per-notebook mechanism is dead; "
"that must fail the build rather than ship silently"
)
def test_build_skips_the_gate_when_vllm_is_absent(sidecar_block: str):
# The vLLM install is fail-soft per arch; with no vLLM there is no constraint
# and every sidecar must survive rather than the build exploding.
assert "HAVE_VLLM" in sidecar_block
def test_compat_reads_the_floor_the_build_writes():
assert ".vllm_min_transformers" in COMPAT_PATH.read_text(), (
"unsloth_nb_compat must read the floor the Dockerfile records, not a "
"literal that rots on the next vLLM bump"
)
# --------------------------------------------------------------------------
# Selection: floor, then ceiling.
# --------------------------------------------------------------------------
def test_floor_is_read_back(fixed_root):
assert _load_compat(fixed_root).min_version() == "5.5.0"
@pytest.mark.parametrize(
"pin, expected",
[
# every pin below the floor clamps UP to the lowest eligible sidecar
("4.48", "t_5_5_0"), ("4.52.3", "t_5_5_0"), ("4.55.4", "t_5_5_0"),
("4.56.1", "t_5_5_0"), ("4.56.2", "t_5_5_0"), ("4.57.0", "t_5_5_0"),
("4.57.1", "t_5_5_0"), ("4.57.3", "t_5_5_0"), ("5.2.0", "t_5_5_0"),
("5.3.0", "t_5_5_0"),
# at and above the floor, the ceiling still decides
("5.5.0", "t_5_5_0"), ("5.10.1", "t_5_10_2"),
# newer than every sidecar -> the baked transformers
("5.11.0", None),
],
)
def test_every_shipped_pin_resolves_to_a_vllm_compatible_sidecar(fixed_root, pin, expected):
got = _load_compat(fixed_root).sidecar_for(pin)
assert (Path(got).name if got else None) == expected
def test_no_shipped_pin_can_reach_an_incompatible_sidecar(stale_root):
compat = _load_compat(stale_root)
for pin in SHIPPED_PINS:
got = compat.sidecar_for(pin)
name = Path(got).name if got else None
assert name not in ("t_4_57_6", "t_5_3_0"), (
f"pin {pin} selected {name}, which the baked vLLM cannot import"
)
def test_model_tier_fallback_is_clamped_too(stale_root):
# tier_for_model maps qwen3-next and friends to 5.3.0; that tier must not
# reach the 5.3.0 sidecar either.
compat = _load_compat(stale_root)
tier = compat.tier_for_model("unsloth/Qwen3-Next-80B-A3B")
assert tier == "5.3.0"
assert Path(compat.sidecar_for(tier)).name == "t_5_5_0"
def test_an_unrecorded_floor_keeps_the_old_ceiling_behaviour(tmp_path):
# No .vllm_min_transformers (an environment that never ran the build-time
# verification): selection must not silently start dropping sidecars.
for name in ("t_4_57_6", "t_5_5_0"):
(tmp_path / name).mkdir()
compat = _load_compat(tmp_path)
assert compat.min_version() is None
assert Path(compat.sidecar_for("4.56.2")).name == "t_4_57_6"