unsloth_zoo/__init__.py guards against being imported standalone:
if "UNSLOTH_IS_PRESENT" not in os.environ:
raise ImportError("Please install Unsloth via `pip install unsloth`!")
The env var is set by unsloth/__init__.py at import time, so importing
unsloth must happen first. The old check_imports() imported xformers,
bnb, transformers, trl, peft, then unsloth_zoo -- which fired the guard
because unsloth had not been imported yet.
Reorder check_imports() to import unsloth (and unsloth_zoo) first, then
the rest. check_unsloth_import() becomes a thin re-import to keep the
"FastLanguageModel reachable" banner in the output.
Same fix the unsloth README has been recommending for years: "import
unsloth at the top of your file, before transformers/trl/peft."
Ubuntu 24.04 (noble) marks the system Python interpreter as
externally-managed per PEP 668, so:
curl get-pip.py | python
python -m pip install -U pip uv
fails inside the builder image with:
error: externally-managed-environment
This environment is externally managed
The system-level pip and uv were never used: the very next RUN creates
the venv at /opt/unsloth-venv, which bootstraps its own pip via the
ensurepip module (provided by the python3.12-venv apt package). uv is
then installed INTO the venv with the venv's pip, and used from there.
Drop the two system-pip bootstrap lines. The venv path is unchanged.
Reproduces on any Docker build of the unsloth-blackwell image against
a noble base image (which our nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04
is).
If the user is not in the 'docker' group, every docker command after the
pre-flight returns "permission denied while trying to connect to the Docker
daemon socket at /var/run/docker.sock". This used to surface as a confusing
buildx failure mid-Block-2, but the actual problem is a host permissions
issue that's settable up front.
Detect by running 'docker info' and checking its exit code (not just grep
on its output -- a permission failure prints to stderr and returns non-zero,
so the old grep-based check was a silent skip).
Also clarify the nvidia-runtime WARN: on Docker 28+ with CDI mode this is
a false positive most of the time. The real GPU-attach test is the smoke
run in Block 3a, where the container entrypoint catches missing GPUs with
an actionable message.
Docker 28 removed the legacy image builder entirely. Setting
DOCKER_BUILDKIT=1 no longer falls back to a builtin builder -- it
delegates to buildx, which then errors out if buildx isn't installed:
ERROR: BuildKit is enabled but the buildx component is missing
or broken.
The Ubuntu docker.io package omits buildx by default, so users on
that path hit this immediately. Detect missing buildx up front and
print exact install commands for apt / dnf / manual binary instead
of attempting a fallback that cannot work.
The Dockerfile uses BuildKit-only features (the # syntax=docker/dockerfile:1.7
parser directive and RUN ... <<'PY' heredocs added in dockerfile 1.3+). The
legacy builder rejects the --progress flag at the CLI level and would fail
later at the heredocs anyway.
Detect docker buildx and use it when available (preserves --progress=plain
output). Otherwise fall back to plain `docker build` with DOCKER_BUILDKIT=1
exported, which gets the BuildKit features without buildx's nicer formatting.
Reproduces the failure path seen on Docker 28.2.2 without buildx installed:
unknown flag: --progress
ERROR docker build exited 125
Single bash script that runs the full validation flow against the image:
1. Host pre-flight: docker version, nvidia-smi, nvidia-container-toolkit
runtime registered with docker.
2. Build the image (auto-detects the build context -- current dir,
docker/ subdir, or clones the docker-blackwell-build branch into
/tmp/unsloth-pr/).
3a. Smoke test: 5-step LoRA on Llama-3.2-1B-Instruct-bnb-4bit.
3b. Real workload: gpt-oss-20B fine-tuning notebook from
unslothai/notebooks, patched to max_steps=10, with the three
pre-train demo generations dropped for brevity. Auto-installs
triton_kernels at the SHA the upstream notebook pins for MXFP4.
All output is teed to /tmp/unsloth-docker-test/ (or --log-dir).
Usage:
bash docker/test_locally.sh # full run, ~15 min
bash docker/test_locally.sh --skip-notebook # blocks 1-3a only, ~3 min
bash docker/test_locally.sh --skip-build # reuse existing TAG
TAG=my:tag HF_TOKEN=hf_xxx bash docker/test_locally.sh
Each block fails fast with the exact log path to paste back.
When someone launches the unsloth container, the common failure modes are not
unsloth bugs -- they're Docker / nvidia-container-toolkit / driver issues that
surface as cryptic CUDA errors deep in torch. The entrypoint catches the three
that cover ~95% of "it doesn't work" reports up front:
1. nvidia-smi inside the container sees no GPU
-> user forgot --gpus all, or host is missing nvidia-container-toolkit
-> entrypoint prints the exact docker run flag and the toolkit install URL
2. nvidia-smi works but torch.cuda.is_available() is False
-> host driver is older than CUDA 12.8 supports
-> entrypoint prints the minimum driver version per architecture
3. compute capability < sm_80
-> entrypoint prints the supported architecture table and exits
Each check fails with a clear, actionable message rather than a stack trace.
Set UNSLOTH_SKIP_GPU_CHECK=1 to bypass (for docs builds, offline tooling, CI).
run.sh wraps `docker run` with the flags people most often forget:
--gpus all (without it, the new entrypoint refuses to start)
--ipc=host (DataLoader workers need >64MB shm)
--ulimit memlock=-1 (NCCL + CUDA pinned host buffers)
--ulimit stack=64MB (some torch kernels OOM the default 8MB stack)
Plus it mounts the host HF cache + Triton JIT cache so model downloads and
compiled kernels persist across container runs, and forwards HF_TOKEN /
WANDB_API_KEY / UNSLOTH_LICENSE only when they are set on the host.
Usage:
bash docker/run.sh # interactive python REPL
bash docker/run.sh bash # shell in container
bash docker/run.sh python /workspace/smoke_test.py
bash docker/run.sh python /workspace/host/train.py # $PWD mounted at /workspace/host
Verified locally:
- No GPU visible: entrypoint refuses with driver-version message, exit 1
- B200 sm_100 visible: entrypoint prints GPU banner, exits cleanly into the
user command (rc=0)
Adds a multi-stage Dockerfile producing an image that works on Ampere through
Blackwell (sm_80 through sm_120: A100, RTX 30/40, H100, B100/B200, RTX 50-series,
RTX 6000 Pro Blackwell). The build itself requires no GPU at all and runs on a
free GitHub-hosted ubuntu-latest runner.
How the GPU-less build works:
1. cu128 PyTorch wheels are fat binaries. torch._C._cuda_getArchFlags() returns
'sm_70 sm_75 sm_80 sm_86 sm_90 sm_100 sm_120' regardless of which GPU
compiled the image, because the wheels are cross-compiled upstream by the
PyTorch team.
2. All deps resolve in a single uv pip install pass with explicit pins
(torch==2.10.0, --extra-index-url cu128, no --torch-backend=auto, no
install.sh). This prevents the silent cu cascade where bitsandbytes'
transitive cuda-toolkit==13 dep upgrades torch to 2.12+cu130 in a later
resolver pass, leaving xformers and other cu128 wheels stranded.
3. Build-time verification uses package metadata (importlib.metadata.version)
and the raw torch._C._cuda_getArchFlags() accessor. We deliberately avoid
import unsloth at build time because unsloth.__init__ calls
torch.cuda.get_device_properties(0), which requires an actual CUDA device
and is not bypassable. Import-time correctness is exercised at deploy time
by smoke_test.py with --gpus all.
4. UNSLOTH_COMPILE_DISABLE=1 and CUDA_VISIBLE_DEVICES="" during the build stage
prevent any code path from JIT-compiling kernels for the build host's
compute capability and baking the resulting cache into the image. The
deploy GPU produces its own cache on first use.
Other notes:
- --index-strategy unsafe-best-match is needed because the PyTorch wheel index
serves an old requests==2.28.1 that conflicts with datasets>=2.32.2, which
the default first-index-wins strategy rejects.
- Extra is cu128-ampere-torch2100 (ampere precedes the torch version in the
pyproject ordering).
- No flash-attn in the base image. FA3 is hard-refused on Blackwell upstream
and unsloth gracefully falls back to xformers + SDPA. Users on Ampere /
Ada / Hopper who want FA2 can pip install flash-attn on top.
- Two stages: nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04 for the build,
-cudnn-runtime for the deploy image. No nvcc in the published image.
- A lockfile is emitted at /opt/unsloth-venv/requirements.lock.txt inside
the image and can be extracted with docker/freeze.sh for byte-identical
rebuilds even after PyPI moves on.
CI workflow .github/workflows/docker-publish.yml:
- Builds on ubuntu-latest on every push to main, every tag, weekly via cron,
and manually via workflow_dispatch. Pushes to docker.io/unsloth/unsloth
with cache via type=gha.
- Optional smoke-test job runs on a self-hosted GPU runner if vars.HAS_GPU_RUNNER
is set; skipped otherwise. End-to-end verification on sm_120 hardware is a
nice-to-have, not a publish blocker.
Validation:
- Install path validated on a B200 host with CUDA_VISIBLE_DEVICES="" set
(simulating the GPU-less CI runner): torch 2.10.0+cu128 holds, xformers
0.0.34, bitsandbytes 0.49.2, triton 3.6.0, transformers 5.5.0, trl 0.24.0,
peft 0.19.1, accelerate 1.13.0. Arch flags include sm_100 and sm_120.
- Runtime path validated end-to-end on B200: smoke_test.py imports unsloth,
loads Llama-3.2-1B-Instruct-bnb-4bit in 4-bit, completes 5 LoRA steps with
loss decreasing 4.11 -> 3.75. xformers fallback active as designed.
Files:
- docker/Dockerfile multi-stage cu128 build
- docker/build.sh local build wrapper
- docker/freeze.sh extract lockfile from a built image
- docker/smoke_test.py runtime verification, run with --gpus all
- docker/.dockerignore
- .github/workflows/docker-publish.yml
