4bfb4b891a
HF Hub does not act as a generic OCI registry for arbitrary Docker images -- the registry.hf.space endpoint only serves images that Spaces have built, not images pushed by `docker push`. So we cannot do `docker push huggingface.co/user/repo:tag` for an Unsloth image. For cross-host testing where we want one canonical place to pull from (and Docker Hub credentials are not yet configured), wrap the manual flow into push/pull-shaped commands: hf_push.sh: docker save | pigz | huggingface-cli upload hf_pull.sh: huggingface-cli download | gunzip | docker load This is approximation, not real OCI semantics -- every push uploads the full ~4 GB blob, no layer dedup, no manifest negotiation. Good for testing across A100 / H100 / RTX 6000 boxes; the real release should go through .github/workflows/docker-publish.yml to Docker Hub, which gets layer dedup, multi-arch manifest support, and standard `docker pull` UX for users. Usage: bash docker/hf_push.sh unsloth-blackwell:test danielhanchen/unsloth-blackwell-docker bash docker/hf_pull.sh danielhanchen/unsloth-blackwell-docker unsloth-blackwell-test.tar.gz unsloth-blackwell:test
41 lines
1.6 KiB
Bash
Executable file
41 lines
1.6 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Simulate `docker pull <image>` against a Hugging Face Hub model repo.
|
|
#
|
|
# Counterpart to docker/hf_push.sh -- downloads the tar.gz blob from the HF
|
|
# repo and `docker load`s it.
|
|
#
|
|
# Usage:
|
|
# bash docker/hf_pull.sh <hf_repo> [<blob>] [<verify_tag>]
|
|
# bash docker/hf_pull.sh danielhanchen/unsloth-blackwell-docker unsloth-blackwell-test.tar.gz unsloth-blackwell:test
|
|
#
|
|
# Requires: docker, pigz (or gzip), huggingface-cli logged in (read scope is
|
|
# sufficient for public repos).
|
|
set -euo pipefail
|
|
|
|
REPO="${1:?usage: hf_pull.sh <hf_repo> [<blob>] [<verify_tag>]}"
|
|
BLOB="${2:-unsloth-blackwell.tar.gz}"
|
|
VERIFY="${3:-}"
|
|
WORK="${HF_PULL_TMP:-/tmp}"
|
|
|
|
command -v docker >/dev/null || { echo "ERROR: docker not on PATH"; exit 1; }
|
|
command -v huggingface-cli >/dev/null || { echo "ERROR: huggingface-cli not on PATH (pip install -U huggingface_hub)"; exit 1; }
|
|
DECOMPRESSOR=$(command -v pigz || command -v gzip) || { echo "ERROR: need pigz or gzip"; exit 1; }
|
|
|
|
DEST="${WORK}/$(basename "${BLOB}")"
|
|
echo ">> downloading ${REPO}/${BLOB} -> ${DEST}"
|
|
huggingface-cli download "${REPO}" "${BLOB}" --repo-type=model --local-dir "${WORK}"
|
|
ls -lh "${DEST}"
|
|
|
|
echo ">> loading into docker (using ${DECOMPRESSOR##*/})"
|
|
"${DECOMPRESSOR}" -d -c "${DEST}" | docker load
|
|
|
|
if [[ -n "${VERIFY}" ]]; then
|
|
if docker image inspect "${VERIFY}" >/dev/null 2>&1; then
|
|
echo ">> verified: ${VERIFY} is loaded"
|
|
docker image inspect --format 'image_id={{.Id}} size={{.Size}}' "${VERIFY}"
|
|
else
|
|
echo "WARN: expected tag ${VERIFY} not found after load. docker images:"
|
|
docker images
|
|
exit 1
|
|
fi
|
|
fi
|