ci(mlx): add Studio prebuilt llama.cpp + GGUF inference on Mac M1
New workflow step exercises the same code path Studio's setup.sh
takes on macOS: studio/install_llama_prebuilt.py with
--published-repo ggml-org/llama.cpp and --published-release-tag
b9049 (latest llama.cpp release at time of writing). The installer
fetches llama-b9049-bin-macos-arm64.tar.gz -- universal Apple
Silicon arm64 build (M1/M2/M3/M4 all OK).
After install, downloads unsloth/gemma-3-270m-it-GGUF Q4_K_M (~241
MB) from HuggingFace and runs the prebuilt llama-cli on it with a
fixed seed + greedy sampling. Asserts the prompt echo "Hello"
appears in stdout. If the install or inference fails, that's an
Unsloth/Studio-side bug.
The b9049 release publishes four macOS-related assets:
* macos-arm64 -- universal Apple Silicon, M1/M2/M3/M4 OK.
Studio picks this asset by default.
* macos-arm64-kleidiai -- KleidiAI dispatches at runtime, falls
back where ISA features are missing on
older Apple Silicon (e.g. M1 lacks I8MM),
so it ALSO runs on M1 -- Studio just
doesn't pick this variant by default.
* macos-x64 -- Intel-only, would require Rosetta 2 on
M1; we deliberately avoid this.
* iOS XCFramework -- iOS-app artifact, not a macOS desktop
build.
Step uses a separate install dir (~/.unsloth-studio-prebuilt-test/
llama.cpp) so it does not collide with the existing MLX export
round-trip's save_pretrained_gguf path that clones+builds llama.cpp
from source under ~/.unsloth/llama.cpp.
This commit is contained in:
parent
df81c35d34
commit
4526641cf4
1 changed files with 86 additions and 0 deletions
86
.github/workflows/mlx-ci.yml
vendored
86
.github/workflows/mlx-ci.yml
vendored
|
|
@ -211,6 +211,92 @@ jobs:
|
|||
tests/studio/test_is_mlx_dispatch_gate.py \
|
||||
tests/studio/test_mlx_training_worker_behaviors.py
|
||||
|
||||
# Studio prebuilt llama.cpp install + GGUF inference. Drives the
|
||||
# exact path Studio's setup.sh takes on macOS: invokes
|
||||
# studio/install_llama_prebuilt.py with --published-repo
|
||||
# ggml-org/llama.cpp and --published-release-tag b9049 (the
|
||||
# latest llama.cpp release at the time this step was added; bump
|
||||
# via UNSLOTH_LLAMA_TAG / DEFAULT_LLAMA_TAG when refreshing).
|
||||
# The installer downloads llama-b9049-bin-macos-arm64.tar.gz,
|
||||
# which is the universal Apple Silicon (arm64) build -- the
|
||||
# same artifact works on M1/M2/M3/M4 because llama.cpp compiles
|
||||
# against the ARMv8.2 baseline.
|
||||
#
|
||||
# The b9049 release also publishes:
|
||||
# - llama-b9049-bin-macos-arm64-kleidiai.tar.gz
|
||||
# KleidiAI dispatches at runtime; on M1 it falls back where
|
||||
# ISA features (e.g. I8MM) are missing, so this asset also
|
||||
# runs on M1 -- Studio just doesn't choose it by default.
|
||||
# - llama-b9049-bin-macos-x64.tar.gz
|
||||
# Intel-only; would only run on M1 via Rosetta 2 emulation,
|
||||
# which we explicitly avoid.
|
||||
# - iOS XCFramework
|
||||
# iOS-app build artifact, unrelated to a macOS desktop CI.
|
||||
#
|
||||
# After install, downloads a small published GGUF
|
||||
# (unsloth/gemma-3-270m-it-GGUF, Q4_K_M) from HuggingFace and
|
||||
# runs the prebuilt llama-cli on it. Asserts the prompt echo
|
||||
# appears in stdout. If the install fails OR the binary exits
|
||||
# non-zero, that's an Unsloth/Studio bug.
|
||||
- name: Studio prebuilt llama.cpp install + GGUF inference (Mac M1)
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
INSTALL_DIR="$HOME/.unsloth-studio-prebuilt-test/llama.cpp"
|
||||
rm -rf "$INSTALL_DIR"
|
||||
python studio/install_llama_prebuilt.py \
|
||||
--install-dir "$INSTALL_DIR" \
|
||||
--published-repo ggml-org/llama.cpp \
|
||||
--published-release-tag b9049
|
||||
|
||||
LLAMA_CLI=""
|
||||
for c in \
|
||||
"$INSTALL_DIR/build/bin/llama-cli" \
|
||||
"$INSTALL_DIR/llama-cli" \
|
||||
"$INSTALL_DIR/bin/llama-cli"; do
|
||||
if [ -x "$c" ]; then LLAMA_CLI="$c"; break; fi
|
||||
done
|
||||
if [ -z "$LLAMA_CLI" ]; then
|
||||
echo "::error::llama-cli not found under $INSTALL_DIR"
|
||||
find "$INSTALL_DIR" -maxdepth 4 -type f -name 'llama-*' || true
|
||||
exit 1
|
||||
fi
|
||||
echo "found llama-cli at: $LLAMA_CLI"
|
||||
"$LLAMA_CLI" --version || true
|
||||
|
||||
mkdir -p /tmp/ggufs
|
||||
python -c "
|
||||
from huggingface_hub import hf_hub_download
|
||||
p = hf_hub_download(
|
||||
'unsloth/gemma-3-270m-it-GGUF',
|
||||
'gemma-3-270m-it-Q4_K_M.gguf',
|
||||
local_dir = '/tmp/ggufs',
|
||||
)
|
||||
print('downloaded:', p)
|
||||
"
|
||||
|
||||
PROMPT="Hello, my name is"
|
||||
echo "=== llama-cli inference ==="
|
||||
OUT=$("$LLAMA_CLI" \
|
||||
-m /tmp/ggufs/gemma-3-270m-it-Q4_K_M.gguf \
|
||||
-p "$PROMPT" \
|
||||
-n 16 \
|
||||
--temp 0 \
|
||||
--seed 3407 \
|
||||
-no-cnv \
|
||||
--no-warmup 2>&1) || {
|
||||
echo "::error::llama-cli exited non-zero"
|
||||
echo "$OUT" | head -80
|
||||
exit 1
|
||||
}
|
||||
echo "$OUT" | tail -40
|
||||
if ! echo "$OUT" | grep -q "Hello"; then
|
||||
echo "::error::llama-cli output did not contain the prompt echo 'Hello'"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: Studio prebuilt llama.cpp on Mac M1 + GGUF inference works"
|
||||
|
||||
# Real MLX training + inference smoke test. Trains
|
||||
# unsloth/gemma-3-270m-it for 7 deterministic LoRA steps
|
||||
# (batch_size=2, gradient_accumulation_steps=3) on a single
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue