ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ci(inference): port main's --local-dir gguf-cache pattern to tool-calling jobs

Browse source 
The Tool calling Tests jobs were the worst offender for HF_HOME cache
inflation. Same Qwen3.5-2B-UD-Q4_K_XL.gguf that's 1.28 GiB on disk
was landing as ~4.7 GiB in the actions/cache archive across all three
OS jobs:

  Linux Qwen IQ3_XXS  889 MB GGUF -> 4313 MB cache (4.85x)
  Mac   Qwen Q4_K_XL 1278 MB GGUF -> 4692 MB cache (3.7x)
  Win   Qwen Q4_K_XL 1278 MB GGUF -> 4692 MB cache (3.7x, 211 s upload)

The 3-5x inflation comes from caching the entire HF_HOME tree:
xet chunks + blobs + snapshots are all stored, plus on Windows
snapshot symlinks materialise as full copies (NTFS symlinks need
admin). main branch has long since moved to a leaner pattern --
hf download with --local-dir gguf-cache stores the flat .gguf only
and Studio's /api/inference/load takes an absolute file path.

Port main's pattern back to PR 5312's three tool-calling jobs:

  Cache step path:  hf-cache       -> gguf-cache
  Cache step key:   <os>-hf-<repo>-<variant>-v1
                 -> <os>-gguf-<repo>-<file>-v1
  Download:         hf download <repo> <file>
                 -> hf download <repo> <file> --local-dir gguf-cache
  Load:             model_path=<repo>, gguf_variant=<variant>
                 -> model_path=$GITHUB_WORKSPACE/gguf-cache/<file>

Cache size drops 4.7 GiB -> 1.28 GiB; Post Cache step time drops
from 211 s -> ~60 s on first runs, and the steady-state cache-hit
restore is also faster (smaller archive).

Windows path handling: GITHUB_WORKSPACE on windows-latest is a
backslash path ("D:\a\unsloth\unsloth"), which would explode JSON
escaping if embedded directly. Use bash parameter expansion to
flip backslashes to forward slashes; pathlib.Path on Windows accepts
forward slashes natively, so Studio's loader sees a normal path.

Trade-off: the tool-calling jobs no longer exercise Studio's
gguf_variant resolution path. The OpenAI/Anth and JSON+images jobs
still cover that path on every PR push, so coverage of the variant-
to-file mapping is retained at the workflow level.

The OpenAI/Anth and JSON+images jobs intentionally stay on HF_HOME --
their GGUFs are smaller (gemma-3-270m at ~250 MB, gemma-4-E2B at
~2.4 GB + mmproj). The post-step upload cost for those is dominated
by their actual file size, not the inflation factor; switching them
adds churn without proportional savings.
This commit is contained in:
Daniel Han 2026-05-08 09:10:57 +00:00
commit 1b7106162a
3 changed files with 68 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

32
.github/workflows/studio-inference-smoke.yml vendored
View file

@ -289,11 +289,19 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 25
env:
# Tool calling is the highest-volume GGUF in this workflow
# (Qwen3.5-2B at IQ3_XXS = ~890 MiB). Caching HF_HOME would
# store xet chunks + blobs + snapshots = ~4 GiB compressed --
# 4-5x file-size inflation, dominated by xet chunks. Use main's
# `--local-dir gguf-cache` pattern to cache the flat .gguf only.
# Studio's /api/inference/load accepts either a HF repo (which
# uses HF_HOME) or an absolute file path; passing the absolute
# path keeps the test off HF_HOME entirely so the cache size
# tracks the GGUF file 1:1. The OpenAI/Anth and JSON+images
# jobs still cover the gguf_variant resolution path.
GGUF_REPO: unsloth/Qwen3.5-2B-GGUF
GGUF_VARIANT: UD-IQ3_XXS
GGUF_FILE: Qwen3.5-2B-UD-IQ3_XXS.gguf
STUDIO_PORT: '18889'
HF_HOME: ${{ github.workspace }}/hf-cache
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
@ -314,20 +322,20 @@ jobs:
python-version: '3.12'
cache: 'pip'
- name: Cache HF_HOME for ${{ env.GGUF_REPO }}
id: cache-hf
- name: Cache GGUF model file
id: cache-gguf
uses: actions/cache@v4
with:
path: hf-cache
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
path: gguf-cache
key: ${{ runner.os }}-gguf-${{ env.GGUF_REPO }}-${{ env.GGUF_FILE }}-v1
- name: Prime HF_HOME with the GGUF
if: steps.cache-hf.outputs.cache-hit != 'true'
- name: Download GGUF if cache miss
if: steps.cache-gguf.outputs.cache-hit != 'true'
run: |
python -m pip install --upgrade huggingface_hub hf_transfer
mkdir -p hf-cache
mkdir -p gguf-cache
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download "$GGUF_REPO" "$GGUF_FILE"
hf download "$GGUF_REPO" "$GGUF_FILE" --local-dir gguf-cache
- name: Install Studio (--local, --no-torch)
env:
@ -376,10 +384,12 @@ jobs:
-H 'content-type: application/json' \
-d "{\"username\":\"unsloth\",\"password\":\"$NEW\"}" | jq -r .access_token)
echo "API_KEY=$TOKEN" >> "$GITHUB_ENV"
GGUF_PATH="$GITHUB_WORKSPACE/gguf-cache/${GGUF_FILE}"
ls -lh "$GGUF_PATH"
curl -fs -X POST "http://127.0.0.1:${STUDIO_PORT}/api/inference/load" \
-H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
--max-time 600 \
-d "{\"model_path\":\"$GGUF_REPO\",\"gguf_variant\":\"$GGUF_VARIANT\",\"is_lora\":false,\"max_seq_length\":2048}" \
-d "{\"model_path\":\"$GGUF_PATH\",\"is_lora\":false,\"max_seq_length\":2048}" \
| jq '{status, display_name}'
- name: Tool calling, server-side tools, thinking on/off

35
.github/workflows/studio-mac-inference-smoke.yml vendored
View file

@ -291,16 +291,17 @@ jobs:
runs-on: macos-14
timeout-minutes: 25
env:
# Tool calling is the highest-volume GGUF in this workflow
# (Qwen3.5-2B at Q4_K_XL = ~1.28 GiB on Mac, where IQ3_XXS
# collapses for tool-call grammar under Metal at temperature=0).
# Caching HF_HOME stores xet chunks + blobs + snapshots = ~4.6
# GiB compressed -- 3.6x file-size inflation. Use main's
# `--local-dir gguf-cache` pattern to cache the flat .gguf only.
# The OpenAI/Anth and JSON+images jobs still cover the
# gguf_variant resolution path.
GGUF_REPO: unsloth/Qwen3.5-2B-GGUF
# The Linux smoke uses UD-IQ3_XXS, but on Mac Metal that quant
# collapses for tool-call grammar at temperature=0 (model emits
# repeated padding tokens until max_tokens). UD-Q4_K_XL is the
# smallest published variant that produces well-formed tool
# calls + non-pathological text on M1.
GGUF_VARIANT: UD-Q4_K_XL
GGUF_FILE: Qwen3.5-2B-UD-Q4_K_XL.gguf
STUDIO_PORT: '18898'
HF_HOME: ${{ github.workspace }}/hf-cache
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
@ -315,20 +316,20 @@ jobs:
python-version: '3.12'
cache: 'pip'
- name: Cache HF_HOME for ${{ env.GGUF_REPO }}
id: cache-hf
- name: Cache GGUF model file
id: cache-gguf
uses: actions/cache@v4
with:
path: hf-cache
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
path: gguf-cache
key: ${{ runner.os }}-gguf-${{ env.GGUF_REPO }}-${{ env.GGUF_FILE }}-v1
- name: Prime HF_HOME with the GGUF
if: steps.cache-hf.outputs.cache-hit != 'true'
- name: Download GGUF if cache miss
if: steps.cache-gguf.outputs.cache-hit != 'true'
run: |
python -m pip install --upgrade huggingface_hub hf_transfer
mkdir -p hf-cache
mkdir -p gguf-cache
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download "$GGUF_REPO" "$GGUF_FILE"
hf download "$GGUF_REPO" "$GGUF_FILE" --local-dir gguf-cache
- name: Install Studio (--local, --no-torch)
env:
@ -385,10 +386,12 @@ jobs:
-H 'content-type: application/json' \
-d "{\"username\":\"unsloth\",\"password\":\"$NEW\"}" | jq -r .access_token)
echo "API_KEY=$TOKEN" >> "$GITHUB_ENV"
GGUF_PATH="$GITHUB_WORKSPACE/gguf-cache/${GGUF_FILE}"
ls -lh "$GGUF_PATH"
curl -fs -X POST "http://127.0.0.1:${STUDIO_PORT}/api/inference/load" \
-H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
--max-time 600 \
-d "{\"model_path\":\"$GGUF_REPO\",\"gguf_variant\":\"$GGUF_VARIANT\",\"is_lora\":false,\"max_seq_length\":2048}" \
-d "{\"model_path\":\"$GGUF_PATH\",\"is_lora\":false,\"max_seq_length\":2048}" \
| jq '{status, display_name}'
- name: Tool calling, server-side tools, thinking on/off

43
.github/workflows/studio-windows-inference-smoke.yml vendored
View file

@ -342,15 +342,19 @@ jobs:
run:
shell: bash
env:
# Tool calling is the highest-volume GGUF in this workflow
# (Qwen3.5-2B at Q4_K_XL = ~1.28 GiB). The previous HF_HOME
# cache stored xet chunks + blobs + snapshots = ~4.7 GiB --
# 3.7x file-size inflation, dominating the post-step upload
# (211 s on first run; subsequent runs hit the cache, but the
# one-time cost recurs every time the cache key bumps). Use
# main's `--local-dir gguf-cache` pattern: cache the flat .gguf
# only, pass an absolute path to Studio's /api/inference/load.
# The OpenAI/Anth and JSON+images jobs still cover the
# gguf_variant resolution path.
GGUF_REPO: unsloth/Qwen3.5-2B-GGUF
# Mirror the Mac job's variant choice (UD-Q4_K_XL). On the
# CPU-only windows-latest runner the smaller IQ3_XXS quant
# behaves OK, but keeping parity with the Mac job means one HF
# asset shared across runners.
GGUF_VARIANT: UD-Q4_K_XL
GGUF_FILE: Qwen3.5-2B-UD-Q4_K_XL.gguf
STUDIO_PORT: '18898'
HF_HOME: ${{ github.workspace }}/hf-cache
# Force UTF-8 for stdio (Windows defaults to cp1252; hf
# download / Studio CLI print "✓" checkmarks and crash
# otherwise).
@ -369,20 +373,20 @@ jobs:
with:
python-version: '3.12'
- name: Cache HF_HOME for ${{ env.GGUF_REPO }}
id: cache-hf
- name: Cache GGUF model file
id: cache-gguf
uses: actions/cache@v4
with:
path: hf-cache
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
path: gguf-cache
key: ${{ runner.os }}-gguf-${{ env.GGUF_REPO }}-${{ env.GGUF_FILE }}-v1
- name: Prime HF_HOME with the GGUF
if: steps.cache-hf.outputs.cache-hit != 'true'
- name: Download GGUF if cache miss
if: steps.cache-gguf.outputs.cache-hit != 'true'
run: |
python -m pip install --upgrade huggingface_hub hf_transfer
mkdir -p hf-cache
mkdir -p gguf-cache
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download "$GGUF_REPO" "$GGUF_FILE"
hf download "$GGUF_REPO" "$GGUF_FILE" --local-dir gguf-cache
- name: Pre-install Windows tweaks (npm 11 + Defender exclusions)
shell: pwsh
@ -506,10 +510,19 @@ jobs:
-H 'content-type: application/json' \
-d "{\"username\":\"unsloth\",\"password\":\"$NEW\"}" | jq -r .access_token)
echo "API_KEY=$TOKEN" >> "$GITHUB_ENV"
# GITHUB_WORKSPACE on windows-latest is a Windows path with
# backslashes ("D:\a\unsloth\unsloth"). Bash handles it as a
# raw string, but we cannot embed `\a` etc. in JSON without
# JSON-string-escaping every backslash. Replace `\` with `/`
# via bash parameter expansion -- pathlib.Path on Windows
# accepts forward slashes natively, so Studio's loader sees
# a normal path.
GGUF_PATH="${GITHUB_WORKSPACE//\\//}/gguf-cache/${GGUF_FILE}"
ls -lh "$GGUF_PATH"
curl -fs -X POST "http://127.0.0.1:${STUDIO_PORT}/api/inference/load" \
-H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
--max-time 600 \
-d "{\"model_path\":\"$GGUF_REPO\",\"gguf_variant\":\"$GGUF_VARIANT\",\"is_lora\":false,\"max_seq_length\":2048}" \
-d "{\"model_path\":\"$GGUF_PATH\",\"is_lora\":false,\"max_seq_length\":2048}" \
| jq '{status, display_name}'
- name: Tool calling, server-side tools, thinking on/off