ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

studio/ci: harden HF_HOME cache against actions/cache v5 silent restore failures (#5396)

Browse source 
* studio/ci: harden HF_HOME/GGUF cache against actions/cache@v5 silent restore failures

actions/cache@v5 has a recurring flake where it logs "Cache hit for: <key>"
and then exits non-zero in well under a second without actually extracting
the archive (see actions/cache#1621 and github community discussion
#163260). When that happens to the JSON, images job the cache step is
marked failure, all downstream steps are skipped (only the if: always()
ones run), and the job never even tries to install Studio. Example:
run 25713577488 / job 75498714730 took 23 s total and bailed at the cache
step despite the cache having been written successfully ~30 min earlier.

Replace the single-step actions/cache usage in all three jobs with the
documented restore + save split:

  - actions/cache/restore with continue-on-error: true on the way in
  - Prime/Download step gated on cache-hit != 'true' OR outcome != 'success'
    so the silent-failure path re-downloads from HF instead of skipping
  - actions/cache/save on the way out, gated on the Prime step's outcome
    so we only write a fresh entry when we actually rebuilt the directory

Same SHA-pinned action (v5.0.5), same cache keys, same paths -- so existing
cache entries keep matching. Only behavior change is that a transient
restore-side failure now falls through to a re-download instead of failing
the job.

* studio/ci: add continue-on-error to the new actions/cache/save steps

Per review of PR 5396: a save-side flake (upload timeout, 5xx from the
cache backend, future-fatal ReserveCacheError) is strictly recoverable
because next run just re-downloads, so it should never fail the job.

Today actions/cache/save@v5.0.5 already swallows ReserveCacheError as a
non-fatal warning, so this is defense in depth. Aligns the save steps
with their matching restore steps which already mask transient failures
via continue-on-error.

* studio/ci: drop continue-on-error from cache/save steps

Reverting the save-side continue-on-error addition from the previous
commit. cache/save@v5.0.5 already swallows ReserveCacheError (the most
common save flake) as a non-fatal core.info, so the mask was rarely
doing anything in practice. A real save-side failure (cache backend
outage, blob server 5xx storm) is signal we want to keep -- without it
we would see slow CI for days without knowing the cache layer is
broken. If save flakes start showing up in practice we add this back
with concrete evidence.

The restore-side continue-on-error stays -- that is the actual fix
for the actions/cache#1621 silent-restore-failure mode. Also strip
the now-stale "continue-on-error" comments above the three save
blocks.

* studio/ci: clarify cache split header comment

Per re-review: the prior wording "the Save step re-uploads on the way out"
implied actions/cache/save would replace a broken existing cache entry,
which is wrong -- cache keys are immutable, so save logs a warning when
the key already exists and the corrupted entry stays until the -v1
suffix is bumped. Rewrite to spell out the actual behavior and the
escape hatch (bump the suffix).
This commit is contained in:
Daniel Han 2026-05-12 05:47:44 -07:00 • committed by GitHub
commit 040b80a60e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 60 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

69
.github/workflows/studio-windows-inference-smoke.yml vendored
View file

@ -73,15 +73,29 @@ jobs:
with:
python-version: '3.12'
- name: Cache HF_HOME for ${{ env.GGUF_REPO }}
# Split restore + save (rather than the one-step actions/cache) so a
# transient restore-side failure does not kill the whole job. v5 has a
# known flake where it logs "Cache hit for: <key>" and then exits
# non-zero without actually extracting the archive (see
# actions/cache#1621 and github community discussion #163260).
# continue-on-error on restore masks that failure so the Prime step
# below can re-download from HF and the job keeps running. Save then
# populates the cache key on a real miss only; cache keys are
# immutable, so a corrupted cached entry persists until the -v1
# suffix below is bumped.
- name: Restore HF_HOME cache for ${{ env.GGUF_REPO }}
id: cache-hf
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
continue-on-error: true
with:
path: hf-cache
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
- name: Prime HF_HOME with the GGUF
if: steps.cache-hf.outputs.cache-hit != 'true'
id: prime-hf
# Run on a real cache miss AND on the silent-restore-failure mode
# described above (outcome != success).
if: steps.cache-hf.outputs.cache-hit != 'true' || steps.cache-hf.outcome != 'success'
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
@ -90,6 +104,16 @@ jobs:
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download "$GGUF_REPO" "$GGUF_FILE"
- name: Save HF_HOME cache for ${{ env.GGUF_REPO }}
# Only write a fresh cache entry when we actually rebuilt the
# directory (Prime ran and succeeded). Skipping when Prime is
# skipped avoids "already exists" save warnings on the happy path.
if: always() && steps.prime-hf.outcome == 'success'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: hf-cache
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
- name: Pre-install Windows tweaks (npm 11 + Defender exclusions)
shell: pwsh
# See studio-windows-update-smoke.yml for the full rationale.
@ -375,15 +399,20 @@ jobs:
with:
python-version: '3.12'
- name: Cache GGUF model file
# Split restore + save so a transient restore-side failure does not
# kill the whole job. See the matching block in the tool-calling job
# above for the full rationale (actions/cache#1621).
- name: Restore GGUF model cache
id: cache-gguf
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
continue-on-error: true
with:
path: gguf-cache
key: ${{ runner.os }}-gguf-${{ env.GGUF_REPO }}-${{ env.GGUF_FILE }}-v1
- name: Download GGUF if cache miss
if: steps.cache-gguf.outputs.cache-hit != 'true'
id: download-gguf
if: steps.cache-gguf.outputs.cache-hit != 'true' || steps.cache-gguf.outcome != 'success'
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
@ -392,6 +421,13 @@ jobs:
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download "$GGUF_REPO" "$GGUF_FILE" --local-dir gguf-cache
- name: Save GGUF model cache
if: always() && steps.download-gguf.outcome == 'success'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: gguf-cache
key: ${{ runner.os }}-gguf-${{ env.GGUF_REPO }}-${{ env.GGUF_FILE }}-v1
- name: Pre-install Windows tweaks (npm 11 + Defender exclusions)
shell: pwsh
# See studio-windows-update-smoke.yml for the full rationale.
@ -771,15 +807,23 @@ jobs:
with:
python-version: '3.12'
- name: Cache HF_HOME for ${{ env.GGUF_REPO }} (model + mmproj)
# Split restore + save so a transient restore-side failure does not
# kill the whole job. See the matching block in the tool-calling job
# for the full rationale (actions/cache#1621). This is the block that
# actually broke in run 25713577488: "Cache hit for: <key>" was
# logged, the step exited non-zero in ~0.3 s without extracting the
# 3.4 GiB archive, and steps 6-15 were skipped.
- name: Restore HF_HOME cache for ${{ env.GGUF_REPO }} (model + mmproj)
id: cache-hf
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
continue-on-error: true
with:
path: hf-cache
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-${{ env.MMPROJ_FILE }}-v1
- name: Prime HF_HOME with the GGUF + mmproj
if: steps.cache-hf.outputs.cache-hit != 'true'
id: prime-hf
if: steps.cache-hf.outputs.cache-hit != 'true' || steps.cache-hf.outcome != 'success'
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
@ -790,6 +834,13 @@ jobs:
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download "$GGUF_REPO" "$MMPROJ_FILE"
- name: Save HF_HOME cache for ${{ env.GGUF_REPO }} (model + mmproj)
if: always() && steps.prime-hf.outcome == 'success'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: hf-cache
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-${{ env.MMPROJ_FILE }}-v1
- name: Pre-install Windows tweaks (npm 11 + Defender exclusions)
shell: pwsh
# See studio-windows-update-smoke.yml for the full rationale.