studio: authenticate HF downloads across Studio CI workflows (#5370)
The Mac json-images job (run 25664825326) hit the 30 min step budget while downloading 4 GiB of GGUF assets unauthenticated. The log shows the explicit "You are sending unauthenticated requests to the HF Hub" warning followed by 30 min of zero progress, then job cancellation. macos-14, ubuntu-latest, and windows-latest runners share NAT egress IP pools across the whole GitHub Actions fleet, so the anonymous per-IP rate limit kicks in well before the file size alone would suggest. An authenticated token shifts the budget to per-user. Add HF_TOKEN: secrets.HF_TOKEN to every hf download step across the nine studio CI workflows that pull from HF. The env is scoped to the download step only, not the job, so every other step still runs without HF_TOKEN in its environment and the GitHub secret-masking layer handles log scrubbing. For the Mac json-images step specifically, the model and mmproj downloads now run in parallel under wait, and an ls -lhL after the wait surfaces a partial download as an obvious failure instead of a silent 30 min timeout on the next inference/load call.
This commit is contained in:
parent
379f5a5aa6
commit
8c606a70b5
9 changed files with 41 additions and 2 deletions
2
.github/workflows/studio-api-smoke.yml
vendored
2
.github/workflows/studio-api-smoke.yml
vendored
|
|
@ -80,6 +80,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
6
.github/workflows/studio-inference-smoke.yml
vendored
6
.github/workflows/studio-inference-smoke.yml
vendored
|
|
@ -94,6 +94,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
@ -331,6 +333,8 @@ jobs:
|
|||
|
||||
- name: Download GGUF if cache miss
|
||||
if: steps.cache-gguf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p gguf-cache
|
||||
|
|
@ -637,6 +641,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF + mmproj
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
2
.github/workflows/studio-mac-api-smoke.yml
vendored
2
.github/workflows/studio-mac-api-smoke.yml
vendored
|
|
@ -65,6 +65,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
19
.github/workflows/studio-mac-inference-smoke.yml
vendored
19
.github/workflows/studio-mac-inference-smoke.yml
vendored
|
|
@ -88,6 +88,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
@ -325,6 +327,8 @@ jobs:
|
|||
|
||||
- name: Download GGUF if cache miss
|
||||
if: steps.cache-gguf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p gguf-cache
|
||||
|
|
@ -679,13 +683,24 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF + mmproj
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
# Authenticated + parallel: shared macos-14 NAT egress stalls
|
||||
# multi-GB anonymous downloads.
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
HF_HUB_ENABLE_HF_TRANSFER=1 \
|
||||
hf download "$GGUF_REPO" "$GGUF_FILE"
|
||||
hf download "$GGUF_REPO" "$GGUF_FILE" &
|
||||
MODEL_PID=$!
|
||||
HF_HUB_ENABLE_HF_TRANSFER=1 \
|
||||
hf download "$GGUF_REPO" "$MMPROJ_FILE"
|
||||
hf download "$GGUF_REPO" "$MMPROJ_FILE" &
|
||||
MMPROJ_PID=$!
|
||||
wait "$MODEL_PID"
|
||||
wait "$MMPROJ_PID"
|
||||
# Fail loud on a partial download instead of in the next step.
|
||||
find hf-cache -name "$GGUF_FILE" -o -name "$MMPROJ_FILE" \
|
||||
| xargs -I{} ls -lhL {}
|
||||
|
||||
- name: Install Studio (--local, --no-torch)
|
||||
env:
|
||||
|
|
|
|||
2
.github/workflows/studio-mac-ui-smoke.yml
vendored
2
.github/workflows/studio-mac-ui-smoke.yml
vendored
|
|
@ -65,6 +65,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
2
.github/workflows/studio-ui-smoke.yml
vendored
2
.github/workflows/studio-ui-smoke.yml
vendored
|
|
@ -79,6 +79,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
|
|
@ -72,6 +72,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
|
|
@ -82,6 +82,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
@ -382,6 +384,8 @@ jobs:
|
|||
|
||||
- name: Download GGUF if cache miss
|
||||
if: steps.cache-gguf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p gguf-cache
|
||||
|
|
@ -776,6 +780,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF + mmproj
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
|
|
@ -81,6 +81,8 @@ jobs:
|
|||
|
||||
- name: Prime HF_HOME with the GGUF
|
||||
if: steps.cache-hf.outputs.cache-hit != 'true'
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
python -m pip install --upgrade huggingface_hub hf_transfer
|
||||
mkdir -p hf-cache
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue