54a86c3514Root cause of the Mac json-images 30 min timeout (run 25950714888 / PR #5430): huggingface_hub>=1.15 deprecated `hf_transfer` and routes every transfer through `hf-xet`. The CI step's unpinned `pip install --upgrade huggingface_hub hf_transfer` jumped to 1.15.0 + hf-xet 1.5.0, the 940 MB mmproj finished in ~21s, then the 3 GB gemma-4 GGUF made it to ~46% and went completely silent for the remaining 29 minutes -- no progress bytes, no error, no exit -- until the job timeout fired. This wraps every CI `hf download` in a new `.github/scripts/hf-download-with-retry.sh`: * Drops the no-op `HF_HUB_ENABLE_HF_TRANSFER=1` prefix and the `hf_transfer` install (both are deprecated on 1.15+ and only emit a FutureWarning now). * Exports the hf-xet high-performance knobs Daniel asked for: HF_XET_HIGH_PERFORMANCE=1 HF_XET_CHUNK_CACHE_SIZE_BYTES=0 HF_XET_NUM_CONCURRENT_RANGE_GETS=64 HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=0 HF_XET_CLIENT_READ_TIMEOUT=500 * Watchdogs each attempt: if `hf download` has not exited after HF_DOWNLOAD_STALL_SECONDS (default 180s = 3 min), SIGTERM, sleep 2, SIGKILL, then loop. Retries are unbounded; the enclosing job's `timeout-minutes` is the real cap. * Optional 3rd positional `LOCAL_DIR` -- omitted lets `hf` use the default HF_HUB_CACHE, which is what the HF_HOME-priming jobs need. 19 call sites migrated across mlx-ci.yml + 9 studio-*-smoke.yml workflows. The inline `python -c "from huggingface_hub import hf_hub_download; ..."` block in mlx-ci.yml is also routed through the wrapper so every hf transfer in CI gets the same treatment. Also reverts the json-images timeout 45 -> 30 from #5475: the bump was masking this hang, not fixing it.
342 lines
14 KiB
YAML
342 lines
14 KiB
YAML
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
|
|
|
# Windows counterpart to studio-ui-smoke.yml / studio-mac-ui-smoke.yml.
|
|
# Same Playwright + Chromium end-to-end chat UI flow + extra UI flow,
|
|
# but on the FREE windows-latest runner so we catch Windows-specific
|
|
# regressions in the install path (install.ps1), the Studio CLI's
|
|
# Windows process-management branches, and the llama.cpp prebuilt's
|
|
# Windows HTTP layer.
|
|
|
|
name: Windows Studio UI CI
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- 'studio/**'
|
|
- 'unsloth/**'
|
|
- 'unsloth_cli/**'
|
|
- 'install.ps1'
|
|
- 'pyproject.toml'
|
|
- 'tests/studio/**'
|
|
- '.github/workflows/studio-windows-ui-smoke.yml'
|
|
push:
|
|
branches: [main, pip]
|
|
workflow_dispatch:
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
ui-smoke:
|
|
name: Chat UI Tests
|
|
runs-on: windows-latest
|
|
timeout-minutes: 45
|
|
# Default every step's shell to Git Bash. windows-latest's default
|
|
# shell is pwsh; without this each curl / heredoc / `kill $PID`
|
|
# step would need its own `shell: bash`. Steps that genuinely
|
|
# need PowerShell (install.ps1 invocation) override per-step.
|
|
defaults:
|
|
run:
|
|
shell: bash
|
|
env:
|
|
GGUF_REPO: unsloth/gemma-3-270m-it-GGUF
|
|
GGUF_VARIANT: UD-Q4_K_XL
|
|
GGUF_FILE: gemma-3-270m-it-UD-Q4_K_XL.gguf
|
|
STUDIO_PORT: '18896'
|
|
HF_HOME: ${{ github.workspace }}/hf-cache
|
|
# Force UTF-8 for stdio so Python tools (hf download, Studio
|
|
# CLI, etc.) can print Unicode characters like the success
|
|
# checkmark "✓". Windows defaults to cp1252 / charmap and
|
|
# any tool that prints "OK ✓" hits a UnicodeEncodeError.
|
|
PYTHONIOENCODING: utf-8
|
|
PYTHONUTF8: '1'
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
|
with:
|
|
node-version: '22'
|
|
# No `cache: 'npm'`. setup-node's npm cache restore silently
|
|
# aborts the entire job on Windows runners when the npm cache
|
|
# path (`C:\npm\cache` per `npm config get cache`) doesn't yet
|
|
# exist on a fresh runner -- the step exits without an error
|
|
# message and every following step gets skipped. See
|
|
# npm/cli#7308. The frontend `npm ci` is fast enough without
|
|
# the cache that the reliability gain is worth the ~30s.
|
|
|
|
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
|
with:
|
|
python-version: '3.12'
|
|
# No `cache: 'pip'`. install.ps1 / setup.ps1 use uv and
|
|
# never populate ~/.cache/pip; setup-python's post-step
|
|
# then fatal-errors with "Cache folder path is retrieved
|
|
# for pip but doesn't exist on disk".
|
|
|
|
- name: Restore HF_HOME for ${{ env.GGUF_REPO }}
|
|
id: cache-hf
|
|
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
|
continue-on-error: true
|
|
with:
|
|
path: hf-cache
|
|
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
|
|
|
|
- name: Prime HF_HOME with the GGUF
|
|
id: prime-hf
|
|
if: steps.cache-hf.outputs.cache-hit != 'true' || steps.cache-hf.outcome != 'success'
|
|
env:
|
|
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
|
run: |
|
|
python -m pip install --upgrade huggingface_hub
|
|
mkdir -p hf-cache
|
|
bash .github/scripts/hf-download-with-retry.sh "$GGUF_REPO" "$GGUF_FILE"
|
|
|
|
- name: Save HF_HOME for ${{ env.GGUF_REPO }}
|
|
if: always() && steps.prime-hf.outcome == 'success'
|
|
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
|
with:
|
|
path: hf-cache
|
|
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
|
|
|
|
- name: Pre-install Windows tweaks (npm 11 + Defender exclusions)
|
|
shell: pwsh
|
|
# See studio-windows-update-smoke.yml for the full rationale.
|
|
# tl;dr: setup.ps1 needs npm >=11 to skip a 35 s winget Node
|
|
# reinstall, and Defender's real-time scan dominates the
|
|
# frontend / uv-pip-extract steps.
|
|
run: |
|
|
$ProgressPreference = 'SilentlyContinue'
|
|
Write-Host "npm version before upgrade: $(npm -v)"
|
|
npm install -g 'npm@^11' 2>&1 | Out-Host
|
|
Write-Host "npm version after upgrade: $(npm -v)"
|
|
# NOTE: do NOT pre-create these directories. See
|
|
# studio-windows-update-smoke.yml for the full rationale --
|
|
# creating an empty studio/frontend/dist trips setup.ps1's
|
|
# mtime-based staleness check into "frontend up to date, skip
|
|
# rebuild" and Studio boots with an empty dist directory.
|
|
# Add-MpPreference accepts paths that do not yet exist.
|
|
foreach ($p in @(
|
|
"$env:USERPROFILE\.unsloth",
|
|
"$env:USERPROFILE\AppData\Local\uv",
|
|
"$env:GITHUB_WORKSPACE\studio\frontend\node_modules",
|
|
"$env:GITHUB_WORKSPACE\studio\frontend\dist"
|
|
)) {
|
|
try {
|
|
Add-MpPreference -ExclusionPath $p -ErrorAction Stop
|
|
Write-Host "Defender exclusion added: $p"
|
|
} catch {
|
|
Write-Host "Defender exclusion skipped ($($_.Exception.Message)): $p"
|
|
}
|
|
}
|
|
|
|
- name: Install Studio (--local, --no-torch)
|
|
# install.ps1 is the supported Windows installer. install.sh
|
|
# has no Windows branch (apt-get / brew calls). The PS1
|
|
# script's `Install-UnslothStudio @args` line at the bottom
|
|
# forwards `--local --no-torch` correctly.
|
|
shell: pwsh
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: |
|
|
New-Item -ItemType Directory -Force -Path logs | Out-Null
|
|
# *>&1 redirects ALL PowerShell streams (stdout, stderr,
|
|
# warning, verbose, debug, information) into the success
|
|
# stream so Tee-Object captures everything. install.ps1
|
|
# and setup.ps1 emit step/substep markers via Write-Host
|
|
# which lands on the Information stream (PS 5+); without
|
|
# the wildcard redirect, those markers (including
|
|
# "prebuilt installed and validated") never reach
|
|
# logs/install.log and the post-step grep asserter fails.
|
|
$ProgressPreference = 'SilentlyContinue'
|
|
& ./install.ps1 --local --no-torch *>&1 | Tee-Object -FilePath logs/install.log
|
|
|
|
- name: Assert install.ps1 used the Windows llama.cpp prebuilt
|
|
run: |
|
|
# install.ps1's setup.ps1 child writes "prebuilt installed
|
|
# and validated" to its own console host -- that output
|
|
# does NOT come back through this parent step's stdout
|
|
# pipeline (no matter how aggressively we redirect: *>&1,
|
|
# tee, etc.). Verify the install via the filesystem
|
|
# instead. setup.ps1 writes UNSLOTH_PREBUILT_INFO.json
|
|
# next to the install dir on success, and lays the
|
|
# binaries under build/bin/Release/ on Windows.
|
|
STUDIO_HOME=~/.unsloth/studio
|
|
LLAMA_DIR=~/.unsloth/llama.cpp
|
|
INFO="$LLAMA_DIR/UNSLOTH_PREBUILT_INFO.json"
|
|
BIN="$LLAMA_DIR/build/bin/Release/llama-server.exe"
|
|
# Source-build fallback grep stays as a fast bail-out.
|
|
if grep -q "falling back to source build" logs/install.log; then
|
|
echo "::error::install.ps1 fell back to source-build llama.cpp on Windows."
|
|
grep -E "llama-prebuilt|llama.cpp" logs/install.log | tail -60
|
|
exit 1
|
|
fi
|
|
if [ ! -f "$INFO" ]; then
|
|
echo "::error::no UNSLOTH_PREBUILT_INFO.json at $INFO; setup.ps1 didn't install the prebuilt."
|
|
ls -la "$LLAMA_DIR" || true
|
|
exit 1
|
|
fi
|
|
if [ ! -f "$BIN" ]; then
|
|
echo "::error::no llama-server.exe at $BIN; prebuilt extraction incomplete."
|
|
ls -la "$LLAMA_DIR/build/bin" || true
|
|
ls -la "$LLAMA_DIR/build/bin/Release" || true
|
|
exit 1
|
|
fi
|
|
echo "install.ps1 installed the Windows prebuilt llama.cpp:"
|
|
cat "$INFO"
|
|
|
|
- name: Add Studio shim to GITHUB_PATH
|
|
# install.ps1 puts unsloth.exe at $StudioHome\bin\unsloth.exe
|
|
# and adds that dir to the User PATH via the Windows registry.
|
|
# Registry-level PATH updates don't propagate to a running
|
|
# Git Bash session, so the next step's `unsloth ...` invocation
|
|
# would hit "command not found". Re-export the shim dir to
|
|
# GITHUB_PATH so every subsequent step in this job sees it.
|
|
run: |
|
|
SHIM_DIR=~/.unsloth/studio/bin
|
|
if [ ! -f "$SHIM_DIR/unsloth.exe" ]; then
|
|
echo "::error::unsloth.exe shim not found at $SHIM_DIR"
|
|
ls -la ~/.unsloth/studio/ || true
|
|
exit 1
|
|
fi
|
|
# GITHUB_PATH wants Windows-style paths; convert via cygpath.
|
|
cygpath -w "$SHIM_DIR" >> "$GITHUB_PATH"
|
|
echo "Added Studio shim dir to PATH: $(cygpath -w "$SHIM_DIR")"
|
|
|
|
- name: Patch Studio venv with full typer / pydantic dep trees
|
|
# Belt-and-suspenders: install.ps1's --no-deps install of
|
|
# no-torch-runtime.txt drops typer's and pydantic's runtime
|
|
# deps unless explicitly pinned. Re-install the ones whose
|
|
# deps don't pull torch.
|
|
run: |
|
|
STUDIO_PY=~/.unsloth/studio/unsloth_studio/Scripts/python.exe
|
|
if [ ! -f "$STUDIO_PY" ]; then
|
|
echo "::error::Studio venv python not at $STUDIO_PY"
|
|
ls -la ~/.unsloth/studio/ || true
|
|
exit 1
|
|
fi
|
|
"$STUDIO_PY" -m pip install --upgrade typer pydantic huggingface_hub
|
|
|
|
- name: Install Playwright + Chromium
|
|
# No --with-deps on Windows: that flag installs Linux apt
|
|
# packages. windows-latest ships the system frameworks
|
|
# Chromium needs (Edge / WebView2) already.
|
|
run: |
|
|
python -m pip install 'playwright>=1.45'
|
|
python -m playwright install chromium
|
|
|
|
- name: Reset auth + boot Studio
|
|
run: |
|
|
unsloth studio reset-password
|
|
mkdir -p logs
|
|
UNSLOTH_API_ONLY=1 unsloth studio -H 127.0.0.1 -p "$STUDIO_PORT" \
|
|
> logs/studio.log 2>&1 &
|
|
echo "STUDIO_PID=$!" >> "$GITHUB_ENV"
|
|
|
|
- name: Wait for /api/health
|
|
run: |
|
|
for i in $(seq 1 180); do
|
|
if curl -fs "http://127.0.0.1:${STUDIO_PORT}/api/health" > /tmp/health.json; then
|
|
jq -e '.status == "healthy"' /tmp/health.json && break
|
|
fi
|
|
sleep 1
|
|
done
|
|
jq -e '.status == "healthy"' /tmp/health.json
|
|
|
|
- name: Pass bootstrap password to the Playwright step
|
|
run: |
|
|
OLD=$(cat ~/.unsloth/studio/auth/.bootstrap_password)
|
|
NEW="CIUi-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')"
|
|
NEW2="CIUi-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')"
|
|
echo "::add-mask::$OLD"
|
|
echo "::add-mask::$NEW"
|
|
echo "::add-mask::$NEW2"
|
|
echo "STUDIO_OLD_PW=$OLD" >> "$GITHUB_ENV"
|
|
echo "STUDIO_NEW_PW=$NEW" >> "$GITHUB_ENV"
|
|
echo "STUDIO_NEW2_PW=$NEW2" >> "$GITHUB_ENV"
|
|
|
|
- name: Drive the chat UI with Playwright
|
|
env:
|
|
BASE_URL: http://127.0.0.1:18896
|
|
PW_ART_DIR: logs/playwright
|
|
STUDIO_UI_STRICT: '1'
|
|
# windows-latest free runner is 4 vCPU / 16 GB; gemma-3-
|
|
# 270m turn latency under llama-server's CPU backend can
|
|
# crowd the 180s default (slower than ubuntu-latest on
|
|
# the same model). Keep the same generous budget the Mac
|
|
# job uses.
|
|
STUDIO_UI_TURN_TIMEOUT_MS: '540000'
|
|
run: |
|
|
mkdir -p logs/playwright
|
|
python tests/studio/playwright_chat_ui.py
|
|
|
|
- name: Stop Studio (chat-ui ends with Shutdown click; this is belt-and-suspenders)
|
|
if: always()
|
|
run: |
|
|
kill "${STUDIO_PID}" 2>/dev/null || true
|
|
sleep 2
|
|
|
|
- name: Reset auth + boot Studio for extra UI tests (port 18897)
|
|
run: |
|
|
unsloth studio reset-password
|
|
mkdir -p logs
|
|
UNSLOTH_API_ONLY=1 unsloth studio -H 127.0.0.1 -p 18897 \
|
|
> logs/studio_extra.log 2>&1 &
|
|
echo "STUDIO_EXTRA_PID=$!" >> "$GITHUB_ENV"
|
|
|
|
- name: Wait for /api/health on 18897
|
|
run: |
|
|
for i in $(seq 1 180); do
|
|
if curl -fs "http://127.0.0.1:18897/api/health" > /tmp/health2.json; then
|
|
jq -e '.status == "healthy"' /tmp/health2.json && break
|
|
fi
|
|
sleep 1
|
|
done
|
|
jq -e '.status == "healthy"' /tmp/health2.json
|
|
|
|
- name: Pass bootstrap pw for extra UI test
|
|
run: |
|
|
OLD=$(cat ~/.unsloth/studio/auth/.bootstrap_password)
|
|
NEW="CIUiExtra-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')"
|
|
echo "::add-mask::$OLD"
|
|
echo "::add-mask::$NEW"
|
|
echo "STUDIO_EXTRA_OLD_PW=$OLD" >> "$GITHUB_ENV"
|
|
echo "STUDIO_EXTRA_NEW_PW=$NEW" >> "$GITHUB_ENV"
|
|
|
|
- name: Drive Compare/Recipes/Export/Studio/Settings with Playwright
|
|
env:
|
|
BASE_URL: http://127.0.0.1:18897
|
|
STUDIO_OLD_PW: ${{ env.STUDIO_EXTRA_OLD_PW }}
|
|
STUDIO_NEW_PW: ${{ env.STUDIO_EXTRA_NEW_PW }}
|
|
PW_ART_DIR: logs/playwright_extra
|
|
STUDIO_UI_STRICT: '1'
|
|
STUDIO_UI_TURN_TIMEOUT_MS: '540000'
|
|
GGUF_REPO: ${{ env.GGUF_REPO }}
|
|
GGUF_VARIANT: ${{ env.GGUF_VARIANT }}
|
|
run: |
|
|
mkdir -p logs/playwright_extra
|
|
python tests/studio/playwright_extra_ui.py
|
|
|
|
- name: Stop second Studio
|
|
if: always()
|
|
run: |
|
|
kill "${STUDIO_EXTRA_PID}" 2>/dev/null || true
|
|
sleep 2
|
|
|
|
- name: Upload Playwright artifacts
|
|
if: always()
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
with:
|
|
name: windows-studio-ui-smoke-artifacts
|
|
path: |
|
|
logs/studio.log
|
|
logs/studio_extra.log
|
|
logs/install.log
|
|
logs/playwright
|
|
logs/playwright_extra
|
|
retention-days: 7
|