CI(studio): new Studio API & Auth Tests workflow + integration test

HTTP-level integration smoke for the Studio FastAPI surface, no
Playwright. ~30 s per run on warm cache. Boots a fresh Studio, then
asserts:

  1. CORS hardening -- no wildcard-origin + credentials=true; cross-
     origin GET / does not leak the bootstrap password to evil.example.
  2. /api/system + /api/system/hardware + /api/system/gpu-visibility
     all require auth (closes the info-disclosure leak).
  3. Auth state machine -- rotation invariants (old=401, new=200),
     refresh-without-body returns 4xx, login burst documents the
     current "no rate-limit" behaviour so future hardening updates the
     test in the same PR.
  4. JWT-expiry forgery -- mint a JWT with exp=now-1 using the install's
     own secret + assert it returns 401.
  5. API key lifecycle E2E -- create -> list -> use against
     /v1/chat/completions -> delete -> verify 401.
  6. Auth file-mode hardening (Linux only): auth/ is 0700, auth.db +
     -wal + -shm + .bootstrap_password are 0600.
  7. Inference lifecycle gaps -- /v1/models lists the loaded model,
     /v1/embeddings + /v1/responses return 200 OR structured 4xx,
     bogus gguf_variant rejected, force-reload swaps the llama-server
     PID.
  8. Endpoint-by-endpoint auth audit -- pins the EXPECTED auth posture
     for known routes; an unauthenticated /api/shutdown is rejected
     BEFORE the shutdown trigger fires.

Reuses the same GGUF cache key as studio-ui-smoke.yml so the model
download is one cache-hit across CI.

Random per-run rotated passwords + ::add-mask:: pattern matches
studio-ui-smoke.yml + studio-inference-smoke.yml.
This commit is contained in:
Daniel Han 2026-05-07 04:06:06 +00:00
commit 29f2829582
2 changed files with 706 additions and 0 deletions

153
.github/workflows/studio-api-smoke.yml vendored Normal file
View file

@ -0,0 +1,153 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
# Studio API & Auth Tests -- HTTP-level integration tests for the
# FastAPI surface. No Playwright, no model UI; tests/studio/test_studio_api_smoke.py
# runs ~30 s and asserts:
# - CORS hardening (no wildcard + credentials, no bootstrap leak)
# - /api/system + /api/system/hardware require auth
# - Auth state machine + JWT expiry
# - API key lifecycle E2E (create / list / use / delete / reject)
# - Auth file-mode hardening (Linux only)
# - Inference lifecycle (force reload, bogus variant, /v1/models, /v1/embeddings, /v1/responses)
# - Endpoint-by-endpoint auth audit
#
# Reuses the GGUF cache key from studio-ui-smoke.yml so the model
# download is one cache-hit on the second job.
name: Studio API CI
on:
pull_request:
paths:
- 'studio/**'
- 'unsloth/**'
- 'unsloth_cli/**'
- 'install.sh'
- 'pyproject.toml'
- 'tests/studio/**'
- '.github/workflows/studio-api-smoke.yml'
push:
branches: [main, pip]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
api-smoke:
name: Studio API & Auth Tests
runs-on: ubuntu-latest
timeout-minutes: 12
env:
GGUF_REPO: unsloth/gemma-3-270m-it-GGUF
GGUF_VARIANT: UD-Q4_K_XL
GGUF_FILE: gemma-3-270m-it-UD-Q4_K_XL.gguf
STUDIO_PORT: '18893'
HF_HOME: ${{ github.workspace }}/hf-cache
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- name: Linux deps
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
libcurl4-openssl-dev libssl-dev jq
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: studio/frontend/package-lock.json
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
cache: 'pip'
- name: Cache HF_HOME for ${{ env.GGUF_REPO }}
id: cache-hf
uses: actions/cache@v4
with:
path: hf-cache
# Same key as studio-ui-smoke.yml so the two jobs share a
# single GGUF download across CI.
key: ${{ runner.os }}-hf-${{ env.GGUF_REPO }}-${{ env.GGUF_VARIANT }}-v1
- name: Prime HF_HOME with the GGUF
if: steps.cache-hf.outputs.cache-hit != 'true'
run: |
python -m pip install --upgrade huggingface_hub hf_transfer
mkdir -p hf-cache
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download "$GGUF_REPO" "$GGUF_FILE"
- name: Install Studio (--local, --no-torch)
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
mkdir -p logs
set -o pipefail
bash install.sh --local --no-torch 2>&1 | tee logs/install.log
- name: Install pyjwt for the JWT-expiry forge test
run: pip install 'pyjwt>=2.6'
- name: Reset auth + boot Studio (API-only)
run: |
unsloth studio reset-password
mkdir -p logs
UNSLOTH_API_ONLY=1 unsloth studio -H 127.0.0.1 -p "$STUDIO_PORT" \
> logs/studio.log 2>&1 &
echo "STUDIO_PID=$!" >> "$GITHUB_ENV"
- name: Wait for /api/health
run: |
for i in $(seq 1 180); do
if curl -fs "http://127.0.0.1:${STUDIO_PORT}/api/health" > /tmp/health.json; then
jq -e '.status == "healthy"' /tmp/health.json && break
fi
sleep 1
done
jq -e '.status == "healthy"' /tmp/health.json
- name: Pass bootstrap password + rotated targets to the test
# The test does its own bootstrap-login + rotation to exercise
# the auth state machine; we just pre-mint two random rotated
# passwords for it. Mask them so the log is clean.
run: |
OLD=$(cat ~/.unsloth/studio/auth/.bootstrap_password)
NEW="ApiSmoke-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')"
NEW2="ApiSmoke-$(python -c 'import secrets; print(secrets.token_urlsafe(16))')"
echo "::add-mask::$OLD"
echo "::add-mask::$NEW"
echo "::add-mask::$NEW2"
echo "STUDIO_OLD_PW=$OLD" >> "$GITHUB_ENV"
echo "STUDIO_NEW_PW=$NEW" >> "$GITHUB_ENV"
echo "STUDIO_NEW2_PW=$NEW2" >> "$GITHUB_ENV"
- name: Run Studio API & Auth tests
env:
BASE_URL: http://127.0.0.1:18893
STUDIO_AUTH_DIR: /home/runner/.unsloth/studio/auth
run: python tests/studio/test_studio_api_smoke.py
- name: Stop Studio
if: always()
run: |
kill "${STUDIO_PID}" 2>/dev/null || true
sleep 2
- name: Upload API smoke logs
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: studio-api-smoke-log
path: |
logs/install.log
logs/studio.log
retention-days: 7