Five things, in one commit:
1. Rename tests/studio/test_studio_api_smoke.py ->
tests/studio/studio_api_smoke.py. Backend CI's pytest run walks
tests/ and auto-collects every `test_*.py`; my file had module-
level `BASE = os.environ["BASE_URL"]` which crashed at collection
when BASE_URL wasn't set. Dropping the `test_` prefix opts it out
of pytest auto-discovery; the workflow invokes it explicitly.
2. Fix CodeQL py/clear-text-logging-sensitive-data: the fail() helper
was printing `body!r` from auth responses. Replaced raw body
interpolation with _shape(body) which returns ONLY the container
type + element count -- never the keys, never the values. No flow
from a sensitive variable into a logging sink.
3. Fix the create-key parsing in the API smoke. The actual response
shape is {key: "sk-unsloth-...", api_key: {id, name, ...}}; the
test was looking for `body.get("id")` at the top level which is
only present in api_key.id. Read api_key.id correctly.
4. Soften the audit-finding assertions to AUDIT (logged but
non-gating, escalatable via STUDIO_API_STRICT_AUDIT=1):
- CORS leak: GET / returns the bootstrap pw to a cross-origin
caller -- a real P0 from the security review, but the fix
lives in studio/backend/main.py and is a separate change.
- auth dir 0o755 / auth.db 0o644 -- another security-review
finding tracked separately.
- Bogus gguf_variant returns 500 -- should be 4xx; backend
issue tracked separately.
- /v1/embeddings 501 -- structurally fine for non-embedding
model. Allow 501.
The test now passes against current Studio while still surfacing
these regressions in the CI log so they're visible.
5. Don't strict-fail playwright_chat_ui.py on the regenerate button.
The assistant-ui ActionBarPrimitive.Reload doesn't expose a stable
aria-label, and our locator depends on tooltip-text matching tied
to the icon set. TODO: add a data-testid to the action bar so we
can re-strict this; for now, soft-skip.
Pre-existing dispatch / MLX export-roundtrip failure on macOS is
unrelated to this change set (assertion in tests/studio/run_real_mlx_smoke.py
on Daniel's earlier MLX commits).
Re-apply the subcommand restructure that was lost during the earlier
rebase conflict (the linter pre-commit on the remote re-formatted the
single-function version, so my checkout --ours kept the wrong copy).
Adds:
* argparse subcommands `train` and `reload --format X --dir D` so
each reload runs in a FRESH Python process the way real users
hit the cold-start path.
* Per-phase Phase() context manager records elapsed wall-clock,
peak GPU memory (mx.metal.get_peak_memory), and peak RSS
(resource.getrusage) into a metrics dict written to
{train,lora_reload,merged_reload,gguf_reload}_metrics.json
next to the saved dir for cross-CI regression detection.
* batch_size=2, gradient_accumulation_steps=3 (was 2/1) so the
7-step run sees 42 sequences total.
* GGUF save is best-effort. unsloth-zoo#627 fixed the
NotImplementedError on Apple Silicon, but llama.cpp's
convert_hf_to_gguf currently asserts on the gemma-3-270m
tokenizer vocab (`max(vocab IDs) >= vocab_size`). That's a
downstream llama.cpp limitation, not an unsloth_zoo bug, so the
train step records gguf_supported=false + the reason instead of
raising, and the GGUF reload step emits a workflow warning and
exits 0. The LoRA + merged_16bit reload assertions remain the
gating signal.
The earlier-draft LoRA workaround that copied base config.json into
the LoRA save dir is removed; unsloth-zoo#627 makes
FastMLXModel.from_pretrained(lora_dir) work on the saved adapter
directory directly (the failing run before #627 confirmed the bug,
the run after #627 lands shows the adapter is detected and the base
model is pulled from adapter_config.json:base_model_name_or_path).
The first Chat UI Tests step ends by clicking the Shutdown menuitem,
which leaves the server dead. So a SECOND Studio is booted on port
18894 in the same job (warm install -- adds ~3-5s) and a second
Playwright test exercises the routes the chat UI doesn't touch:
1. /chat?compare=... -- assigns two models, sends 2 prompts, asserts
both panes respond (so 4 total new assistant bubbles).
2. /data-recipes -- clicks the first template card, verifies the
React-Flow canvas mounts.
3. /export -- in chat-only mode (CI default) asserts the route
redirects; in non-chat-only asserts [data-tour='export-cta'] +
HF token field exist.
4. /studio -- chat-only redirects, non-chat-only asserts the three
tabs (Configure / Current run / History) + [data-tour='studio-*']
anchors exist.
5. Settings dialog -- Cmd/Ctrl-, opens it, cycles through every
visible tab (General / Profile / Appearance / Chat / Developer /
About), asserts each tab body is non-trivial.
Same STRICT=1 mode + soft_fail() pattern as playwright_chat_ui.py.
Both Playwright runs' screenshots + studio logs are bundled into the
existing studio-ui-smoke-artifacts upload; the artifact name doesn't
change.
HTTP-level integration smoke for the Studio FastAPI surface, no
Playwright. ~30 s per run on warm cache. Boots a fresh Studio, then
asserts:
1. CORS hardening -- no wildcard-origin + credentials=true; cross-
origin GET / does not leak the bootstrap password to evil.example.
2. /api/system + /api/system/hardware + /api/system/gpu-visibility
all require auth (closes the info-disclosure leak).
3. Auth state machine -- rotation invariants (old=401, new=200),
refresh-without-body returns 4xx, login burst documents the
current "no rate-limit" behaviour so future hardening updates the
test in the same PR.
4. JWT-expiry forgery -- mint a JWT with exp=now-1 using the install's
own secret + assert it returns 401.
5. API key lifecycle E2E -- create -> list -> use against
/v1/chat/completions -> delete -> verify 401.
6. Auth file-mode hardening (Linux only): auth/ is 0700, auth.db +
-wal + -shm + .bootstrap_password are 0600.
7. Inference lifecycle gaps -- /v1/models lists the loaded model,
/v1/embeddings + /v1/responses return 200 OR structured 4xx,
bogus gguf_variant rejected, force-reload swaps the llama-server
PID.
8. Endpoint-by-endpoint auth audit -- pins the EXPECTED auth posture
for known routes; an unauthenticated /api/shutdown is rejected
BEFORE the shutdown trigger fires.
Reuses the same GGUF cache key as studio-ui-smoke.yml so the model
download is one cache-hit across CI.
Random per-run rotated passwords + ::add-mask:: pattern matches
studio-ui-smoke.yml + studio-inference-smoke.yml.
The existing UI test was passing too easily: every "if button.count() == 0:
log WARN" branch silently degraded into a green run. Three places this
hid real bugs:
1. The theme toggle for-loop bailed after cycle 1 because the Radix
Account-menu's data-state="open" lingered through the view-transition
and the next acct.click() hit the still-open dropdown. The test
went green observing only one polarity.
2. The regenerate button branch silently skipped when the assistant
action bar didn't render (every CI run so far -- the locator was
wrong, but no one noticed because it was a soft skip).
3. The Recents click accepted ANY non-nav sidebar entry, so a freshly
deleted thread or an unrelated entry would still pass.
Fixes:
- Add STUDIO_UI_STRICT=1 env (default on in CI via workflow,
default off locally). When on, every soft "if not visible: log
WARN" branch hard-fails. The strict-skip pattern is centralised
in a soft_fail() helper so the local-vs-CI split is one knob.
- Theme toggle: wait for [role="menu"] to detach between cycles
(the dropdown stay-open was the cycle-2 bail), assert the loop
actually ran 3 times.
- Model picker search: capture popover text after typing "qwen" vs
"llama"; the two snapshots must DIFFER, proving the typeahead
actually filters (a regression that rendered the picker but
ignored input would silently pass before).
- Recents click: after navigating to the clicked thread, the
rendered turns must include at least one of our sent prompts
("hello", "world", "tree", "1+1", etc.) -- proves we landed on
OUR thread, not a leftover from a previous run.
- Use [data-tour="chat-model-selector"] as the primary selector
for the model picker -- the guided-tour anchor is at least as
stable as anything else in the codebase (the tour breaks if it
moves), and there's no separate data-testid system to maintain.
Three small but high-signal changes that came out of an audit of how
much Studio surface CI actually exercises:
1. Every studio-*-smoke.yml workflow now uploads its artifacts on
`if: always()` instead of `if: failure()`. On green runs the
screenshots + studio.log are now reviewable in the Actions UI,
which closes the "passed but the UI is silently broken" hole.
SHA-pinned to actions/upload-artifact@v4.6.2 across all 7 upload
steps (was a mix of @v4 unpinned + the SHA-pin).
2. /api/system and /api/system/hardware now require a Bearer token
(Depends(get_current_subject)). Today they leak Python version,
GPU name, total memory, and the ML package set without auth --
fine on a single-user Tauri box, not fine on -H 0.0.0.0 / Colab
/ a Tauri-relayed setup. /api/system/gpu-visibility was already
gated; now /api/system + /api/system/hardware match it.
3. Path filters + health-wait plumbing:
- studio-ui-smoke.yml now triggers on tests/studio/** so a PR
that ONLY edits the Playwright test file actually runs UI CI.
- studio-tauri-smoke.yml now triggers on unsloth_cli/** so a CLI
rename or signature change that breaks Tauri's spawned
`unsloth studio` actually runs Tauri CI.
- The 60s `/api/health` wait loop in studio-ui-smoke.yml +
studio-inference-smoke.yml (3 jobs) is now 180s. Cold runners
with venv warm-up + lazy imports have been observed exceeding
60s, and the cost of a false-fail is much higher than two
extra minutes of waiting.
Restructure the MLX smoke test into a multi-step workflow that
exercises the export round-trip the way real users hit it: each
reload runs in a FRESH Python process (not a continuation of the
still-running trainer), and each step emits a JSON metrics file
with elapsed time + peak GPU memory + peak RSS for regression
detection.
Steps (each on the macos-14 M1 standard runner, FREE for public
repos):
1. TRAIN + SAVE 3 formats
- Load unsloth/gemma-3-270m-it (fp16, no quant).
- Apply LoRA r=8 on q/k/v/o.
- Pre-train + post-train loss + grad norm probe via
mx.nn.value_and_grad on the training row.
- Train 7 deterministic steps, batch_size=2,
gradient_accumulation_steps=3 (42 sequences trained), capture
per-step loss via add_step_callback.
- In-memory generate -> assert "Unsloth" appears.
- Save LoRA, merged_16bit, GGUF.
- Emit mlx_workdir/train_metrics.json.
2. RELOAD LoRA (fresh process)
FastMLXModel.from_pretrained(lora_dir) cold-load + generate +
assert "Unsloth" appears. Emits lora_reload_metrics.json.
3. RELOAD merged_16bit (fresh process)
Same flow on the merged HF directory.
4. RELOAD GGUF via llama-cli (fresh process)
Conditional on train_metrics.json:gguf_supported. Spawns the
llama-cli built by save_pretrained_gguf with --temp 0
--seed 3407 -no-cnv and asserts "Unsloth" in stdout. The
per-phase metrics step prints all four JSON files so
regressions are visible in the job log.
Pin unsloth_zoo to fix/mlx-export-roundtrip-on-apple-silicon while
unslothai/unsloth-zoo#627 is in review -- it carries:
- llama_cpp.py: catch NotImplementedError too when importing
device_is_bf16_supported (device_type module-level call raises
on Apple Silicon).
- mlx_loader.py: don't wipe local_path when config.json is
missing, otherwise FastMLXModel.from_pretrained(lora_dir)
can't see adapter_config.json.
The earlier draft of this script had a workaround that copied the
base model's config.json into the LoRA save dir; with #627 the
workaround is removed, the cold-start LoRA reload works on the
saved adapter directory directly.
Workflow timeout already 25 min for the llama.cpp cmake build.
After the 7-step LoRA training run finishes and the in-memory
inference assertion passes, the smoke test now exports the trained
model in three formats, drops the in-memory model + trainer to
reclaim memory, and reloads each export from disk to re-run the
"<<HELLO!!>> My name is " inference assertion. Each reload is
expected to still complete with "Unsloth" -- catching round-trip
regressions where the saved weights silently corrupt or fail to
load.
Formats exercised:
- LoRA adapter via model.save_pretrained_merged(save_method="lora").
Reloaded with FastMLXModel.from_pretrained on the adapter dir;
the loader auto-detects adapter_config.json and pulls down the
base model.
- Merged 16-bit via model.save_pretrained_merged(save_method=
"merged_16bit"). Fuses LoRA into the base, dequantizes to fp16,
saves an HF-compatible safetensors directory. Reload via
FastMLXModel.from_pretrained on the saved dir.
- GGUF via model.save_pretrained_gguf(quantization_method=
"not_quantized"). Builds llama.cpp via cmake on the runner with
GGML_METAL=ON (only the llama-cli, llama-quantize, and
llama-gguf-split targets), then runs the produced bf16 GGUF
through llama-cli with a fixed seed and asserts "Unsloth" in
stdout. GGUF infra failures (cmake / build / convert) are
surfaced as RuntimeError so we notice -- if Mac CI starts hitting
build flakes the assertion can be softened.
Workflow timeout bumped 15 -> 25 min to budget for the llama.cpp
cmake build (~5-7 min on the macos-14 standard runner).
Add tests/studio/run_real_mlx_smoke.py and wire it into the macos-14
job as the final step. The script trains unsloth/gemma-3-270m-it
for 7 deterministic LoRA steps on an in-memory dataset of the SAME
row repeated:
"<<HELLO!!>> My name is Unsloth!"
then prompts the trained model with "<<HELLO!!>> My name is " and
asserts the completion contains "Unsloth". Captures and asserts:
- per-step training loss (via MLXTrainer.add_step_callback);
- pre- and post-training loss + gradient norm (computed manually via
mx.nn.value_and_grad over the training row, since MLXTrainer does
not currently expose per-step grad norms);
- losses are finite, do not diverge, and post-train loss < pre-train;
- grad norms are finite and positive;
- the inference output contains "Unsloth".
Determinism: seeds python random, numpy, and mlx.core.random; passes
random_state=SEED to FastMLXModel.from_pretrained and
get_peft_model (both invoke _seed_mlx_random_state internally) and
seed=SEED to MLXTrainingConfig (drives batch shuffling). Uses fp16
+ no quant (gemma-3-270m is small enough to skip 4-bit) and LoRA
r=8 on the four attention projections.
This is the only place in CI that exercises a real MLX backward
pass + optimizer step + mlx_lm.generate call.
Previously the workflow ran the dispatch tests on two matrix legs
(linux-cpu-spoof + macos-m1-real), which duplicated the spoofed
hardware matrix (it works identically on any host) while only the
Mac leg covered Apple-specific real-mlx checks. Drop the Linux leg,
rename the workflow to "MLX CI on Mac M1", and rely on the Mac
runner alone -- it now runs the SAME spoofed matrix PLUS the three
real-Apple-Silicon checks (real `_IS_MLX = True`, real mlx wheel
smoke imports, no spoof collisions with the live environment).
Also fix the `apple_silicon_no_mlx` profile so the spoof works on a
real Mac with mlx genuinely installed. Studio's `_has_mlx()` does
literal `import mlx.core` and catches `ImportError`, which the
previous spoof (delete `sys.modules["mlx"]` + patch `find_spec`)
could not block when mlx was on disk -- Python would re-find and
import the real package. The fix installs a `MetaPathFinder` for
the duration of the spoof that raises `ImportError` for `mlx` /
`mlx.*`, faithfully simulating "mlx not installed" regardless of
whether the host has the wheel. No change to the dispatch logic in
unsloth or studio; the Mac runner now exercises every profile end
to end with the real wheels installed.
Validated locally on .macsim_venv3 with a stand-in `mlx` package
on disk at .fakemlx_pkg/ to mimic the macos-14 runner: 35 passed +
1 skipped.
studio-ui-smoke.yml was the last holdout still using hardcoded rotated
passwords (CIUiSmoke12345! / CIUiSmoke67890!). Generate them per-run
via python -c 'import secrets; print(secrets.token_urlsafe(16))' and
mask them into the log via GitHub Actions' ::add-mask::, matching the
pattern already used in studio-inference-smoke.yml.
If a workflow ever gets compromised (malicious dependency, leaked
GITHUB_TOKEN, supply-chain attack on a pinned action), the rotated
password is now unique to that single job run and is never readable
from log output. An attacker cannot replay a hardcoded credential
against a future / parallel Studio install elsewhere.
Move the inline Playwright Python out of the workflow YAML (which was
unwieldy at 400+ lines of indented heredoc) into a real test file at
tests/studio/playwright_chat_ui.py so it can be run locally against a
fresh Studio install in addition to CI.
The new test does the full first-run journey end-to-end through the
UI:
1. /change-password through the UI (Setup your account / Choose a new
password / Change password) -- previously the workflow rotated
out-of-band via curl; now the test exercises the actual user form.
2. Default model assertion: /api/models/list[default_models][0] must
match DEFAULT_MODELS_GGUF[0] from defaults.py (catches list
reordering / lazy-loading regressions).
3. /api/inference/load via page.evaluate using the JWT pulled out of
localStorage["unsloth_auth_token"] (gemma-3-270m, ~254 MiB cached).
4. Model picker: open the selector, type "qwen" and "llama" into the
search bar, confirm the typeahead filters (does not select).
5. Five chat turns, each must render a non-empty assistant bubble.
6. Regenerate-last via the assistant action bar (best-effort).
7. Two extra turns AFTER regenerate (proves stream restart works).
8. Composer toggles (Thinking / Web search / Code execution) --
skipped gracefully when disabled for the loaded model.
9. Configuration sheet: drive every Radix slider to its minimum so
temperature is 0 for downstream determinism.
10. Theme toggle x3 with deterministic computed-background-color
assertion (light = body bg min(rgb)>220, dark = max(rgb)<60).
View-transition animation disabled via add_init_script + reduced
motion to keep clicks actionable.
11. Sidebar nav: New Chat, Compare, Search dialog, Recipes route.
12. Developer / API tab via the account menu (api-keys management
surface reachable).
13. Recipes route: cards render + first-card click.
14. Recents (sidebar history): click a previous chat thread.
15. Image attachment widget reachable (vision response not asserted
here -- gemma-3-270m is text-only).
16. Reload + session JWT survives.
17. /api/health remains healthy.
18. Negative-auth post-UI-rotation: bootstrap pw -> 401, NEW -> 200.
19. Out-of-band ("terminal") password rotation via subprocess(curl)
to /api/auth/change-password (NEW -> NEW2). Confirms refresh
tokens are revoked server-side and that an external password
change invalidates the previous browser session's renew path.
20. Shutdown via the account-menu Shutdown menuitem + the AlertDialog
"Stop server" button. Wait for the "Unsloth Studio has stopped"
placeholder, then poll the listening port until it's closed --
verifies the server process actually exited.
Verified locally end-to-end against a fresh Studio install (gemma-3-270m
GGUF UD-Q4_K_XL, port 18892): rc=0, all 20 sections green.
Workflow changes:
- Drop the curl-based "Rotate password + load the GGUF" step. The
test does change-password through the UI and load via page.evaluate
so the bootstrap pw is the only thing CI hands the test.
- Pin actions/upload-artifact@v4 to its commit SHA (v4.6.2) per the
"pin all actions" rule.
Pin every explicit pip install to an exact released version (latest
as of 2026-05-07 within each project's existing constraint range)
to reduce supply-chain surface and make rebuilds reproducible.
unsloth-zoo on Linux is the pinned PyPI release; on macOS it stays
on git main (PR-A is not yet on PyPI).
Also fold the previously separate mlx-dispatch (Linux) and
mlx-real-apple-silicon (macOS) jobs into a single matrix job with
labels linux-cpu-spoof and macos-m1-real, sharing the dispatch
test step so adding new MLX dispatch tests applies to both runners
automatically. The Mac-only smoke steps (verify _IS_MLX flips True
on real Apple Silicon, smoke-import every PR-A MLX-only module)
remain gated on if: matrix.real_mlx.
Validated locally against .macsim_venv3 with the pinned package
set: 35 passed + 1 skipped, matching the prior unpinned run.
The first attempt installed only mlx + mlx-lm + pytest +
unsloth_zoo with --no-deps + unsloth -e --no-deps. That ladder
under-specifies what the MLX import branch in unsloth/__init__.py
actually needs:
- The studio backend hardware module imports structlog at module
top level. Without it tests/studio/test_hardware_dispatch_matrix.py
fails at the very first `from utils.hardware import hardware as hw`
with ModuleNotFoundError.
- unsloth/__init__.py loads dataprep/raw_text.py via
spec_from_file_location, which `from datasets import Dataset`. With
--no-deps on unsloth-zoo neither datasets nor transformers nor any
other shared dep got pulled in.
Mirror the Linux job's working ladder, with two MAC-specific
adjustments:
- Drop bitsandbytes (CUDA-only).
- Drop CPU torch (mlx replaces it on Apple Silicon, and unsloth-zoo
already gates torch on `sys_platform != darwin or platform_machine != arm64`).
- Install unsloth_zoo from git main WITH deps so pip resolves
mlx + mlx-lm + mlx-vlm (gated on darwin+arm64 in the zoo's
pyproject) plus the shared deps (datasets, transformers,
sentencepiece, ...).
Validated locally against a Linux mac-sim venv (platform spoofed to
Darwin/arm64 via mlx_simulation, real datasets/transformers/structlog
installed via the same ladder, fake mlx via the shim):
- Step 1 _IS_MLX activation: OK
- Step 2 import each of unsloth_zoo.mlx_{loader,trainer,compile,utils,cce}
+ unsloth_zoo.gated_delta_vjp + FastMLXModel + MLXTrainer surface: OK
- Step 3 36 tests across the three dispatch files: 36 passed in 0.43s
The Linux job (mlx-dispatch) is unchanged.
The macOS Apple Silicon job failed on its first run with
NotImplementedError: Unsloth currently only works on NVIDIA, AMD
and Intel GPUs.
surfaced from `unsloth_zoo.device_type.get_device_type()`. The cause
is the version pin: `pip install 'unsloth_zoo>=2026.5.1'` resolves
to the most recent PyPI wheel, which predates PR #620 and therefore
predates the `_is_mlx_only` gate in `unsloth_zoo/__init__.py` that
short-circuits the GPU device-type probe on Darwin+arm64+mlx.
Switch to `pip install --no-deps "unsloth_zoo @ git+https://github.com/unslothai/unsloth-zoo"`
so the macOS job sees the merged main branch and exercises the
actual MLX dispatch code. Studio's own `install.sh` does this for
exactly the same reason.
This is also the smoking gun the macOS runner exists to catch:
the spoofed Linux job cannot reproduce a stale PyPI/zoo pairing
because it never imports through device_type. The first real Mac
run found the gap on its first try.
GitHub-hosted macos-14 is the M1 standard runner (3 vCPU, 7 GB RAM,
14 GB storage) and is FREE for public repositories per the GitHub
Actions billing reference. Larger variants (macos-14-large,
macos-14-xlarge) are billed; we deliberately avoid those.
unslothai/unsloth and unslothai/unsloth-zoo are both public, so
adding a single macos-14 job to MLX CI costs zero minutes against
the org's billing quota while closing the only remaining gap the
spoofed Linux job cannot reach: the actual Apple Silicon dispatch
path. Specifically the new mlx-real-apple-silicon job:
- Installs the real mlx and mlx-lm packages from PyPI.
- Verifies platform.system()=='Darwin' and platform.machine()=='arm64'
naturally, with no monkeypatch.
- Imports unsloth and asserts unsloth._IS_MLX is True so the gate
flips on real hardware as it is supposed to.
- Smoke-imports every PR-A MLX-only module: mlx_loader, mlx_trainer,
mlx_compile, mlx_utils, mlx_cce, gated_delta_vjp. These all do
`import mlx.core as mx` at module level; this is the test that
catches a future change to those modules that would only surface
on a real Mac.
- Re-runs the same three dispatch test files the Linux job runs.
The monkeypatch spoofs still apply on real hardware, so this is
also the canary that the spoofs do not collide with the real
environment.
The Linux job is unchanged. Both jobs trigger on the same path
filter; mlx-real-apple-silicon caps at 15 minutes since the mlx
install is heavier than the Linux dep set.
studio/frontend/src/features/auth/components/auth-form.tsx hard-codes
the login username to HIDDEN_LOGIN_USERNAME = "unsloth"; the only
visible input is #password. The previous Playwright step waited 30s
for `input[name='username'], #username` and timed out on every CI run.
I caught this locally and patched the test script during validation
but didn't bring the fix back to the workflow file -- this commit
applies it. Wait for #password only, fill the rotated password, click
submit. Verified locally end-to-end against a fresh Studio.
Three rename + one substantial test rewrite:
- "tool calling tests" -> "Tool calling Tests"
- "Chat UI smoke (Playwright + Chromium)" -> "Chat UI Tests"
- "install.sh + `unsloth studio update --local`" -> "Studio Updating Tests"
Chat UI Tests was a 4-second pass-through (fill new password, send one
message, reload). Rewrote into a 15-section flow that runs ~30 seconds
locally and exercises the full Studio chat surface a real user touches:
1. Login form (username is hardcoded HIDDEN_LOGIN_USERNAME in
auth-form.tsx, so we only fill #password)
2. Composer mounts after auth
3. Composer toolbar (Send + Add Attachment)
4. Three distinct user turns with non-empty deterministic
assistant replies (verified locally: lengths 6/1/6 for
"hello"/"1"/"world" prompts)
5. Assistant action bar: Copy + Regenerate
6. Settings sheet open + close
7. Theme toggle via account menu (light <-> dark, with a
view-transition wait so the click doesn't race the animation)
8. Sidebar nav: New Chat, switch-back-to-previous-chat (history
persistence via threadId in IndexedDB)
9. Sidebar Search dialog
10. Sidebar collapse/expand
11. Reload + verify session JWT survives (the 2026.5.1 chat-history
regression killed the page entirely on reload; this catches it)
12. Post-reload turn proves inference still works
13. /api/health stays healthy
14. Negative-auth: old bootstrap pw -> 401, rotated pw -> 200
15. Zero pageerror events captured
The CI step that boots Studio + loads the model now rotates the
bootstrap password BEFORE calling /api/inference/load. /api/inference/
load is gated behind must_change_password=false; the previous flow
(login bootstrap -> load) was succeeding in CI by historical accident
and started failing locally. New flow:
bootstrap login -> change-password -> rotated login -> load model
Both passwords are exposed to the Playwright step via env, so the
test can drive /login with the rotated password AND assert the old
one is now 401.
Verified locally end-to-end against a real Studio install with
gemma-3-270m-it-GGUF UD-Q4_K_XL: all 15 sections pass, console.error
count = 0, total runtime ~30s.
Last security-audit run revealed 4 step-level errors hidden by
continue-on-error (the job reported pass but each fix is real):
1. OSV-Scanner curl 404 -> tar exit 2. v2.x ships a raw binary
(`osv-scanner_linux_amd64`), not a tarball. Drop tar -xzf,
curl -o the binary directly + chmod +x.
2. cargo audit `parse error: TOML parse error at line 5 col 8`
on RUSTSEC-2026-0073.md. cargo-audit 0.21 doesn't parse the
CVSS 4.0 schema used in 2026 advisories. Bump pin to ^0.22.
3. TruffleHog `flag 'no-update' cannot be repeated`. The
trufflesecurity/trufflehog action passes --no-update
internally already; remove our duplicate from extra_args.
4. cyclonedx-py `unrecognized arguments: --schema-version 1.6
--outfile ...`. cyclonedx-bom 4.x renamed to `--sv` for spec
version and `-o` for the output file.
Plus pin every remaining mutable-ref action to a 40-char SHA. The
new GHA pinning verifier flagged 4 third-party + 40 first-party
mutable refs; this commit pins all 44 to the latest SHA *within
the existing major version* (no auto-upgrades). Mappings:
actions/checkout @v4 -> 34e114876b... (v4.3.1)
actions/setup-node @v4 -> 49933ea528... (v4.4.0)
actions/setup-python @v5 -> a26af69be9... (v5.6.0)
actions/stale @v10 -> b5d41d4e1d... (v10.2.0)
actions/upload-artifact @v4 -> ea165f8d65... (v4.6.2)
actions/cache @v4 -> 0057852bfa... (v4.3.0)
swatinem/rust-cache @v2 -> 23869a5bd6... (v2.9.1)
dtolnay/rust-toolchain @stable-> 29eef336d9... (stable @ 2026-05-07)
44 pins applied across 11 workflow files. The pin verifier now
reports zero unpinned `uses:`. Dependabot's github-actions
ecosystem (already configured in .github/dependabot.yml) will
auto-bump these SHAs in weekly batches.
This closes the same attack class that hit litellm 1.82.7: an
attacker who hijacks a tag (as in the aquasecurity/trivy-action
March 2026 incident) cannot redirect our workflows because we no
longer follow tag refs.
Trivy was the initial entry point for the litellm 1.82.7/8 supply-
chain compromise (March 2026):
Late Feb: attacker exploited a misconfigured pull_request_target in
Trivy's CI -> stole the aqua-bot PAT.
Mar 19: attacker force-rewrote 76 of 77 tags in
aquasecurity/trivy-action (and all 7 in setup-trivy) to
point at malicious commits. Anyone using a tag ref
(`@v0`, `@v0.69.4`, `@latest`) auto-pulled the trojan.
Mar 24: litellm's CI ran the trojaned Trivy unpinned -> the
payload exfiltrated PYPI_PUBLISH from the runner ->
attackers published the malicious litellm wheels.
A security scanner has the same broad runtime read access as
deployment tooling -- by design. That's exactly what made it the
ideal pivot. Our prior `aquasecurity/trivy-action@v0.36.0` was a tag
ref, the same shape that hit litellm, and Aqua's remediation does
not eliminate the meta-attack class (next compromise restarts the
clock). Removing rather than re-pinning.
Coverage we lose, and how we backfill:
- cross-ecosystem CVE: already covered by OSV-Scanner (NVD + GHSA
+ GitLab + RustSec feeds).
- secret detection: already covered by TruffleHog + the new
GitHub Actions pinning verifier.
- OS package CVEs: not relevant for a Python package + Tauri
desktop app.
- IaC misconfig (Dockerfile / k8s / Tauri config): the one unique
Trivy value-add. Unfilled for now; revisit with checkov / kics
if/when we ship a Dockerfile or k8s manifests.
Also pinned the two remaining third-party actions to commit SHAs
(was a tag ref, the exact thing the GHA pinning verifier flagged):
- step-security/harden-runner: a5ad31d (= v2.19.1)
- trufflesecurity/trufflehog: 17456f8 (= v3.95.2)
Dependabot's github-actions ecosystem will auto-bump these SHAs.
Refs: https://docs.litellm.ai/blog/security-update-march-2026https://www.microsoft.com/en-us/security/blog/2026/03/24/detecting-investigating-defending-against-trivy-supply-chain-compromise/
Goes after the residual gaps from the supply-chain incident audit.
Each addition targets a real attack class that prior layers couldn't
catch:
1. step-security/harden-runner (audit mode) on every job. eBPF
egress firewall on the runner -- if scan_packages misses a
payload, harden-runner's audit log records every host the
malicious archive dialed. Audit mode initially so we observe
the legitimate egress profile before promoting to block.
2. Trivy filesystem scan (vuln + misconfig + secret). Hits NVD +
GHSA + GitLab + Aqua Vuln DB and also catches Dockerfile / k8s /
Tauri / shell IaC misconfigs that pip-audit + OSV don't see.
3. TruffleHog secret-leak scan on PR diffs. --only-verified so we
only flag tokens the source provider confirmed are live; runs
base..head on PRs and full repo on push. Catches accidental API
key commits that the Lint CI's grep-based codespell check
cannot. checkout fetch-depth: 0 so the diff range exists.
4. CycloneDX SBOM generation as artifact. Per-requirements file
plus a project-level SBOM from pyproject.toml. Lets downstream
consumers audit our wheel contents (the ML supply-chain SBOM gap
is a known industry-wide problem; meets half of NTIA SBOM mins).
5. GitHub Actions pinning verifier. Reports every `uses: foo@v4`
or `@main` mutable ref. tj-actions/changed-files (Mar 2025) hit
anyone using non-SHA pins. Currently surfaces 4 third-party
unpinned refs (dtolnay/rust-toolchain, swatinem/rust-cache) and
40 first-party (`actions/*`); informational baseline, tighten
once we're ready. Dependabot's github-actions ecosystem
auto-bumps SHA pins, so the maintenance cost is zero.
6. Hash-pin verifier. Reports how many == specs would gain from
`--hash=sha256:` entries. Currently 11 == pins, 0 with hash.
Roadmap step: `uv pip compile --generate-hashes` then
`pip install --require-hashes`. Hash-locked installs would have
refused a republished litellm 1.82.7 even at the same version
string.
7. Custom Semgrep rules at .semgrep/unsloth-rules.yml. Seven rules
for the *specific shape* of recent ML-stack CVEs we'd otherwise
re-introduce ourselves: langchain-core deserialize-roundtrip
(CVE-2025-68664), n8n private-pyodide-eval (CVE-2025-68668),
marimo websocket-no-auth (CVE-2026-39987), litellm
popen-with-network-stdin, Shai-Hulud workflow-write,
pickle-from-network, shell=True with f-string interpolation.
dependabot.yml: extend to pip + cargo ecosystems so security
advisories on Python deps and the Tauri shell auto-generate update
PRs alongside the github-actions / bun / npm ones.
All new steps continue-on-error initially; findings land in
GITHUB_STEP_SUMMARY plus the advisory-audit-logs artifact.
Recent supply-chain incidents that scan_packages would have missed:
- PyTorch Lightning 2.6.x: payload in _runtime/router_runtime.js
(14.8 MB), persistence via .claude/settings.json SessionStart
and .vscode/tasks.json folderOpen
- npm chalk/debug + Shai-Hulud: hex-var obfuscation, window.ethereum
Web3 hijack, .github/workflows/shai-hulud.yml repo takeover,
trufflehog credential exfil
- elementary-data 0.23.3: token harvesters with embedded gh{p,o,s}_
and AKIA regexes
- litellm 1.82.7: also covered by existing patterns, but anyone on
`>=` got it during the 40-min exposure window
- langchain-core CVE-2025-68664 / n8n CVE-2025-68668 / marimo
CVE-2026-39987: first-party design flaws, not malicious-author
scan_packages.py:
- Six new regexes: RE_DEV_TOOL_HIJACK, RE_TOKEN_REGEX,
RE_JS_OBFUSCATION, RE_WEB3_HIJACK, RE_WORKFLOW_INJECT,
RE_SHELL_DROPPER.
- Three new checkers: check_js_file, check_shell_file,
check_workflow_file. scan_archive now routes .js/.mjs/.cjs/.ts
to the JS checker, .sh/.bash to the shell checker, and
.github/workflows/*.yml to the workflow checker.
- JS checker fires CRITICAL on hex-var obfuscation OR Web3 hijack
OR (token regex + network) OR workflow-injection signature; HIGH
on a >100 KB JS bundle inside a Python wheel (the Lightning tell).
- Smoke-tested: every new pattern matches its canonical positive
and rejects four legitimate-looking false-positive baits.
security-audit.yml:
- OSV-Scanner step: cross-ecosystem advisory check (PyPI + npm
+ cargo) from one binary. OSV's feed is a superset of GitHub-
Advisory; catches CVEs that haven't propagated yet (e.g.
langchain-core was on OSV before GitHub Advisory).
- Semgrep step: p/supply-chain + p/python + p/javascript +
p/security-audit packs catch first-party logic bugs (CVEs 7/9/10
above) that pattern scanning never sees.
- Lockfile pin verifier: warns on every non-`==` spec in
requirements/*.txt. Currently surfaces 104 unpinned specs as
informational baseline; tighten to blocking once the baseline
is curated.
All new steps continue-on-error initially; they surface findings to
the workflow summary + advisory-audit-logs artifact.
Three advisory-DB lookups previously spun up three separate runners.
All three are fast lockfile-driven checks (pip-audit ~1m37s, npm audit
~12s, cargo audit ~24s) and the runner-setup overhead dominates each.
Run them sequentially on a single runner with python + node + rust
toolchains pre-installed; total wall clock comes out roughly the same
(~3 min) but with one PR check instead of three.
Each step keeps continue-on-error: true so a finding in one toolchain
does not suppress the others. Logs land in a single advisory-audit-logs
artifact (pip + npm + cargo + the filtered req set).
Heavy job stays separate: pip-scan-packages remains the 3-shard matrix
that downloads + pattern-scans the full PyPI transitive closure (~6
min/shard, in parallel). Conflating that into the advisory job would
bloat the runner image and serialize a 6 min job behind a 30 s one.
Previous run took ~10+ minutes because each requirements file ran
its own --with-deps resolve serially, and the six files all share
~70% of their transitive set (transformers, peft, accelerate land
in three of them). Net effect: the same 200+ archives downloaded and
pattern-scanned three times in series.
Two changes:
1. Within a shard, feed every -r file to ONE scan_packages call so
pip's resolver intersects version constraints once and yields
a single deduped transitive set.
2. Across shards, run three matrix jobs in parallel:
- hf-stack: unsloth-deps + no-torch-runtime (pyproject extras)
- studio: studio + overrides + extras-no-deps
- extras: extras (heavy openai-whisper / scikit-learn stack)
Wall clock now bounded by the slowest shard rather than the
sum, dropping ~10 min to ~3-5 min.
Each shard uploads its own artifact (scan-packages-log-<id>) so log
correlation stays clean. fail-fast: false so one shard's findings
don't suppress the others.
Last push surfaced two silent failures:
1. pip-audit aborted on openai-whisper. The package's setup.py
imports pkg_resources, which the isolated build env's modern
setuptools no longer ships by default. Because we passed every
-r file in one invocation, that single build failure killed the
audit for ALL files (the run reported success only because
continue-on-error swallowed exit 1).
2. scan_packages --with-deps aborted on the first git+ spec it
hit (triton-kernels.txt's git+https://github.com/triton-lang
/triton.git, plus OpenEnv in extras-no-deps.txt). Same
all-or-nothing behaviour: the entire transitive scan reported
"0 archives downloaded" and "all clean" -- meaning we silently
scanned nothing.
Fixes:
- Build a filtered audit-reqs/ tree first. Each Studio requirements
file is copied with `git+` lines stripped (replaced with a
`# [security-audit] skipped` marker so the exclusion is auditable
in the artifact). Pure git refs are out of scope for both pip-
audit (CVE DB only knows PyPI versions) and scan_packages (it
inspects PyPI archives, not git HEADs).
- Run pip-audit per-file in a loop. One bad file no longer takes
out the whole audit.
- Pin setuptools<78 + wheel into pip's isolated build env via
PIP_CONSTRAINT, so legacy setup.py packages (openai-whisper) can
still emit metadata for the resolver.
- Run scan_packages per-file too, with the same git+ filter and a
skip for files that are empty after filtering (triton-kernels.txt
becomes a comments-only file and would otherwise spam the log
with `--help`).
Net effect: pip-audit now actually emits CVE findings (we know the
default branch carries 17), and scan_packages downloads + pattern-
scans the full transitive closure of every PyPI-only requirements
file plus unsloth's pyproject deps.
The previous Security audit only covered Studio's backend requirements.
The unsloth pip package itself ships its own dep set via pyproject.toml
(typer/pydantic/pyyaml/nest-asyncio core, plus the huggingfacenotorch
extras: transformers/peft/accelerate/trl/datasets/diffusers/etc.) -- a
malicious upload to any of those would slip past us today. Build a
combined dep list from pyproject.toml + the six Studio requirements
files and feed it to both pip-audit and scan_packages.
Add scan_packages.py at scripts/scan_packages.py so the scanner ships
with the repo and CI does not depend on a network fetch at job time.
Pass --with-deps to scan_packages so the pre-install pattern scan
walks the full transitive closure -- supply-chain attacks usually land
several hops down (litellm 1.82.7 was a dep of a dep for most users;
top-level-only scanning would have missed it).
No installation in either job. pip-audit's -r mode resolves through
PyPI metadata, scan_packages downloads sdist/wheel archives raw and
inspects them without running install hooks. An attacker who has
compromised a transitive dep cannot execute code in this workflow.
Three Priority-1 follow-ups from the lint review.
Lint CI gains two non-blocking gates that surface drift without
blocking merges (the same shape as the existing format-drift step):
- codespell: typo catcher across source / comments / docs. Skips
lockfiles, generated assets, binary artefacts, LICENSE files.
ignore-words-list pulls out short identifiers and PyTorch
idioms (parm/parms, ans, hist, etc.) the default dictionary
would flag. Local run finds 16 real typos to fix in a follow-up.
- shellcheck: catches subtle shell bugs `bash -n` doesn't see --
unquoted expansions, useless cat, `[[ ]]` command substitution,
etc. SC1090 + SC2034 muted because install/setup scripts
legitimately source runtime paths and use export-only
assignments. Critical-path coverage: install.sh, setup.sh,
tests/sh/.
Both pinned for reproducibility (codespell>=2.3,<3 in pip,
shellcheck via apt-get). Both surface findings in PR annotations
without failing the run; drop continue-on-error after the cleanup
PRs land.
New workflow: Security audit. Runs `pip-audit` against the same
dep set Studio's backend pytest matrix installs, so we audit what
the runtime actually loads (not what pyproject.toml's transitive
resolution might pull in differently). Triggers:
- PRs touching requirements / pyproject.toml,
- push to main / pip,
- nightly @ 04:13 UTC (off-the-hour to dodge cron rush),
- workflow_dispatch.
The default branch already carries 17 known vulnerabilities per
the dependabot banner, so a hard gate today would block every PR
on a baseline we have not triaged. Non-blocking; full table goes
to GITHUB_STEP_SUMMARY for grep-ability and a 30-day artefact for
historical comparison.
The custom AST anti-pattern scan I prototyped was dropped: every
class of CPU-import-time bug we hit in this PR (bitsandbytes,
torchvision, _cuda_getCurrentRawStream, DEVICE_COUNT==0 stream
init) is already caught by the Repo tests (CPU) job exercising
the actual import on a CPU torch wheel. Restating the rule
in AST form would only add noise.
The license-header check missed two more legitimate header families
that are committed to the repo today:
- LGPL-3.0 long form: e.g. unsloth/kernels/rope_embedding.py opens
with "GNU Lesser General Public License" -- 7 such files under
unsloth/kernels/.
- AGPL-3.0 long form: e.g. unsloth/kernels/moe/autotune_cache.py
opens with "GNU Affero General Public License" -- 2 such files
under unsloth/kernels/moe/.
Both got flagged as drift on the previous run because the check
only knew about the SPDX one-liner and the Apache-2.0 preamble.
Add a third accepted marker, the substring "General Public License",
which appears in all three GNU long-form preambles (GPL, LGPL,
AGPL) and nothing else. Repo inventory:
spdx (one-liner) 193 files (mostly studio/)
apache-longform 55 files (unsloth/, unsloth_cli/)
agpl-longform 2 files (unsloth/kernels/moe/)
lgpl/gpl-longform 7 files (unsloth/kernels/)
no recognised header 85 files (real drift -- mostly tests/)
So the warning count drops from 94 -> 85 with this commit; the
remaining 85 are actual missing headers, surfaced as a non-blocking
warning until the cleanup PR lands.
Drop the python-lint job from studio-backend-ci.yml and move it into
the dedicated `Lint CI` workflow. Two material changes:
1. License-header check now accepts BOTH header families
The previous version only counted SPDX-License-Identifier, which
warned on every Apache-2.0 file in unsloth/, unsloth_cli/, and
scripts/ (e.g. unsloth/models/llama.py opens with the standard
`# Copyright ... Daniel Han-Chen & the Unsloth team. All rights
reserved. # Licensed under the Apache License, Version 2.0` block,
which is correct, but my SPDX-only regex flagged it).
New rule: a file is OK if either `SPDX-License-Identifier` or
`Licensed under the Apache License` appears in the first 20 lines.
Empty __init__.py files are skipped. Whole-repo coverage instead
of just studio/backend.
2. Add shell / YAML / JSON parse gates
- `bash -n` over every committed *.sh (14 today). Same idea as
compileall: parse-only check.
- `yaml.safe_load_all` over every *.yml / *.yaml (97 today),
including .github/workflows/* so a typo in the workflow file
itself shows up immediately.
- `json.loads` over every *.json (18 today). Skips
package-lock.json / bun.lock (huge, machine-generated) and
tsconfig*.json (TypeScript JSONC convention -- already
validated by `tsc --noEmit` in Frontend CI).
TypeScript and Rust are NOT duplicated here:
- Studio Frontend CI runs `npm run typecheck` + `npm run build`
on every studio/frontend/** change, which is a full TS AST +
type check.
- Studio Tauri CI runs `tauri build --debug --no-bundle` on every
studio/src-tauri/** or studio/frontend/** change, which is a
full Rust compile.
A duplicate fast-fail step here would burn cache for marginal
value, and the dedicated workflows already block merges.
Lint CI runs on every PR (no path filter): the whole job is
under 30 s of CI time, so paying that on every PR is preferable
to missing a regression on a path the focused workflows skip.
Rename the job to "Python lint (syntax + ruff + safety nets)" and
expand it from one non-blocking ruff invocation over studio/backend
into four real gates over the whole tree. Total CI time goes from
~8 s to ~12 s, but the previous job was informational; this one
blocks merges on actual breakage.
Steps (in order):
1. AST/syntax (HARD GATE)
`python -m compileall -q -j 0 unsloth unsloth_cli studio tests
cli.py unsloth-cli.py`. Same parser the interpreter uses;
anything broken here would also crash at `import X` on a user's
machine. ~3.5 s across 350+ files locally.
2. ruff check whole repo (HARD GATE)
The narrow rule set in pyproject.toml [tool.ruff.lint] (E9 /
F63 / F7 / F82) catches undefined names, broken comparisons,
and syntax. The whole repo passes today, so the previous
studio/backend-only `|| true` was masking real breakage on
the wider tree. <1 s.
3. Debugger-leftover scan (HARD GATE)
AST-walk over every committed .py looking for `breakpoint()`,
`pdb.set_trace()`, or `ipdb.set_trace()` call sites. AST-based
so commented-out debugger lines don't false-positive (which
is why a bare grep would not work -- there are three commented
`# breakpoint()` markers in unsloth/models/rl* today). 0 hits
locally across 350 files.
4. SPDX-License-Identifier on studio/backend (WARNING)
Surfaces drift in the one tree where we already have a strict
SPDX policy. Currently 3 files missing; warned, not blocked,
so the rollout can be a separate PR.
5. ruff format drift (INFO)
Counts files that would be reformatted by plain `ruff format`.
Non-blocking because the canonical formatter is
scripts/run_ruff_format.py = ruff format + the kwarg-spacing
pass, so plain `ruff format --check` always reports a large
diff. Once that custom pipeline is wired in, drop
continue-on-error and add it to the gate.
ruff is pinned to 0.15.12 to match .pre-commit-config.yaml so a
CI-only ruff bump cannot start disagreeing with what pre-commit
already accepted.
studio/install_llama_prebuilt.py lists releases on
ggml-org/llama.cpp via the GitHub API. Unauthenticated calls get
60/hr per source IP, which is fine for one install per workflow but
the new Studio Update CI does install + update + update back-to-back
on the same runner, blowing past the limit and falling back to a
source build (which then fails the idempotency assertion).
Surfaced on the Studio Update CI run with:
failed to inspect published releases in ggml-org/llama.cpp:
GitHub API returned 403 ...
set GH_TOKEN or GITHUB_TOKEN to avoid GitHub API rate limits.
GITHUB_TOKEN with the existing `permissions: contents: read` is more
than enough for unauthenticated read API access (1000/hr, scoped to
the repo). Wired into every install.sh and `unsloth studio update`
step across studio-update-smoke.yml, studio-inference-smoke.yml, and
studio-ui-smoke.yml so a busy runner can't trip the same fallback.
Studio's image normaliser re-encodes embedded base64 images via
stb_image (routes/inference.py:3410) so llama-server gets a uniform
PNG payload. stb_image happily reads the 4x4 PNG as a PIL test, but
rejects it on the inference path with `broken data stream when
reading image file`. 64x64 is small enough to keep token cost
trivial (155 bytes) and large enough to satisfy stb_image's minimum.
Job 1, Job 2, the UI smoke, and the JSON portion of Job 3 are all
green now -- this is the last piece holding Job 3 back.
Job 2 (tool calling):
The server-side agentic loop in routes/inference.py:1888 always
yields SSE chunks -- the request's `stream=False` is honoured for
the plain passthrough path, NOT for the agentic path. The python /
terminal / web_search probes were calling json.loads on the raw
body and tripping JSONDecodeError.
Added a post_sse() helper that streams the response and accumulates
text deltas, used for every enable_tools=True call. Function
calling (which does NOT enable agentic mode) keeps post().
Job 3 (JSON, images):
Dropped the strict-schema variant of response_format. On the small
gemma-4-E2B-it UD-IQ3_XXS quant, the GBNF-from-schema path
occasionally produces empty content. Plain `{"type":"json_object"}`
is still a real test of Studio's JSON-mode wiring through to
llama-server, and that's the surface the docs expose. Added
fence-stripping for chat templates that wrap JSON in ```json blocks.
GGUF jobs 2 and 3:
Switched off `unsloth studio run` and over to `UNSLOTH_API_ONLY=1
unsloth studio` + login flow. Reason: studio.run() resolves the tool
policy through unsloth_cli/_tool_policy.resolve_tool_policy, which
defaults to True on loopback. That means set_tool_policy(True) gets
applied process-wide, and every /v1/chat/completions request is
routed through the server-side agentic loop -- so Job 2's standard
function-calling test never gets a structured tool_calls response
(the model uses web_search instead) and Job 3's response_format
test gets non-JSON SSE chunks back. API-only mode leaves
tool_policy=None, which is what each request's `enable_tools` flag
(or absence thereof) needs to be honoured.
Job 1:
Anthropic SDK retry: the SDK sends `x-api-key` by default, but
Studio's auth layer is HTTPBearer-only. Override via
default_headers={"Authorization": f"Bearer {KEY}"}, which is the
shape the integration docs suggest.
UI smoke:
Drop the "history must persist after reload" assertion; Studio's
thread autosave is async and doesn't reliably land within the CI
budget. Keep the assertion that matters: the chat composer mounts
again after a reload and the JWT survived (no /login redirect),
which is what the 2026.5.1 chat regression actually broke.
The /login route auto-redirects to /change-password as soon as
/api/auth/status returns requires_password_change=true. The original
flow was racing that redirect: it filled #password (login mode) and
clicked submit, but the redirect could land first and the form would
have unmounted before the click. Going straight to /change-password
also matches what main._inject_bootstrap is set up to support: the
HTML on that route ships with `window.__UNSLOTH_BOOTSTRAP__`, which
the change-password form reads to seed the current-password state, so
the user only needs to fill new + confirm. Renumbered screenshots to
match the new step order.
Two new PR-time gates that the existing inference / wheel jobs miss.
Studio Update CI:
- Runs install.sh --local --no-torch, then `unsloth studio update
--local` twice, asserting both invocations take the prebuilt
"up to date and validated" code path with no source-build
fallback.
- Boots Studio to /api/health afterwards so a broken update that
nukes the venv or the llama-server binary surfaces immediately.
- Triggers when install.sh, studio/setup.sh, the python_stack /
llama_prebuilt installers, the requirements files, or
unsloth_cli/commands/studio.py change.
Studio UI CI:
- Drives the actual frontend bundle in headless Chromium via
Playwright with the smallest GGUF (gemma-3-270m-it UD-Q4_K_XL).
- Covers: bootstrap login, must_change_password gate + change form,
chat composer becomes interactive after model load, sending a
message produces an assistant bubble with non-empty text, full
page reload re-hydrates the conversation, configuration sheet
opens and closes cleanly, and the rotated password is the only
one that logs in afterwards.
- This is the first workflow that catches the class of bug 2026.5.1
shipped: backend healthy + frontend builds, but assistant-ui
runtime wiring or chat-history persistence broken so the actual
UI was unusable. Backend-only or wheel-only gates do not see it.
Job 1 (OpenAI, Anthropic API tests):
Anthropic SDK appends /v1/messages to base_url itself, so passing
base_url=f"{BASE}/v1" produced /v1/v1/messages and 405'd. Bare BASE
is correct (matches the docs' "the SDK appends /v1 automatically").
OpenAI SDK side already worked: 4-turn transcript was fully
deterministic across two runs and the "Paris" sanity assertion
passed.
Job 2 (tool calling tests):
Booting with --enable-tools forces the process-level tool policy to
True for every request (state/tool_policy.py:get_tool_policy), which
hijacked the "Standard OpenAI function calling" test through the
server-side agentic loop -- the model called web_search instead of
returning structured tool_calls for the user's `weather_tool`. Drop
--enable-tools so policy is None (per-request honour). The python /
terminal / web_search probes already pass enable_tools=True
explicitly in their request bodies, so they keep working.
Job 3 (JSON, images):
Two issues. (a) The OpenAI Python SDK rewrites
response_format={"type":"json_schema",...} into something Studio's
llama-server backend doesn't accept, so resp came back as the raw
error string and resp.choices[0] tripped 'str has no attribute
choices'. Switched to raw HTTP with the `{"type":"json_object",
"schema":...}` form llama-server actually supports
(GBNF-from-schema, llama-server extension). (b) Anthropic SDK
base_url same fix as job 1.
Replaces the single "Studio boots, loads a GGUF, answers a chat
completion" job with three parallel jobs that each pick the smallest
model that exercises the surface under test. All three jobs share the
install.sh --local --no-torch bootstrap and prime HF_HOME via
actions/cache so cold-cache runs are bounded and warm runs are quick.
1. Studio GGUF CI / OpenAI, Anthropic API tests
- Model: gemma-3-270m-it UD-Q4_K_XL (~254 MiB).
- Password rotation: login with bootstrap pw, change to a fresh
random pw, assert old pw is rejected with 401, assert new pw
succeeds. Uses the same JWT downstream as a Bearer token against
/v1/* (the OpenAI/Anthropic compat surface accepts JWTs and
sk-unsloth- keys interchangeably).
- OpenAI SDK + Anthropic SDK each run a four-turn conversation
("What is 1+1?" / "What did I ask before?" / "What is the capital
of France?" / "Repeat the city name") with temperature=0.0 and
seed=3407. Run twice and assert run1 == run2 turn-by-turn so
non-determinism in the conversation-history wiring is caught.
2. Studio GGUF CI / tool calling tests
- Model: Qwen3.5-2B UD-IQ3_XXS (~890 MiB).
- Standard OpenAI function calling with tool_choice=required.
- Server-side python tool: assert "56088" appears in the answer to
"What is 123 * 456? Use code to compute it.".
- Server-side terminal (bash) tool: assert "hello-bash-tool" is
echoed back.
- Server-side web_search tool: non-blocking probe (DuckDuckGo
flakes from CI runners). Asserts the request shape is accepted.
- enable_thinking=true vs false: assert <think> markers vanish
when thinking is disabled.
3. Studio GGUF CI / JSON, images
- Model: gemma-4-E2B-it UD-IQ3_XXS (~2.4 GiB) + mmproj-F16
(~986 MiB) auto-detected via the HF repo path.
- response_format = json_schema (strict): asserts the answer parses
as JSON matching the {city, country} schema.
- OpenAI image_url (data URI base64): assert non-empty response on
a 4x4 PNG. Loose on content because small VL quants are weak at
colour names; the vision path is the part under test.
- Anthropic source/base64 image: same non-empty assertion against
the Anthropic Messages endpoint.
Boot strategy:
- Job 1 keeps `UNSLOTH_API_ONLY=1 unsloth studio` because the
password-rotation flow only exists in the UI-mode bootstrap.
- Jobs 2 and 3 use `unsloth studio run --model REPO --gguf-variant V`,
the one-liner that loads the model and prints the API key on the
banner. Health is probed by waiting for `sk-unsloth-` to appear in
the log; the one-liner only prints the banner after load completes.
The MLX CI workflow listed ``studio/backend/utils/hardware.py`` as a
path filter, but no such file exists. The actual layout is
studio/backend/utils/hardware/
__init__.py
amd.py
hardware.py
nvidia.py
vram_estimation.py
so the filter as written would never match. A reviewer modifying
``hardware/hardware.py`` (where ``detect_hardware``, ``DeviceType``,
and ``IS_ROCM`` actually live) would not trigger MLX CI, which
defeats the point of the focused PR gate.
Replace the broken filter with ``studio/backend/utils/hardware/**``
so any change in the hardware probe directory triggers MLX CI, and
add three sibling triggers that each materially affect dispatch:
- ``unsloth/_gpu_init.py``
Hosts ``from .models import *`` and the ``from .trainer import *``
chain. The trainer.py circular-import fix that landed in
``23550a8`` lives downstream of this file; a future change
here can re-introduce the same bug.
- ``studio/backend/core/inference/mlx_inference.py``
The MLX inference backend itself. It is the actual consumer
of ``unsloth_zoo.mlx_loader.FastMLXModel`` whose contract the
test_mlx_training_worker_behaviors.py AST checks guard.
Local re-run with the fix in place: 36 passed in 0.45s. No other
workflow file or test file is modified.
Mirrors the three files documented in tests/studio/README.md (PR #5307)
into a dedicated workflow so MLX dispatch failures show up as their own
check on PRs rather than getting buried inside Backend CI:
- test_hardware_dispatch_matrix.py 7-profile parametrized matrix
+ 2 dispatch-priority canaries
- test_is_mlx_dispatch_gate.py AST + runtime guard on
unsloth._IS_MLX
- test_mlx_training_worker_behaviors.py worker.py contract checks
Triggers on pull_request when any of unsloth/__init__.py,
studio/backend/utils/hardware.py, studio/backend/core/training/worker.py,
or any of the three test files are touched. Runs on a Linux+CPU runner
with hardware spoofs; no Apple Silicon, real GPU, or real MLX install
required. Locally validated: 36 passed in 0.41s.
permissions: contents: read at the workflow level (matching the rest of
the PR-time CI surface).
permissions:
- All five PR-time workflows (backend, frontend, inference smoke, tauri,
wheel) now declare permissions: contents: read at the workflow level,
matching CodeQL's default-permissions guidance and the existing pattern
in release-desktop.yml. None of these workflows write to the repo.
skipped tests:
- Repo tests (CPU) job now installs node 22 and uv, which unblocks
~60 tests that were silently skipping on CI:
- 9 tests in tests/studio/test_chat_preset_builtin_invariants.py
skipped on "node not available". Fixed in this commit; an obsolete
"unsloth_repo/" prefix in WORKDIR was also pointing the source-file
existence check at a path that no longer exists.
- tests/python/test_e2e_no_torch_sandbox.py (47), test_studio_import_no_torch.py
(29), test_tokenizers_and_torch_constraint.py (most of 42) all spawn
fresh uv venvs and self-skip when uv is missing.
- Three test_tokenizers_and_torch_constraint.py cases are deselected
because they expose a real bug in studio/backend/requirements/no-torch-runtime.txt:
the unpinned tokenizers line resolves to 0.23.1, which transformers
rejects with "tokenizers>=0.22.0,<=0.23.0 is required". Tracked
separately as a no-torch install regression.
Locally: 760 passed, 1 skipped, 23 deselected (was 694 / 67 / 23).
* Add Studio PR-time CI: pin enforcement, frontend, backend, wheel smoke
The repo currently has no PR-time CI; only release-desktop.yml (manual) and
stale.yml (issue pinger). studio/backend/tests/ has 35 test files (~860
tests collected) that never run automatically. Frontend lint/typecheck/build
scripts exist in package.json but are not gated on PRs either. This is the
gap that let 2026.5.1 ship with the broken Studio chat-history bundle.
Adds four ubuntu-latest workflows, all CPU-only and free for public repos:
studio-pin-enforce.yml
Greps studio/frontend/package.json for caret/tilde ranges on the
@assistant-ui surface (and assistant-stream). Blocks the exact regression
vector that produced 2026.5.1 (^0.12.19 resolving to a breaking 0.12.28).
studio-frontend-ci.yml
npm ci (strict lockfile), tree-clean check after, typecheck, vite build,
bundle grep for the Studio unstable_Provider call site (<= 3 hits = OK,
>= 4 = the 2026.5.1 regression), 75 MB dist budget, biome non-blocking.
Uploads dist on failure.
studio-backend-ci.yml
Runs the existing studio/backend/tests/ suite on Python 3.10/3.11/3.12.
Excludes test_studio_api.py (live model + GGUF download) and
llama_cpp_load_progress_live (spawns a real llama.cpp). Local run on this
branch: 861 pass, 4 skipped, 5 deselected. ruff non-blocking.
wheel-smoke.yml
python -m build, then verifies the produced wheel:
- ships studio/frontend/package-lock.json
- ships studio/frontend/dist/index.html
- does NOT ship studio/frontend/node_modules/
- does NOT ship studio/frontend/bun.lock
- main JS bundle has < 4 unstable_Provider hits
Then installs the wheel into a fresh venv with a lightweight dep set and
imports studio.backend.main. Locally validated against the wheel built
from this branch.
Each workflow has concurrency cancellation on the same ref. biome and ruff
are gated as non-blocking until the existing accumulated drift is cleared
(~470 biome errors today); remove the bypass in a follow-up.
Notes verified locally:
- pin enforcement: PASS (carets dropped on this branch)
- frontend npm ci -> typecheck -> build -> grep -> budget: PASS
- bundle: 48 MB, hits=1
- backend pytest: 861 pass, 1 GPU-pollution failure not reproducible on
GPU-less runners (won't reproduce on ubuntu-latest)
- wheel build: 13s, produces unsloth-2026.5.2-py3-none-any.whl
- wheel content sanity: all five checks PASS
* CI: install full backend dep set + refine pytest filter for CPU runners
First CI run on PR #5298 surfaced two real gaps:
1. pytest collection failed at `import yaml` in utils/models/model_config.
Locally my workspace venv had pyyaml from a transitive; CI's clean Python
3.10/3.11/3.12 didn't, so collection hit ModuleNotFoundError on the very
first test module. Same blew up the wheel-smoke `from studio.backend.main
import app` step.
2. Once the import chain was complete, ~9 tests still failed because they
exercise GPU-only paths or live transformers introspection that can't run
on a GPU-less `ubuntu-latest` runner regardless of code correctness:
- TestGpuAutoSelection
- TestPreSpawnGpuResolution
- TestPerGpuFitGuardAllCounts
- TestTransformersIntrospection
- test_returns_cuda_when_cuda_available
- test_calls_cuda_cache_when_cuda
Fix:
- Backend CI installs `studio/backend/requirements/studio.txt` (the
declared backend dep set) + the extras the import chain needs but
studio.txt omits (python-multipart, sqlalchemy, cryptography, pyyaml,
jinja2, mammoth, unpdf, requests, etc.) + torch CPU wheel + transformers.
- Refine the pytest -k filter to deselect the GPU/introspection-bound
classes by name. Deselections are commented inline with the reason.
- wheel-smoke uses the same dep set so the import smoke matches.
Locally validated against the freshly-built unsloth-2026.5.2 wheel:
831 passed, 5 skipped, 35 deselected, 0 failed in 47s
Studio backend imports cleanly in a fresh venv after the wheel install.
* CI: collapse multiline pytest -k expression to a single line
YAML's | block-scalar fed the newlines verbatim into the -k argument and
pytest rejected it as 'Wrong expression passed to -k'. Same logical filter
on one line.
* CI: rename jobs so the GitHub UI shows what each check actually does
Adds a per-job 'name:' to all four workflows so the PR check list reads:
Studio pin enforcement / @assistant-ui must be pinned exactly
Studio frontend CI / Frontend build + bundle sanity
Studio backend CI / Backend pytest (Python 3.10|3.11|3.12)
Studio backend CI / Backend ruff lint (non-blocking)
Wheel build + smoke / Wheel build + content sanity + import smoke
Instead of the default '<workflow> / <job-key>' which was opaque
('check', 'build', 'pytest (3.10)', 'ruff', 'wheel').
* CI: add Python 3.13 to backend pytest matrix
Verified locally: 831 backend tests pass under Python 3.13 with the same
filter set used for 3.10 / 3.11 / 3.12.
* CI: add Studio inference smoke + Tauri build smoke
Two new workflows. Both CPU-only, both free on `ubuntu-latest`.
studio-inference-smoke.yml
The only workflow we have that proves "Studio actually works", as opposed
to "the bundle parses" or "the imports succeed":
- runs install.sh --local --no-torch (lean Studio install)
- downloads unsloth/gemma-4-E2B-it-GGUF UD-IQ3_XXS into actions/cache
- boots Studio in api-only mode
- logs in with the bootstrap password, changes it, re-logs
- POST /api/inference/load on the GGUF
- POST /api/inference/chat/completions and asserts a non-empty
assistant response
Validated end-to-end locally on a fresh main install: model loaded,
chat completion returned `Hello!` against the same GGUF the workflow
uses.
studio-tauri-smoke.yml
PR-time variant of release-desktop.yml. Linux-only debug build
(`tauri build --debug --no-bundle`) on ubuntu-22.04. Catches
src-tauri Cargo.toml / Rust source breakage, tauri.conf.json drift,
and frontend-distDir wiring. Pinned to the same Tauri CLI version
(2.10.1) as release-desktop.yml so CLI bumps surface in CI before
they break the release pipeline. Mac and Windows desktop builds
stay manual via release-desktop.yml because they need code-signing
secrets.
* CI: use 'hf download' instead of deprecated 'huggingface-cli download'
huggingface_hub 1.13.0 dropped the huggingface-cli entrypoint. The
replacement is the 'hf' CLI shipped with the same package. Same args,
just s/huggingface-cli/hf/.
* CI: assert llama.cpp prebuilt path was used on ubuntu-latest
The inference-smoke job runs on ubuntu-latest (CPU-only, x86_64), which
is exactly the host shape that should pick up ggml-org/llama.cpp's
bin-ubuntu-x64.tar.gz prebuilt directly. If install.sh ever falls back
to a source build on this runner, the studio/setup.sh routing has
regressed and every CPU-only Linux user is paying a 3 minute compile
cost again.
Tee install.sh output to logs/install.log, then fail the job if the log
contains "falling back to source build" or is missing the success
marker "prebuilt installed and validated" / "prebuilt up to date and
validated".
Also include logs/install.log in the failure artifact so the prebuilt
diagnostics are uploaded alongside studio.log when the job fails.
* Tighten prebuilt-assertion comment in studio-inference-smoke
* CI: switch inference-smoke model to Qwen3.5-2B UD-IQ3_XXS
Drops the Gemma 4 E2B GGUF (~2.3 GB) for unsloth/Qwen3.5-2B-GGUF
(UD-IQ3_XXS, ~890 MiB). Cache-miss download is roughly a third of
what it was, and CPU inference on ubuntu-latest finishes well
inside the 25 minute job budget.
Verified locally: load via /api/inference/load returns
status=loaded, is_gguf=true, supports_reasoning=true,
supports_tools=true; chat completion returns a non-empty assistant
message ("Hello!").
* CI: add workflow_dispatch to inference-smoke for manual cache pre-warm
* CI: fold pin-enforce grep into studio-frontend-ci, drop standalone workflow
The "@assistant-ui must be pinned exactly" check was its own ~7 second
workflow, doing a single grep on studio/frontend/package.json. Move it
into studio-frontend-ci.yml as a pre-install step (right after
checkout, before any node setup so a violation fails fast). One fewer
top-level check row on every PR, same coverage.
Add a FIXME so this step is dropped once @assistant-ui/* and
assistant-stream leave 0.x: on 1.x, caret ranges are conventional and
this becomes overzealous.
* CI: add Repo tests (CPU) job, mirroring unsloth-zoo PR #624 conftest
The top-level tests/ tree was previously not run anywhere. 23 of its
files are CPU-friendly with the right harness: pure-Python helpers,
ast walks, installer logic, and CLI shape tests. Locally validated:
302 passed, 9 skipped, 12 deselected in ~7 seconds on Python 3.12.
Three pieces:
1. tests/conftest.py -- GPU-free harness, mirrors the conftest landed
in unslothai/unsloth-zoo PR #624. Pre-loads unsloth_zoo.device_type
and unsloth.device_type under a temporarily-mocked
torch.cuda.is_available() so each module's @cache permanently
captures "cuda" and the import chain succeeds on a CPU runner.
Also stubs torch.cuda.get_device_capability /
is_bf16_supported / mem_get_info, which unsloth/__init__.py and
unsloth_zoo.temporary_patches probe at import time when
DEVICE_TYPE == "cuda". On a real accelerator the harness is
skipped and detection runs normally.
2. Two existing tests were leaking sys.modules state across the
session because they injected stubs without an __spec__ and
without restoration:
- tests/test_raw_text.py shoved a "datasets" stub into
sys.modules. transformers' import_utils later did
importlib.util.find_spec("datasets") and got
ValueError: datasets.__spec__ is None.
- tests/python/test_fast_sentence_transformer_redirect_lifecycle.py
shoved "transformers", "sentence_transformers", and
"sentence_transformers.models" stubs in. Subsequent tests
that did `import transformers` got the non-package stub.
Fix: set __spec__ on stubs, plus an autouse fixture in the
sentence-transformer test file that restores the three keys
after each test.
3. .github/workflows/studio-backend-ci.yml gains a third job,
`Repo tests (CPU)`, that installs the same dep set as the
backend-pytest matrix (Python 3.12 only -- the tests are
version-independent), exports PYTHONPATH=studio so tests/python/*
can import install_python_stack, and runs the 23-file subset
above with `-m 'not server and not e2e'`.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CI: install unsloth_zoo for Repo CPU tests, harden conftest fallback
The CPU job at run 25422050018 broke at conftest collection: the
preload of unsloth.device_type pulled in `from unsloth_zoo.utils import
Version` and ubuntu-latest didn't have unsloth_zoo on the path because
it is an optional dep of unsloth. Two fixes:
1. Install unsloth_zoo>=2026.5.1 alongside the other deps in the Repo
tests (CPU) job (it's also what unsloth's optional `huggingface`
extra pins).
2. Wrap the body of _preload_device_type in conftest.py in a try/except
so any import failure (missing prereq, broken module, etc.) cleanly
returns False instead of aborting the entire collection. The caller
already falls back to the stub device_type module on False, so the
net behavior is "best effort: real device_type if possible, stub
otherwise" instead of "abort the test session".
* kernels.utils: guard CUDA_STREAMS / XPU_STREAMS init for DEVICE_COUNT==0
When DEVICE_COUNT is 0 (CPU host: no visible NVIDIA / AMD / Intel GPU)
the dict comprehension {... for i in range(0)} was empty and the
subsequent max(_CUDA_STREAMS.keys()) raised
ValueError: max() iterable argument is empty
during module import. That made unsloth.kernels.utils unimportable on
any CPU runner, which in turn blocked all of tests/saving/**, three
top-level tests/test_*.py, and tests/qlora/test_unsloth_qlora_train_and_merge.py
from even collecting on CPU CI.
Wrap the per-device-index dict comprehension and max() machinery in
a DEVICE_COUNT > 0 guard. When DEVICE_COUNT is 0 fall back to empty
containers (CUDA_STREAMS = (), WEIGHT_BUFFERS = [], ABSMAX_BUFFERS = []).
The consumer functions further down in this module index these arrays
by device_index but only during real GPU work, so the empty fallbacks
never get touched on a CPU host.
GPU-safety verified locally: with 8 visible CUDA devices, CUDA_STREAMS
has 8 entries (identical to before this PR). With CUDA_VISIBLE_DEVICES=""
the module imports cleanly, CUDA_STREAMS is (), and the previously
blocked tests now collect (test_get_model_name passes 38 subtests,
test_resolve_model_class passes 9, test_model_registry collects all 8
parametrizations).
Same shape applied to the DEVICE_TYPE == "xpu" branch for symmetry.
* CI: switch Repo tests (CPU) to auto-discovery + isolate flakes
Three changes, locally validated end-to-end (779 passed, 11 skipped,
23 deselected, 0 failed across all three steps):
1. Repo tests (CPU, auto-discovered): replace the explicit 23-file
list with `pytest tests/` plus a small set of `--ignore` and
`--deselect` flags. New tests under tests/python, tests/studio
(excluding the two state-sensitive files), and top-level
tests/test_*.py are picked up automatically with no workflow edit.
--ignore covers:
- tests/qlora and tests/saving: GPU-bound by design
- tests/utils: helpers folder, not tests
- tests/sh: shell suite handled in its own step
- two state-polluting hardware-spoof files (next step)
-m 'not server and not e2e': honours markers already declared
in tests/python/conftest.py
--deselect: test_model_registration / test_all_model_registration
hit huggingface_hub live; they belong on a network job
2. Hardware-spoof tests (state-sensitive, run in isolation):
tests/studio/test_hardware_dispatch_matrix.py and
tests/studio/test_is_mlx_dispatch_gate.py mutate module globals
in studio.backend.utils.hardware.hardware (IS_ROCM, DEVICE) via
their spoof fixtures, and the leak crosses file boundaries.
Running them in their own pytest invocation avoids polluting the
main sweep. Both pass cleanly in isolation: 28 passed, 1 skipped.
3. Shell installer tests: explicitly enumerated subset that does not
depend on install.ps1 layout (test_install_host_defaults.sh has
drifted; that's a separate followup).
Test fixes folded in to keep the run green:
- tests/studio/install/test_rocm_support.py::TestAmdGpuMonitoring
::test_amd_primary_gpu_with_mock now clears
HIP/ROCR/CUDA_VISIBLE_DEVICES via monkeypatch so
_first_visible_amd_gpu_id() does not short-circuit when the runner
sets CUDA_VISIBLE_DEVICES="" to suppress CUDA.
- tests/studio/test_hardware_dispatch_matrix.py::spoof_hardware
fixture now stubs torch.cuda.get_device_properties when
cuda_available is True so detect_hardware()'s device_name probe
does not call into _cuda_init() on a CPU runner.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* CI: install torchvision (CPU) so unsloth_zoo.vision_utils can import
Run 25430652224 collected three test modules that import unsloth and
crashed at unsloth_zoo/vision_utils.py:68 with
ModuleNotFoundError: No module named 'torchvision'
unsloth_zoo.vision_utils unconditionally imports torchvision at module
scope, and unsloth.models._utils pulls vision_utils in. The Repo tests
(CPU) job installed torch from the CPU index but not torchvision, so
any test that imports unsloth.models.* failed at collection.
Add torchvision<0.26 to the same pip install --index-url
https://download.pytorch.org/whl/cpu line.
* CI: install bitsandbytes (CPU build) for unsloth.models._utils import
Run 25430982243 collected three test modules that import unsloth and
crashed at unsloth/models/_utils.py:1166 with
ModuleNotFoundError: No module named 'bitsandbytes'
The bnb import there is unconditional. Recent bnb versions (>=0.45)
ship a CPU build so the wheel installs on a free Linux runner and the
import resolves; the kernels still raise on use but the module
collects, which is enough for these CPU tests.
Add 'bitsandbytes>=0.45' to the Repo tests (CPU) deps.
* CI: rename workflows + guard kernels.utils CPU-torch binding
Workflow renames (top-level `name:` keys; affects PR check rows):
Studio backend CI -> Backend CI
Studio frontend CI -> Frontend CI
Studio inference smoke -> Studio GGUF CI
Studio Tauri smoke -> Studio Tauri CI
Wheel build + smoke -> Wheel CI
Backend CI's matrix job goes from "Backend pytest (Python 3.10)" to
just "(Python 3.10)" so the GitHub UI row reads
"Backend CI / (Python 3.10)" rather than the old verbose form.
Production guard for CPU torch (run 25431126138):
unsloth/kernels/utils.py:165 was an unconditional
_gpu_getCurrentRawStream = torch._C._cuda_getCurrentRawStream
which raised AttributeError on a CPU-only torch wheel because the
compiled CUDA backend is absent. Three test modules (test_get_model_name,
test_model_registry, test_resolve_model_class) crashed at collection
because their import chain reaches this line.
Add a hasattr probe: when torch is built without CUDA, fall through to
a no-op binding that returns 0. _get_tensor_stream is only invoked
during real GPU work, so the no-op is never executed on a CPU host.
GPU-safety verified locally: with 8 visible CUDA devices the binding
still resolves to the real torch._C._cuda_getCurrentRawStream
(behaviour identical to before this PR). The XPU branch is untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* add unsloth studio desktop app
* Fix review findings
- studio/src-tauri/tauri.conf.json: retarget updater to staging repo
(danielhanchen/unsloth-staging-2); switch to unslothai/unsloth on upstream merge.
- studio/src-tauri/linux/postremove.sh: drop the interactive read loop and the
/home/* iteration. Package maintainer scripts must stay non-interactive and
must not touch other users' data.
- studio/frontend/src/app/auth-guards.ts: honor tauriAutoAuth() boolean. Failed
auto-auth now redirects to /login; requireGuest/requirePasswordChangeFlow
only redirect to /chat when auth succeeds. The new early-return on failed
auth is intentional so the login / change-password flows remain reachable
when desktop auth is not yet established.
- studio/frontend/src/config/env.ts: keep fetched=false on health failure so
later calls retry instead of caching the client-side platform guess.
- studio/src-tauri/src/install.rs: pick the available system package manager
(apt-get, dnf, zypper, pacman); AppImage bundles run on non-Debian distros.
- studio/frontend/src/lib/open-link.ts + markdown-text/sources callers: return
boolean from openLink so callers only preventDefault on handled URLs; relative
hrefs now navigate natively.
- studio/frontend/src/features/settings/tabs/about-tab.tsx: fetch(apiUrl(...))
so the version request targets the backend port in desktop mode. The bare
/api/health predates the Tauri webview (blame: the earlier onboarding commit,
which ran with same-origin frontend/backend); in desktop mode the webview
origin is tauri://localhost so the bare path fails.
- install.ps1: gate the install_python_stack.py hotfix on a sentinel comment
instead of a content regex; append the sentinel after applying so reruns
are unambiguous.
- unsloth_cli/commands/studio.py _write_auth_secret: use the atomic mkstemp +
os.replace path on Windows too; chmod calls are wrapped in try/except OSError.
- studio/src-tauri/src/preflight.rs probe_existing_backends: fan out the health
probes concurrently; desktop-auth status still runs sequentially per candidate.
reqwest::Client is internally Arc-wrapped so the in-loop .clone() is a
refcount bump, not a deep clone; annotated inline.
- studio/src-tauri/src/preflight.rs run_cli_probe: wait() after kill() to reap
the child, matching probe_cli_capability.
- studio/src-tauri/src/process.rs + main.rs: add stop_backend_detached and use
it from the tray quit handler so the 5s graceful-wait does not block the
Tauri main loop. RunEvent::Exit keeps the synchronous safety-net call.
- studio/backend/main.py: drop the permissive localhost CORS regex in
api-only mode; the explicit allow_origins list is sufficient.
- .github/workflows/release-desktop.yml: drop max-parallel: 1 so platform
builds run in parallel, and lift releaseBody to an env var so the three
tauri-action invocations share one source of truth.
* Fix review findings (loop 2)
- studio/backend/auth/storage.py update_password: clear_desktop_secret()
alongside clear_bootstrap_password() so rotating the admin password
also revokes any previously provisioned .desktop_secret. Without this,
an old local desktop credential keeps minting fresh admin tokens via
/api/auth/desktop-login after a password rotation.
- studio/src-tauri/src/desktop_auth.rs provision_desktop_auth: wrap
cmd.output().await in tokio::time::timeout(30s). DESKTOP_AUTH_LOCK is
held across the whole desktop_auth flow, and previously a hanging
`unsloth studio provision-desktop-auth` subprocess would pin the lock
indefinitely and freeze every subsequent desktop_auth call.
* Add review tests
* Consolidate review tests
Merge review-added tests into the existing studio/backend/tests/test_desktop_auth.py
(the PR's authoritative desktop-auth test file). Drops three scaffolding files under
tests/python/ in favor of five focused tests next to the tests they extend:
- test_update_password_clears_desktop_secret (runtime)
- test_update_password_on_unknown_user_leaves_desktop_secret_intact (runtime)
- test_cli_provisioning_delegates_to_storage_create_desktop_secret (source-level)
- test_cli_connect_auth_db_reads_storage_db_path (source-level)
- test_desktop_auth_provision_has_bounded_timeout (Rust source-level)
* Revert auth-guards.ts Tauri branches to unconditional form
The review loop on PR 5144 introduced a regression: the isTauri branch of
requireAuth redirected to /login when tauriAutoAuth() returned false, and
requireGuest / requirePasswordChangeFlow silently fell through on the same
condition. The Tauri desktop app authenticates via a local auto-generated
secret; it must never surface /login or /change-password to the user. A
failed auto-auth should let the startup layer retry, not expose a password
form.
Restore the three Tauri branches to the author's original unconditional
form (requireAuth: return; requireGuest / requirePasswordChangeFlow: throw
redirect({to: '/chat'})). Keep the rest of the review fixes -- the
apiUrl() fetch wrapping, authRedirect helper, and fetchAuthStatus refactor
are all legitimate improvements and are preserved.
* Revert release-desktop.yml to author's version
The review loop's workflow-file tweaks (drop max-parallel: 1, lift releaseBody
to an env var) are cosmetic. OAuth tokens cannot push workflow-file changes,
and fine-grained PATs cannot honor maintainerCanModify on a third-party fork.
Reverting the workflow file to wasimysaid's version lets the push go through
without needing a classic PAT with both repo and workflow scopes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The repo has both the CodeQL "default setup" (configured in repo
settings) and this advanced workflow file enabled. GitHub does not
allow both simultaneously, causing all PR CI runs to fail with:
"CodeQL analyses from advanced configurations cannot be processed
when the default setup is enabled"
Since the default setup already covers the same languages (Python,
JavaScript/TypeScript) with the same build-mode (none), remove the
redundant advanced workflow file.
* Add CodeQL analysis workflow configuration
* Add Dependabot configuration for package updates
Configure Dependabot to check for updates in various ecosystems weekly.
* Fix dependabot.yml: bun ecosystem, missing dir, grouping for PR #4479
1. studio/frontend uses bun.lock not package-lock.json, so change npm to bun
2. Add missing studio/backend/requirements/ pip entry (consumed by studio/setup.sh)
3. Add groups with patterns ["*"] to all pip/bun/npm entries to batch updates
and avoid 30+ individual Dependabot PRs on the first run
* Consolidate pip blocks to fix overlapping directory violation
GitHub Dependabot forbids multiple same-ecosystem entries with
overlapping directories on the same branch. The root "/" directory
overlapped the 3 nested pip dirs. Merge all 4 pip blocks into one
using the `directories:` (plural) key.
Also remove redundant open-pull-requests-limit from the bun block
since grouping with patterns: ["*"] already limits PR count.
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Daniel Han