Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the
chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm
/ XPU; this covers the hardware diffusers serves poorly, consuming the same
split GGUF assets Studio already curates.
- sd_cpp_args.py: pure sd-cli command builder. Maps the family to its
text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1
CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential)
to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu /
--vae-tiling / --diffusion-fa), so one user knob drives both engines.
- sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary()
with the same precedence as the llama finder (env override, then the Studio
install root, then in-tree, then PATH), an is_available/version probe, and a
one-shot subprocess generate that streams progress and returns the PNG.
runtime_env() prepends the binary's directory to the platform library path
so a prebuilt's bundled libstable-diffusion.so resolves.
select_diffusion_engine() is the pure routing decision (GPU backends to
diffusers, CPU/MPS to native when present).
- install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt
(macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the
Studio install root. resolve_release_asset() is a pure, unit-tested
host-to-asset matrix.
- scripts/sd_cpp_smoke.py: end-to-end native generation harness.
Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine,
routing, runtime env, and the installer resolver. Full diffusion suite 166
passing.
Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both
generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group
offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the
dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images.
Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant
(fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao
NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8
stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run
before placement; status reports the mode actually engaged. This is the lean
realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the
3045-line port.
Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the
bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE
option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and
both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default);
size it per model with the Phase 5 quality harness. diffusion_bench gains
--text-encoder-quant.
129 CPU tests pass.
Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3)
storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16
compute dtype while normalisations and embeddings stay full precision. Applied
before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder
dense). status reports which encoders were cast.
Verified on Z-Image (B200, balanced/group mode where the encoder stays resident):
generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload)
at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR
vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off
by default and documented as such, with the Phase 5 harness to size the cost.
127 CPU tests pass.
Add a speed_mode knob (off by default, so the render path stays bit-identical):
default applies channels_last VAE + regional torch.compile of the denoiser's
repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional
compile is gated off for the GGUF transformer (dequantises per-op) and for families
flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image),
so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed
optims run before placement/offload, per the diffusers composition order. status now
reports speed_mode + the optims actually engaged.
Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last',
'tf32'], compile correctly skipped for GGUF; generation works in every mode.
121 CPU tests pass.
Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level,
use_stream) that keeps the transformer flowing through the GPU a few blocks at a
time while the text encoder / VAE stay resident, and fix VAE tiling to drive the
VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the
pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so
status never overstates either, and group falls back to whole-module offload when
the transformer can't be streamed.
Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group
cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 ->
2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names
now match that tradeoff: balanced = stream the transformer, low_vram = offload
every component. auto picks group when the companions fit resident, else model.
112 CPU tests pass.
Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and
VAE tiling/slicing from measured free device memory vs the model's estimated
resident footprint, then applies it to the built pipeline. auto stays resident
when the model fits (byte-identical to the prior resident path), and falls to
whole-module offload when tight; fast/balanced/low_vram are explicit overrides.
Sequential submodule offload is unreliable for GGUF transformers on diffusers
0.38, so it falls back to whole-module offload and status reports the policy
actually engaged.
Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with
no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM
47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost.
73 prior + 35 new CPU tests pass.
Phase 1 of porting the richer diffusion stack onto the image-generation backend.
- Add a compartmentalized device/dtype policy module (diffusion_device.py)
resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA
capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or
fp32, never a silent fp16 that renders a black image.
- Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved
float16 to float32 for those families so they do not produce black images.
- Split the backend locks: a generation holds only _generate_lock, so status,
unload, and a new load are never blocked by a long denoise. Add per-generation
cancellation via callback_on_step_end so an eviction or a superseding load
preempts a running generation; a replacement load waits for it to stop before
allocating, so two pipelines never sit in VRAM at once.
- Validate a load request before the GPU handoff so an unloadable pick never
evicts a working chat model, and reject missing local paths up front.
- Add CPU-only tests for the device policy, dtype guard, lock split and
cancellation, and validate-before-evict, plus a GPU benchmark/regression
script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR
against a saved reference.
* Studio: honor stream=false on the GGUF agentic tool path (#6570)
* Studio: dedup the #6570 non-streaming tool tests and cover cached_tokens
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover the cached_tokens metadata fix and clarify the drain comment (#6570)
* Studio: align the GGUF tool drain naming and tighten its comment (#6570)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
ensure_diffusion_visual_server() downloaded the visual-server release
asset with the unverified download_file() and marked it executable,
bypassing the approved-checksum manifest that gates every other prebuilt
llama.cpp artifact. The backend later auto-discovers that binary and
launches it through DG_VISUAL_BIN, so a compromised or substituted
release asset could place attacker-controlled native code in the install
tree and have it executed under the Studio user.
Require the matched asset to be present in the approved checksum manifest
and download it through download_file_verified() with the published
sha256. A name-matching asset that is absent from the manifest is refused
rather than executed.
Add regression tests covering the verified-download path and the refusal
of an unapproved asset.
* checkpoint preview endpoint
* harden new preview endpoints
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review
* Studio preview: pin adapter, guard streaming submit, robust copy-link
Harden the public per-checkpoint preview surface:
- Pin use_adapter=True in the preview payload sanitizer. Otherwise an
unauthenticated /p caller can POST use_adapter=false, which calls
disable_adapter_layers() on the shared in-memory model without restoring
it; since load_model skips reloads for the same checkpoint, every later
visitor (the page never sends the field) keeps getting base-model output
instead of the fine-tuned checkpoint. Forcing it on also re-enables a
previously disabled adapter and no-ops on merged checkpoints.
- Ignore preview-page submits while a response is streaming. The send
button was disabled but the Enter handler still called requestSubmit(),
so a second request could start before the first reply landed in msgs and
reorder the chat history. Both the keydown and submit handlers now honor
the disabled button.
- Keep the cloudflare-URL polling loop alive across transient startup fetch
errors instead of letting one rejection halt it.
- Build the copy-link from a backend preview_ref (output dir relative to
outputs_root, gated on previewability and the two-segment /p route limit)
so a nested output dir no longer copies a basename-only link that 404s.
Expose preview_ref on training run summaries.
Add route-level security tests (path traversal, payload sanitization,
asset containment, CSP header, HTML title escaping, streaming lock held
until drained) and preview_ref unit tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio preview: Safari-safe submit and adapter pin only for LoRA
Follow-ups from cross-browser and route simulations:
- Preview page: send the message from a shared send() helper called by both
the form submit and the Enter key, instead of form.requestSubmit(). The
latter throws on Safari < 16 and older iOS, which broke Enter-to-send there.
Verified across Chromium, Firefox and WebKit with Playwright.
- Only pin use_adapter=True when the resolved checkpoint is a LoRA adapter
(adapter_config.json present); for a merged checkpoint strip it to None.
A merged model has no adapter to toggle, so forcing it on only produced a
per-request "not a PeftModel" warning. The cross-request base-model
contamination fix still holds for LoRA previews.
Add a merged-checkpoint test asserting use_adapter is stripped to None.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio preview: trim verbose comments
Tighten comments across the preview routes, page, checkpoint helpers, and tests
to short single-line notes; drop ones that just restate the code. No behavior
change (verified comment/docstring-only with comment_tools.py check).
* Harden preview routes for PR #6486
- Return a generic 400 detail on a rejected preview path so the public /p
route never echoes the absolute install path (the real reason is logged
server-side instead).
- Strip confirm_tool_calls, session_id and rag_scope in the preview payload
sanitizer so the public surface stays inert regardless of the tool gate.
- Use Path.is_relative_to for the asset containment check, matching the rest
of the codebase.
- Add img-src 'self' and font-src 'self' to the preview page CSP.
- Preview page: on a mid-stream error keep the streamed text, flag the break,
and restore the prompt so the user can retry; drop the unused --font-sans var.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Pin isolated Node.js installer to committed sha256 digests
The isolated Node installer verified each downloaded archive only against
SHASUMS256.txt fetched from the same nodejs.org origin as the archive, so a
compromised CDN or TLS path could serve a malicious archive plus a matching
checksum and gain code execution when the extracted node is run during the
npm floor check and version probe.
Anchor trust in studio/node_prebuilt_pins.json, a committed manifest of
per-arch sha256 digests, and verify archives against it. The default channel
installs the pinned version and never fetches the remote SHASUMS. Unpinned
lts, latest, or explicit versions fail closed via UnpinnedNodeRefused unless
UNSLOTH_NODE_ALLOW_UNVERIFIED=1, and the refusal is not swallowed by the
keep-existing-on-transient-failure path. Ship the manifest in package-data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review nits on the pinned Node installer
- Drop the unused npm_min_major field from node_prebuilt_pins.json; the floor is
the NPM_MIN_MAJOR module constant and the dead field could silently drift.
- Reword the unpinned-refusal message so it does not tell a user already on the
default to install it, and point the "add a pin" hint at the exact asset.
- Decode the opt-in SHASUMS body with errors="replace" so a non-UTF8 response
yields a clean PrebuiltFallback instead of an uncaught UnicodeDecodeError.
- Tests: assert the refusal message (guards the main() catch order, not just the
exit code), cover malformed-manifest parsing, and drive the opt-in remote-SHASUMS
path end to end through install_prebuilt.
* Tighten comments in the pinned Node installer
Collapse multi-line rationale comments to single lines, drop docstrings on the
obvious internal helpers (load_pins, pinned_sha256), and shorten the manifest
note. Comments/docstrings only; verified code-unchanged via AST comparison.
* Address Codex review: verify pins on existing installs; tomllib fallback
- existing_install_matches now takes an expected_sha and the short-circuit passes
the committed pin, so a version-matching but non-pinned or tampered install (e.g.
from the old remote-SHASUMS path) is re-verified instead of kept. An unpinned
target without opt-in no longer short-circuits on an existing install; it falls
through to the UnpinnedNodeRefused fail-closed path.
- The package-data test uses pytest.importorskip(tomllib/tomli) so it does not
ModuleNotFoundError on the supported 3.9/3.10 interpreters.
* Make the transient-failure keep-existing path pin-aware
The previous commit added the pinned-digest check to the existing-install
short-circuit but not to the post-download-failure fallback, which still kept any
runnable same-version install via existing_install_usable(). A same-version
install whose recorded sha256 is not the pin could therefore be kept on a
transient download failure, the exact artifact the short-circuit rejects. Refuse
to keep a same-version pin-mismatched install there too; a different usable
version is still kept for offline resilience.
* Bump pinned default Node to the current 24 LTS (24.18.0)
Node 24 LTS moved to 24.18.0; since the default channel now resolves straight to
the manifest, a frozen 24.17.0 would downgrade fresh installs and make
UNSLOTH_NODE_VERSION=lts refuse the current LTS as unpinned. Update default_version
and all six per-arch digests (verified against the official SHASUMS256.txt), and
point the test INDEX/short-circuit fixtures at the new LTS.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The fade is recomputed on a group's open/close state flip, but the groups
animate their height, so it measured scrollHeight mid-animation and the
fade could vanish at random. Re-measure on the collapsible animationend,
and add the missing pinnedOpen dep to the recompute effect.
Footer padding moves from pt-3 pb-4 to pb-3 with a conditional top: pt-1.5
when the update card is shown so the fade hugs it, pt-2.5 for the profile
on its own. When the update card is shown, shorten the fade above it
(h-10 -> h-3) so the list reads closer to the card.
* Studio: group the project export menu by Combined and Per chat
Replace the repeated (combined)/(per chat) suffix on every export row with a
section subheading, so the rows read Raw JSONL / CSV / ShareGPT JSONL under
Combined and Per chat headings.
* Studio: group export sections with DropdownMenuGroup
Wrap each Combined and Per chat section in DropdownMenuGroup for screen-reader
semantics, and drop the redundant px-3 already set by DropdownMenuLabel.
* Studio: cap GGUF context to unified memory on Apple Silicon
* Studio: tighten Apple ctx-cap comments and drop the overstated MLX-sync claim
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reserve flat MTP fraction and floor sparse-KV ctx in the Apple unified-memory cap
The Apple Silicon GGUF context cap mirrored the discrete-GPU auto-fit branch but
missed two protections the discrete path already applies:
- It passed the full unified-memory budget with budget_frac=1.0 without first
reserving the flat MTP fraction the discrete path takes off via _pin_fraction.
With an MTP draft whose KV cannot be byte-sized (e.g. Qwen3.6-MTP, #6529), the
cap filled the whole budget and left nothing for the draft, so unified memory
could still over-commit. Reserve _flat_mtp_reserve up front; this is a no-op
when MTP is not engaged.
- It required _can_estimate_kv(), so a GGUF with sparse KV metadata skipped the
cap entirely and launched at full native context. Mirror the discrete
file-size-only fallback and floor the auto context to 4096 when the cache
cannot be sized.
Adds regression tests for both paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments in the Apple unified-memory context cap
Condense the verbose comment blocks in the Apple budget helper, the no-GPU
Metal branch, and the context-fit tests. Comments only, no code change
(verified with ast-based comment_tools check); suite still green.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Copy uv `-c`/`-r` paths to a space-free temp dir on macOS/Linux
uv 0.11.x truncates a constraints/requirements path passed via `-c`/`-r`
at the first space, so `unsloth studio` setup from a repo cloned under a
path containing a space (e.g. `/Users/me/Open Source/unsloth`) fails with:
error: File not found: `/Users/me/Open`
_uv_safe_path() already worked around this on Windows via the 8.3 short
path but returned the space-containing path unchanged on macOS/Linux,
which have no 8.3 equivalent. Extend it to copy the (small, flat)
constraints/requirements file into a space-free temp dir and hand uv the
copy; the temp dirs are removed at process exit. Falls back to the
original path on any error, so it is never worse than before.
Refs unslothai/unsloth#6503
* Route UV_OVERRIDE through _uv_safe_path and fix temp-dir leak (#6503)
The -c/-r fix did not cover UV_OVERRIDE, which uv also truncates at the first
space. On Apple Silicon the overrides file is handed to uv via UV_OVERRIDE at
install time (install_python_stack.py) and during the MLX self-heal
(utils.mlx_repair), so a repo under a path containing a space still broke every
uv call there. Move _uv_safe_path into backend.utils.uv_path_safety so both
sites share it, and route UV_OVERRIDE through it.
Also stop leaking the temp dir when shutil.copyfile fails after mkdtemp, and add
tests for the UV_OVERRIDE channel, the TMPDIR-with-space fallback, the atexit
cleanup, and the no-leak path.
---------
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Tidy verbose Studio launch messages
The reachability failure hint, the loopback deploy block, and the stop
hint were several lines longer than they needed to be, and the banner
printed a Tip line that just repeated the URL already shown above. Shorten
them while keeping the useful detail (cloud provider names, the SSH
local-forward workaround, the relaunch command, the trusted-network and
macOS Ctrl+C notes). No behavior change, output wording only.
* Refine launch-message wording after review
Apply review feedback so the launch messages read well for both experts
and general users:
- reachability hint: restore the 'from your own computer' cue and put the
ssh command on its own line so it stops wrapping, name the cloud rules
precisely (GCP firewall / Azure NSG rule), and restore 'in your browser'
- loopback banner: add the missing colon, drop the Ctrl+C duplication
(the stop hint already covers it), and explain the exposure in plain
language instead of 'exposes the API on every interface'
- stop hint: trim so it fits an 80-column terminal without wrapping
- replace two pre-existing em dashes with --
