A scanned On-Device model's id is its full filesystem path, and the family-token
matcher treats any path segment as a hint, so a folder under a family-named
parent (e.g. /models/qwen-image/misc) tagged an unrelated single-file as
text-to-image. That surfaced it in the Images picker, and a load would evict the
GPU owner before from_single_file failed on the unrelated weights. Match on
Path(model.id).name so only the leaf name counts, for both the image and video
task-tag loops (the video loop is gated on the same detection).
The native sd.cpp one-shot engine re-reads its companion VAE and text-encoder
files from the HF cache on every generation, but the delete-cached guard only
compared the loaded main diffusion repo_id. Deleting a companion repo such as
comfyanonymous/flux_text_encoders while a FLUX GGUF was loaded therefore
succeeded and broke the next generation. Add a loaded_repo_ids() accessor to the
native backend (mirroring loading_repo_ids(), reconstructed from the committed
family's VAE + text-encoder repos) and have the guard refuse those companions
too. Server mode and the diffusers engine hold companions in-process/VRAM, so
they are unaffected.
The /delete-cached guard refused deleting a cached repo whose id is a bare
prefix of the loaded one: with Qwen/Qwen-Image-2512 loaded, deleting the
separate cached Qwen/Qwen-Image returned 400 "Unload the model before
deleting" because loaded_id.startswith(repo_id) matched across the repo
boundary. Both are real, independently cached catalog artifacts, so a
supported delete was wrongly blocked.
Add a /-boundary aware _loaded_id_matches_repo helper (mirroring
hub/services/models/deletion.py) and use it at all six guard sites (chat,
Images, Video, and their in-flight loading-id loops), so only the loaded
repo itself or a file within it blocks deletion. Add a regression test
covering the sibling-prefix pair.
- Chat load defers the CHAT arbiter handoff until after identifier / gpu_ids /
training-memory validation, so a doomed chat load (bad id, unsupported gpu_ids on
GGUF, or a training 409) no longer evicts a resident Images/Video pipeline and then
errors. The already-loaded fast paths re-assert CHAT ownership themselves. Mirrors
the image and video loaders, which validate before acquire_for.
- Both training-start GPU cleanups now unload a resident Video pipeline and release the
VIDEO arbiter owner, not just Images/DIFFUSION, so starting LLM or diffusion training
after a video generation session no longer competes with the still-resident video
model and OOMs the run.
- Example dataset import materializes into a private staging dir and promotes into the
dataset folder only after the whole import succeeds. A materialize that fails partway
no longer leaves a partial dataset that a retry would treat as complete (imported=0),
stranding the user with a truncated dataset. Hidden dirs are skipped by the dataset
scan so the staging dir never surfaces as a dataset.
- Cached/local Wan GGUFs are now classified name-aware: arch "wan" alone is ambiguous
between the loadable single-DiT TI2V-5B and the dual-expert A14B MoE the loader
refuses, so _arch_to_task falls back to the repo/file name (as the loader's own
detect_video_family does) and tags only a non-MoE match text-to-video, surfacing
loadable Wan GGUFs in the Video picker without surfacing unloadable A14B files.
- The diffusion dataset upload route is added to the MaxBodyMiddleware upload
passthrough, so its own get_upload_limit_bytes() cap (plus multipart overhead, and a
raised max_upload_size_mb) applies instead of the default body limit rejecting
near-limit batches with 413 before the handler runs.
Adds regression tests for each: the chat handoff not evicting on a doomed load, the
video unload on diffusion-training start, the atomic import leaving no partial dataset,
the name-aware Wan GGUF classification (TI2V-5B video, A14B unsupported, bare arch
unsupported), and the diffusion upload passthrough cap.
- _local_model_task now tags a local diffusers pipeline that resolves to a video
family (LTX / Wan / Hunyuan) as text-to-video, mirroring the cached-repo
_cached_repo_task, so supported local video pipelines surface in the Video
On-Device picker instead of being routed to the Images picker where the image
loader rejects them. Gated on _local_is_diffusers so only a real loadable
pipeline dir reaches the video check.
- Video validate_load_request now rejects a local pipeline pick whose directory has
no model_index.json before the GPU handoff, mirroring the image loader, so a bad
local pipeline can no longer evict the resident model and only then fail deep in
from_pretrained.
- The custom/env-mode uninstall now removes a sibling stable-diffusion.cpp only when
it carries the Studio owner marker. install_sd_cpp_prebuilt writes the canonical
.unsloth-studio-owned marker on install; uninstall.sh and uninstall.ps1 keep any
unowned checkout (a user's own git clone of stable-diffusion.cpp beside a custom
Studio root is no longer deleted). A pre-marker Studio build is left behind rather
than a user file removed.
Adds regression tests: local video pipeline tagged text-to-video (and a video-named
non-pipeline dir stays untagged so it can never trigger a doomed pipeline load), the
video local-pipeline preflight rejection, the install ownership marker, and the
uninstall keeping an unowned sibling while removing an owned one.
* Auto policies: deferred dense compile, video compile default, step cache and precision auto
Image dense loads with speed unset no longer sit at plain off: the load stays
bit-identical eager, and the 3rd generation in a session engages the default
compile profile plus the cuDNN attention upgrade mid-session (a one-off image
never pays the warmup, repeated use amortises it). Video dense loads resolve
straight to the default profile since a clip denoise amortises the compile
within a single run, and never to max.
Video also gains the image backend's tri-state auto policies: unset step cache
now decides from the default schedule and re-checks the actual step count per
generation, and unset precision (transformer_quant) hands the decision to the
hardware ladder instead of staying off. Memory badge reason now says plainly
that everything fits when no offload is planned.
* Rename Dtype to Precision, add the video Precision control, step cache Auto option
The images Advanced panel's Dtype row is now Precision (same control, clearer
name), and the video Advanced panel gains the matching Precision select wired
to the load route's existing transformer_quant field, gated to full-pipeline
loads the way the image control gates to GGUF. Step cache selects on both
pages gain an explicit Auto option as the default (the previous Off default
silently behaved as auto and never let anyone pin off), and the Speed and
Attention tooltips now state the deferred dense compile and the SageAttention
black-frame caveat.
* Model catalog: canonical diffusion model groups with device-aware routing
One canonical name per image/video model, its published artifacts (GGUF, FP8,
bnb-4bit, official BF16) as data, and pure routing helpers: suffix-stripped
canonical keys (owner-preserving; cross-owner merges only via explicit
aliases), group/artifact lookups, a flat back-compat options shim, load-spec
resolution replacing the pages' lookup tables, search matching over old ids
and format tokens, the GGUF fit ladder extracted from the variant expander,
and pickDefaultArtifact/pickDefaultQuant deciding what a bare group click
loads (downloaded first, then the best quality that fits 70 percent of VRAM,
GGUF as the safe fallback). Checked by npm run catalog:check, following the
i18n:check pattern.
* Picker: one canonical row per diffusion model with a format second level
The Images and Video pickers now render the curated catalog as one row per
model in Recommended: clicking loads the best artifact for the device (the
routed GGUF quant, a prequant FP8/bnb-4bit that fits, or the official BF16),
and a chevron opens the per-format list, with the GGUF row nesting the usual
quant expander. Live HF listing rows that belong to a group are deduplicated,
search collapses member repos into their group (old ids and format tokens
still match), and the On Device sections group cached member repos under the
same canonical name with the per-repo rows inside. Curated groups render from
the catalog rather than the HF listing, which finally surfaces LTX-2.3 in the
video Recommended list (its hub pipeline_tag is image-to-video, so the
text-to-video listing always missed it) and exposes the HunyuanVideo 720p
repack next to 480p.
Backend: /cached-models now tags trusted video-family repos text-to-video
instead of blanket text-to-image, and the pickers admit catalog-known
non-unsloth repos On Device, so cached Lightricks/Wan/Hunyuan pipelines
finally appear in the Video picker. Chat pickers pass no catalog and are
unchanged.
* Download formats, tab icons, plain-language train tips, 3-loop autoplay
The image Download button becomes a menu: PNG saves the original bytes with
the embedded recipe, JPEG and WebP re-encode client-side from the fetched
blob (JPEG flattened onto white). The video Download button gains MP4
(original, keeps audio), WebM and GIF; the latter two transcode server-side
from the stored MP4 via PyAV (VP9 realtime profile for WebM, ~12 fps adaptive
palette for GIF) behind a new gallery export route that 501s with a readable
message when a codec is missing.
Generated clips no longer loop forever: the player replays a clip three times
per selection, then pauses with controls up; a new generation or a refresh
gets its own three plays. The Create/Train tabs reuse the sidebar's New Chat
and Train icons (TestTubeOutlineIcon moved to a shared lib module), and every
Train tab helper text is now one plain sentence.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the Create/Train tab icon and label on one line
TabsTrigger renders its children inside a plain inline span and the
Tailwind preflight gives svg display:block, so the HugeiconsIcon forced
the label onto a second line. Wrap icon plus label in their own
inline flex row inside each trigger.
* Strip -int8 and -nvfp4 prequant suffixes in the model catalog key
canonicalKeyFor already lowercases before matching, so -GGUF/-FP8 in any
case were covered; -int8 and -nvfp4 were not in the suffix table, so
such repos rendered as standalone rows in Recommended and On Device
instead of standardizing into their base-name group and routing through
pickDefaultArtifact. Added both suffixes plus case-insensitivity and
routing assertions to the catalog check.
* Standardize non-catalog picker rows to their base model name
The curated catalog already collapses its own groups, but hub listing
rows and cached repos outside the catalog (ERNIE-Image, FLUX.2-klein,
Qwen-Image-Edit-2509, FLUX.2-dev) still rendered raw ids with -GGUF /
-FP8 style suffixes in Recommended and On Device.
- model-catalog.ts: new stripArtifactSuffixesForDisplay, a
case-preserving twin of canonicalKeyFor's stripping that keeps the
owner prefix and original casing for display.
- pickers.tsx: recommended hub rows and the downloaded GGUF/model rows
pass their labels through it when a catalog is present, so only the
diffusion pickers change; chat rows keep raw ids. Click targets keep
the full repo id, and the format badge still shows the artifact kind.
- Catalog check covers the new helper across GGUF/FP8/int8/nvfp4 in
both cases plus no-op and suffix-only names.
* Offer official BF16/FP8 artifacts per model group and fix gallery label clipping
Model picker changes so groups are not limited to unsloth quant repos:
- model-catalog.ts: each image group that has an official vendor pipeline
now carries its BF16 (official) artifact as the top (highest quality)
entry - Tongyi-MAI/Z-Image-Turbo, Qwen/Qwen-Image, Qwen/Qwen-Image-2512,
Qwen/Qwen-Image-Edit-2511, black-forest-labs/FLUX.1-dev, FLUX.1-schnell
and FLUX.1-Kontext-dev. The LTX-2.3 video group now lists Lightricks'
own bf16 and fp8 distilled single-file checkpoints alongside the GGUF.
Resident sizes are set from the actual weight totals (FLUX ships a
duplicate single-file that from_pretrained ignores, so FLUX bf16 is ~32
GB not 54). The repos that used to be aliases are now real artifacts.
- The router already prefers the highest-quality artifact that fits the
0.7 x GPU budget, so a datacenter GPU now defaults to official BF16
while consumer GPUs still route to the fitting quant or GGUF. That is
why bnb-4bit was the Z-Image-Turbo default before: it was the only
non-GGUF artifact and it was already downloaded.
- diffusion.py: allowlist the four official image repos not previously
trusted (qwen/qwen-image-2512, qwen/qwen-image-edit-2511,
black-forest-labs/flux.1-schnell, flux.1-kontext-dev). All verified as
safetensors-only diffusers model_index pipelines. The LTX-2.3
checkpoints are already on the video trust list.
- catalog check: BF16-wins-on-datacenter, quant-wins-on-consumer, and the
single-file load specs for the LTX-2.3 checkpoints.
Also fixes the video gallery thumbnail caption: the leading duration was
clipped by the rounded corner and selection border, so the strip now has
enough left/bottom padding to clear the curve.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* video gallery: guard export transcode against a stream-less clip
_transcode_webm and _transcode_gif indexed src.streams.video[0] before
checking the stream list, so a container with no video stream raised a bare
IndexError that the broad handlers then re-labeled as a missing libvpx or
decoder. Raise an explicit RuntimeError naming the real cause in both the
WebM and GIF paths.
* Studio: honor explicit attention/format choices, fix distilled-LTX defaults and On Device catalog routing
* Remove stray planning notes accidentally committed to the branch
* video: add transformerQuant to the load callback deps
handleLoad reads transformerQuant but omitted it from the useCallback dep array,
so after the user changes only Precision and then selects a model or clicks
Reapply, the memoized callback keeps the stale closure and loads the previous
precision. The image page's equivalent callback already lists it.
* model picker: honor the format filter when routing catalog clicks; add catalog rows to the roving list
- routedArtifactFor now scopes a group's artifacts to the active format filter
(the same matchesFormatFilter predicate the visibility check uses) before
pickDefaultArtifact, so a group shown only because it owns a GGUF no longer
routes a click to a large non-GGUF download. Covers both the Recommended and
On Device grouped paths.
- hubOptionKeys now includes the catalog-group, search-catalog-group, and grouped
On Device row keys in exact render order, so arrow/Home/End roving reaches the
catalog rows instead of giving them a duplicate missing id and skipping them.
* model picker: don't treat a partial base cache as downloaded
A partially-cached base repo (a cancelled download that left only some weights)
was counted as downloaded, so an On Device click routed to a fresh multi-GB
re-download instead of the complete GGUF. The picker's endpoint (/api/models/
cached-models) did not carry a partial flag at all, so a frontend-only guard
could not see it. Surface partial from that endpoint by reusing the hub inventory
scan's snapshot-partial detector, plumb it through CachedModelRepo (backend +
frontend types), and skip partial base repos when building the downloaded set.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* model picker + diffusion: drop partial/unloadable cached rows, skip defer-compile before a LoRA gen
- On Device (cached non-GGUF) rows filtered partial-download snapshots back in: sortedCachedModels
gated on passesTaskGate + a groupForRepoId key match but, unlike downloadedSet, never checked
c.partial, so an incomplete unsloth snapshot showed as a loadable On Device row (click errors or
silently re-fetches multi-GB). It also admitted repos that only match the catalog by group KEY
(a base / uncurated-quant sibling like Qwen/Qwen-Image-2512) which have no loadable artifact and
dead-end at the trust gate. Add !c.partial and gate on artifactForRepoId (what loadSpecFor
resolves) instead of groupForRepoId, so a cached row shows only when the backend can load it.
- Deferred speed-auto engaged the compile profile on the 3rd generation BEFORE _apply_loras. A
compiled transformer rejects LoRA (supports_lora is False) and _apply_loras raises before its
unchanged-selection no-op, so once compile engaged every LoRA generation on that load failed
permanently. Skip the deferral when a LoRA is requested (compile and LoRA are mutually exclusive)
and let it engage on a later LoRA-free generation.
* Scope the cached-model partial probe to the listed snapshot dir
list_cached_models builds each row from the largest/complete copy across HF cache
roots, but _cached_repo_partial probed is_snapshot_partial with no repo_cache_dir,
so the scan spanned every root: a stale .incomplete copy in one root would flag a
complete copy in another as partial and hide the usable model from the picker (the
click then routes to a re-download). Forward the winning snapshot's repo_path so all
three partial signals are scoped to that copy, matching the sibling inventory paths
(models/dataset cache_inventory, local_inventory).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Do not auto-route to gated repos, prefer complete cached copies, defer compile past attached LoRA, scope group expand keys
Four fixes:
- pickDefaultArtifact's not-downloaded ladder returned the gated BF16 FLUX.1-dev / Kontext-dev
before the open GGUF on a large GPU, so a bare group click routed to a repo the user may lack
license/token access to. Add a gated flag and skip gated artifacts in the not-downloaded ladder
(an already-downloaded gated artifact is still returned).
- list_cached_models picked the largest duplicate cache copy and computed partial only on it, so a
larger partial copy shadowed a smaller complete one; since partial rows are dropped from the
picker the usable model vanished. Prefer completeness, then size.
- the deferred-speed compile engaged on a no-LoRA generation while an adapter from a prior
generation was still attached, baking it into the compiled graph (the later unload is swallowed
on a compiled pipe); also defer while adapters remain attached.
- routeGroupClick's GGUF fallback toggled the context-free canonicalId while the chevron toggles
the context-scoped expandKey, leaving the format list un-collapsible in one context, dead in the
other, and risking cross-context expansion; thread expandKey through.
* Guard video pipeline repos from deletion, drop the always-failing LTX FP8 artifact, prefer 720p Hunyuan
Three round-6 fixes:
- cached non-GGUF video repos now surface in the Video On-Device picker with the normal delete
action, but /delete-cached only guarded chat + the Images engine, so a loaded/loading Wan / LTX /
Hunyuan pipeline could have its HF snapshot removed from under it. Add a VideoBackend
loading_repo_ids accessor and a video loaded/loading guard mirroring the Images one.
- the catalog advertised Lightricks/LTX-2.3-fp8 as loadable, but the LTX-2.3 loader refuses the
official scaled-FP8 single file (.weight_scale/.input_scale) and points to GGUF/BF16, so a pick
routed to a ~76 GB download that always fails on load. Remove the FP8 artifact.
- pickDefaultArtifact only sorts by format, so the HunyuanVideo group's 480p (listed first) beat
the 720p even on GPUs where 720p fits the budget. List 720p first so the fit loop prefers it and
falls back to 480p only on smaller cards.
* diffusion: add compute int8/fp8_dynamic text-encoder quant, wire into video
Add two torchao compute text-encoder quant modes to the diffusion precision
engine, alongside the existing layerwise fp8 and weight-only nvfp4:
- int8: per-token activation + per-channel weight (torch._int_mm), with per-layer
keep-bf16 selection. int8 degrades on large encoders unless the most
quant-sensitive decoder blocks stay bf16, so it engages only for families with
a measured keep-bf16 schedule (qwen-image / qwen-image-edit keep first+last 6,
flux.2-dev keeps first 3); a family without one falls back to fp8.
- fp8_dynamic: per-row fp8 compute (torch._scaled_mm), keeping the matmul in fp8
on the tensor cores instead of upcasting each forward like the layerwise fp8.
The selective int8 caster reuses the committed transformer-quant factory
(_make_quant_config / make_filter_fn / exclude_tokens_for_scheme) plus a small
structural first/last-N block skip, so it depends only on committed APIs.
Wire text-encoder quant into the video backend, which previously loaded the
companion encoder (Gemma3 / UMT5 / Qwen2.5-VL) dense bf16 while quantising only
the DiT. text_encoder_quant is plumbed through the load request, validation, the
load chain, the resolved record, and status, mirroring the image backend; it
applies for every load kind (the encoder is dense regardless of how the DiT was
sourced). Widen the image and video load request Literals and add the video
status field.
Tests: int8 family-schedule routing and fp8 fallback, fp8_dynamic routing,
hardware gates (int8 sm_80+, fp8_dynamic sm_89+), the structural block selection,
the real int8 filter closure (keeps the first blocks plus the vision tower /
lm_head / T5 wo dense), and the video route threading and 422 validation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* text-encoder quant: skip the torchao modes under offload (both backends)
quantize_text_encoders applied int8-with-schedule / fp8_dynamic / nvfp4 (all torchao) to the
text encoder regardless of the offload policy. An offload placement then moves the quantized encoder
with Module.to(), which torchao tensor subclasses reject (aten._has_compatible_shallow_copy_type is
unimplemented) -- a hard crash, the same one the DiT path already skips torchao quant under offload to
avoid. Add offload_active to quantize_text_encoders and skip the torchao modes when set; layerwise fp8
is not torchao and still streams under offload. Both the video and image loaders pass
offload_active = (offload policy != none).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* diffusion: skip non-bf16 linears for scaled_mm quant schemes
The fp8 / mxfp8 / nvfp4 schemes run on torch._scaled_mm and the fp4 / mx GEMMs,
which assert a bfloat16 input weight. On a mixed-precision DiT that keeps some
linears in fp32 for numerical stability (the Wan and Hunyuan video transformers
do this), quantize_ hits the first fp32 linear, raises, and the best-effort
wrapper swallows it to None, so the whole transformer stays dense with no error
and no speedup or memory saving.
Add a require_bf16 gate to make_filter_fn and pass it for the scaled_mm schemes
in quantize_transformer (and the fp8_dynamic text-encoder caster). The gate
skips non-bf16 linears so the scheme engages on the bf16 ones. int8 uses
torch._int_mm, which quantizes fp32/fp16 weights fine, so it leaves the gate off
and keeps its current coverage.
Verified on Wan2.2-TI2V-5B: fp8 and mxfp8 now quantize 303 linears via the
committed quantize_transformer path where they previously engaged 0.
* prequant builder: mirror the scaled-mm bf16 gate offline
The runtime DiT quantizer skips non-bf16 Linears for the scaled_mm schemes (fp8,
nvfp4, mxfp8) so the scheme engages on a mixed-precision transformer instead of
aborting on the first fp32 Linear. The offline prequant builder reused make_filter_fn
without that gate, so building an fp8/nvfp4/mxfp8 checkpoint for a mixed-precision DiT
(Wan, Hunyuan keep _keep_in_fp32_modules in fp32 even under torch_dtype=bf16) would hit
the same fp32 Linear and abort, breaking the builder's stated offline == runtime,
LPIPS-0 invariant. Thread require_bf16 = scheme in _SCALED_MM_SCHEMES through the builder,
record it in the checkpoint metadata, and verify it on load (mirrors the existing
exclude_name_tokens guard) so a future _SCALED_MM_SCHEMES change cannot silently load a
checkpoint built under the old filter.
* Keep nvfp4 fp32 linears quantised (bf16 gate is fp8/mxfp8 only)
Verified on torchao 0.17 / B200: fp8 per-row asserts 'PerRow quantization only
works for bfloat16 precision input weight' and mxfp8 asserts 'Only supporting bf16
out dtype', but NVFP4's high-precision conversion quantises an fp32 weight fine
(forward included). So the bf16 skip-gate must be fp8/mxfp8 only, not all scaled_mm
schemes -- otherwise nvfp4 leaves large fp32 projections dense, losing the intended
memory/speed gain. Rename _SCALED_MM_SCHEMES -> _REQUIRE_BF16_SCHEMES = (fp8, mxfp8)
and thread it through the runtime filter, the offline builder, and the loader
require_bf16 verification (offline == runtime preserved).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
_arch_to_task mapped every arch in _VIDEO_GGUF_ARCHS to text-to-video, so a cached wan GGUF was
surfaced in the Video picker even though no Wan VideoFamily is registered in this build, and
selecting it 400s on load. Gate the video task on detect_video_family actually resolving the
arch: an unregistered video arch (wan here) falls to the unsupported bucket (hidden from chat and
the Video/Images pickers) until its family lands, so the picker never offers a GGUF that cannot
load. Correct in both stacked video PRs (ltxv resolves; wan resolves only once its family exists).
Three video-tab fixes:
- delete-cached refused a loaded video repo but not one a background load is still downloading
(status().loaded is False in that window); add VideoBackend.loading_repo_ids() mirroring the
image backend and check it in the route so deleting mid-download can no longer yank blobs.
- the cached-gguf picker tags a cached HUB GGUF by its general.architecture, but the loader's
arch fallback only read a LOCAL file, so an opaquely-named cached hub LTX GGUF the picker
offered 400d on load; read the arch from the cached blob (network-free) too.
- on a mount with a model already loaded (refresh / load from another client) steps and guidance
stuck at the pre-load default, so a base checkpoint silently generated a degraded clip; seed
them from the backend-authoritative status.defaults once per newly-loaded model.
- delete_cached_model guarded chat (llama.cpp/transformers) and Images but not the
Video backend, so a loaded video GGUF (which shares the On-Device GGUF delete UI)
could be removed from under a live pipeline. Add the mirror guard on
get_video_backend().status().
- The Video picker admits a local GGUF by its general.architecture, but the loader
detected the family only from path/name tokens, so a renamed ltxv file (e.g.
model.gguf) was offered yet failed validate_load_request with a 400. _detect_load_family
now falls back to reading the arch (its string is a family alias) when name detection
misses, so the loader accepts exactly what the picker offered; an unsupported video arch
(wan) still resolves to None and 400s as before.
Tag the ltxv and wan GGUF archs text-to-video so cached video checkpoints
actually surface in the Video picker (they were classed unsupported and
hidden everywhere). Adopt the loaded family's default clip length instead
of silently keeping the 25-frame pre-load fallback, and derive steps and
guidance from the picked GGUF filename so a distilled variant gets its
few-step schedule. Suppress the error toast for the user's own Cancel and
disable the Video nav item on chat-only hosts with a hint, matching Train.
- Unload the ACTIVE image engine (sd_cpp or diffusers) before diffusion training starts, not just the diffusers singleton
- Count metadata.jsonl captions in dataset summaries so metadata-captioned datasets are not reported as uncaptioned
- Sidecar captions now override metadata rows everywhere (grid edits win); trainer and dataset API agree
- Tag local diffusers image checkpoints with text-to-image so they appear in the Images picker
- Family LoRA targets (_FLUX_TARGETS etc) apply when the config carries the generic defaults; explicit overrides still win
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict
Phase 1 of porting the richer diffusion stack onto the image-generation backend.
- Add a compartmentalized device/dtype policy module (diffusion_device.py)
resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA
capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or
fp32, never a silent fp16 that renders a black image.
- Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved
float16 to float32 for those families so they do not produce black images.
- Split the backend locks: a generation holds only _generate_lock, so status,
unload, and a new load are never blocked by a long denoise. Add per-generation
cancellation via callback_on_step_end so an eviction or a superseding load
preempts a running generation; a replacement load waits for it to stop before
allocating, so two pipelines never sit in VRAM at once.
- Validate a load request before the GPU handoff so an unloadable pick never
evicts a working chat model, and reject missing local paths up front.
- Add CPU-only tests for the device policy, dtype guard, lock split and
cancellation, and validate-before-evict, plus a GPU benchmark/regression
script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR
against a saved reference.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy
Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and
VAE tiling/slicing from measured free device memory vs the model's estimated
resident footprint, then applies it to the built pipeline. auto stays resident
when the model fits (byte-identical to the prior resident path), and falls to
whole-module offload when tight; fast/balanced/low_vram are explicit overrides.
Sequential submodule offload is unreliable for GGUF transformers on diffusers
0.38, so it falls back to whole-module offload and status reports the policy
actually engaged.
Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with
no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM
47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost.
73 prior + 35 new CPU tests pass.
* Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling
Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level,
use_stream) that keeps the transformer flowing through the GPU a few blocks at a
time while the text encoder / VAE stay resident, and fix VAE tiling to drive the
VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the
pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so
status never overstates either, and group falls back to whole-module offload when
the transformer can't be streamed.
Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group
cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 ->
2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names
now match that tradeoff: balanced = stream the transformer, low_vram = offload
every component. auto picks group when the companions fit resident, else model.
112 CPU tests pass.
* Studio diffusion (Phase 5): image quality-vs-quant accuracy harness
Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold
prompt + seed fixed, render a grid with a reference quant (default BF16), then render
each candidate quant and measure drift from the reference. Records mean PSNR + SSIM
(pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity
(transformers, --clip), plus file size, latency, and peak VRAM, then prints a
quality-vs-cost table and recommends the smallest quant within a quality budget.
--selftest validates the metrics on synthetic images with no GPU or model.
Verified on Z-Image (B200): the table degrades monotonically with quant size
(Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat
(~0.34) -- quantization erodes fine detail far more than prompt adherence.
* Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32)
Add a speed_mode knob (off by default, so the render path stays bit-identical):
default applies channels_last VAE + regional torch.compile of the denoiser's
repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional
compile is gated off for the GGUF transformer (dequantises per-op) and for families
flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image),
so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed
optims run before placement/offload, per the diffusers composition order. status now
reports speed_mode + the optims actually engaged.
Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last',
'tf32'], compile correctly skipped for GGUF; generation works in every mode.
121 CPU tests pass.
* Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting
Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3)
storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16
compute dtype while normalisations and embeddings stay full precision. Applied
before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder
dense). status reports which encoders were cast.
Verified on Z-Image (B200, balanced/group mode where the encoder stays resident):
generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload)
at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR
vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off
by default and documented as such, with the Phase 5 harness to size the cost.
127 CPU tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob)
Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant
(fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao
NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8
stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run
before placement; status reports the mode actually engaged. This is the lean
realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the
3045-line port.
Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the
bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE
option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and
both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default);
size it per model with the Phase 5 quality harness. diffusion_bench gains
--text-encoder-quant.
129 CPU tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac
Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the
chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm
/ XPU; this covers the hardware diffusers serves poorly, consuming the same
split GGUF assets Studio already curates.
- sd_cpp_args.py: pure sd-cli command builder. Maps the family to its
text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1
CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential)
to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu /
--vae-tiling / --diffusion-fa), so one user knob drives both engines.
- sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary()
with the same precedence as the llama finder (env override, then the Studio
install root, then in-tree, then PATH), an is_available/version probe, and a
one-shot subprocess generate that streams progress and returns the PNG.
runtime_env() prepends the binary's directory to the platform library path
so a prebuilt's bundled libstable-diffusion.so resolves.
select_diffusion_engine() is the pure routing decision (GPU backends to
diffusers, CPU/MPS to native when present).
- install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt
(macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the
Studio install root. resolve_release_asset() is a pure, unit-tested
host-to-asset matrix.
- scripts/sd_cpp_smoke.py: end-to-end native generation harness.
Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine,
routing, runtime env, and the installer resolver. Full diffusion suite 166
passing.
Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both
generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group
offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the
dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine
Builds on Phase 4's native stable-diffusion.cpp engine, extending it from
text-to-image to the wider feature surface, since sd.cpp supports all of these
through the binary already. Pure command-builder additions plus one engine
method, so the txt2img path is unchanged.
- sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img +
strength make a run img2img, adding mask makes it inpaint, ref_images drives
FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and
lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New
SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run
mode (input image + esrgan model, no prompt / text encoders).
- sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so
generate() (now carrying the conditioning flags) and a new upscale() reuse
the same streaming / error / output-check path.
- scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img /
--strength / --upscale-model / --upscale-repeats.
Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the
upscale builder and its validation, and the engine's img2img + upscale paths.
Full diffusion suite 176 passing.
Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the
init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale
(512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing
coherent images. Video and the diffusers-path feature wiring are deferred.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7): accuracy-preserving speed pass
Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy
bug and a dead-on-arrival speed path; this fixes both and adds the lossless /
near-lossless wins, all measured on a B200.
Correctness:
- TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32
process-wide and never restored them, so a later `off` load silently inherited
TF32 and was no longer bit-identical. Added snapshot_backend_flags /
restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer
runs and restored on unload. Verified: load max -> unload -> load off is now
byte-identical (PSNR inf) to a fresh off.
- sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and
only checked the timeout after EOF, so a child stuck in model load / GPU init
with no output ignored the timeout. Drained stdout on a reader thread with a
wall-clock deadline. Added a silent-hang regression test.
Speed (diffusers path), near-lossless, opt-in tiers:
- Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and
Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks
compiles and runs ~2.2x faster on the GGUF Z-Image transformer on
torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the
block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB
vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move
output quality. Gate relaxed; default tier delivers it.
- cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs).
- torch.inference_mode() around the pipeline call (lossless, strictly faster than
the no_grad diffusers uses internally).
Memory path:
- VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers;
the balanced (group) tier keeps exact slicing only, so it is now bit-identical to
the resident image (verified PSNR inf) and slightly faster.
- Group offload adds non_blocking + record_stream on the CUDA stream path to
overlap each block's H2D copy with compute (lossless; gated on the installed
diffusers signature so older versions still work).
Native (sd.cpp) path:
- native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a
near-lossless CUDA win that was previously only added on offload tiers; max also
-> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so
it is never auto-on. Engine generate() merges it, de-duped against offload flags.
Default profile: a GGUF model with no explicit speed_mode now resolves to the
`default` profile (resolve_speed_mode), since compile's perturbation sits below the
quantisation noise floor and so does not reduce quality versus the dense reference;
out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models
stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is
always honored, so the byte-identical path remains one flag away and is the
regression reference.
Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/
perf_verify.py (the B200 verification above), and diffusion_bench.py gains
--speed-mode so the speed tiers are benchmarkable.
Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore,
GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags +
the engine de-dup, and the sd-cli silent-hang timeout.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks
The opt-in `max` speed tier now compiles the repeated block with
mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode:
Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer
cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are
deliberately avoided -- both crash on the regionally-compiled block (its static
output buffer is overwritten across denoise steps), measured.
Adds two reproducible benchmarks used to validate the optimization research:
- scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head.
- scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes.
Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen;
coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune);
FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source)
Add an opt-in transformer_quant mode that loads the dense bf16 transformer and
torchao-quantises it onto the low-precision tensor cores, instead of the GGUF
transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a
B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's
0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor.
GGUF+compile stays the low-memory default and the fallback. The mode is gated on
CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB);
any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged
reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe
(Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips
the tiny projections that crash int8's torch._int_mm. New module mirrors
diffusion_precision.py; quant runs before compile before placement.
184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route
coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py
is the standalone torchao lever probe.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity
Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838
TFLOPS with FP16 accumulate; bf16 209), so:
- fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on;
pinning it guards consumer cards against a default change. On B200 it is identical
speed and slightly better quality (LPIPS 0.050 vs 0.091).
- the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate).
2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py):
2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the
cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile
(our main ~2x). Documented as a dead end, not shipped.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe
scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image
generation and reports max-abs + non-finite counts for use_fast_accum True vs False.
Confirms fast accumulation is an accumulation-precision knob, not an overflow one:
across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16),
0 non-finite elements and identical max-abs for both modes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override
Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want
fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and
prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match
on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40;
GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it.
Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s),
no overflow, quality below the quant noise floor. So the default leans to accuracy on
data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force)
lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off).
187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override
threading).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9
scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer.
Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with
use_triton_kernel=False (the default triton path needs the missing MSLK library), but
only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than
using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's
CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto
on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder
Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4
tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS,
2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the
diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both
slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166
vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is
correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add
scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 9): pre-quantized transformer loading
The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on
the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs
13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline
(scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton
on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True)
the quantized weights, so the dense bf16 never touches the GPU.
Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching
GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same
torchao config + min_features filter the runtime path uses, applied ahead of time.
New core/inference/diffusion_prequant.py (resolve_prequant_source +
load_prequantized_transformer, best-effort, lazy imports). diffusion.py
_load_dense_quant_pipeline tries the pre-quant source first and falls back to the
dense materialise+quantise path, then to GGUF, so the default is unchanged.
DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty
prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for
the resolver, the meta-init+assign loader, and the backend branch selection +
fallbacks; GPU verification via scripts/verify_prequant_backend.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 10): attention-backend selection
Add a selectable attention kernel via the diffusers set_attention_backend
dispatcher. Attention is memory-bandwidth bound, so a better kernel is an
end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV
matmuls torchao never touches) and composes with torch.compile.
auto picks the best exact backend for the device: cuDNN fused attention
(_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x
end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default
(below the compile/quant noise floor); native SDPA elsewhere and when speed=off
(so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/
xformers/aiter are honored, and an unavailable kernel falls back to the default
rather than failing the load.
New core/inference/diffusion_attention.py (normalize + per-device select + apply,
best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline;
attention_backend threads through begin_load / load_pipeline / status like the other
load knobs. New request field attention_backend + status field. Hermetic CPU tests
for normalize / select policy / apply fallback, plus route threading + 422. Measured
via scripts/perf_levers_probe.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder
Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core
throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public
benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm
is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada
consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the
arch tier to put int8 first on a consumer/workstation GPU (detected by the existing
_is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first.
Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression
on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation
(-> int8) and data-center Ada/Hopper/Blackwell (-> fp8).
* Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT
Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across
denoise steps a DiT's output settles, so once the first block's residual barely
changes the remaining blocks are skipped and their cached output reused. diffusers
ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the
standalone apply_first_block_cache hook as a fallback).
Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 ->
2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar.
OFF by default and a per-load opt-in: the win scales with step count, so it is for
many-step models (Flux / Qwen-Image) and pointless for few-step distilled models
(e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction
of the trajectory. It composes with regional compile only with fullgraph=False (the
cache's per-step decision is a torch.compiler.disable graph break), which the speed
layer now switches to automatically when a cache is engaged. Best-effort: a model
whose block signature the hook does not recognise is caught and the load proceeds
uncached.
- new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache
(enable_cache / apply_first_block_cache fallback; threshold auto-raised for a
quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import).
- diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph
when a cache is engaged.
- diffusion.py: apply_step_cache before compile; thread transformer_cache /
transformer_cache_threshold through begin_load -> load_pipeline and report the
engaged mode in status().
- models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and
transformer_cache_threshold request fields, engaged mode in the status response.
- hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold
selection, and best-effort failure handling, plus route threading + validation.
- scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM /
LPIPS vs the compiled no-cache baseline).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears)
The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with
'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant
goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN
modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2
*_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are
computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so
they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing
min_features filter did not exclude them.
Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a
modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed,
guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a
negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers
(M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and
quantises these layers fine, so the exclusion is int8-only. Sequence embedders
(context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note
'context_embedder' contains the substring 'text_embed', which is why the token is the
specific 'time_text_embed', not 'text_embed'.
Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every
supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's
FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s
(5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already
working) are unchanged.
- diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes
exclude_name_tokens; quantize_transformer passes it for int8 only.
- hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps
attention / FFN / sequence-embedder linears), while fp8 keeps them.
- scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's
Linear layers and derive the exclusion list.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears)
The prequant-checkpoint builder applied the dense quant filter without the int8-only
M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8
checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the
first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared
exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder
so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now
produces a working checkpoint on every supported model, giving int8 (the consumer-preferred
scheme) the same ~2x load-VRAM and download reduction fp8 already had.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine
When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the
native stable-diffusion.cpp engine instead of diffusers, with diffusers as the
guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM.
- diffusion_engine_router: centralised engine selection (built on the existing
select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason.
- sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface
backed by sd-cli, with lazy binary install, registry-driven asset fetch,
step-progress parsing, and cancellation.
- diffusion_families: per-family single-file VAE + text-encoder asset mapping.
- sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled).
- routes/inference + gpu_arbiter: drive the active engine via the router; the
API now reports the active engine and any fallback reason.
- tests for the backend, router, route selection, and cancellation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler
Address review feedback on #6724:
- engine router: unload the engine being deactivated on a switch, so the old
model is not left resident-but-unreachable (the evictor only targets the active
engine).
- generate route: sd.cpp execution errors (nonzero exit / timeout / missing
output) now map to 500, not 409 (which only means not-loaded / cancelled).
- native batch: return per-image seeds and persist the actual seed for each image
so every batch image is reproducible.
- Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the
stable-diffusion.cpp docs; other families keep sd-cli defaults.
- honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload
offload flags on the native load instead of hardcoding them off.
- fail the load when the sd-cli binary is present but not runnable (version()
now returns None on exec error / nonzero exit).
- size estimate: only treat the transformer asset as a possible local path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in
load_prequantized_transformer ends in torch.load(weights_only=False), which executes
arbitrary code from the pickle. The transformer_prequant_path load-request field reached
that unpickle for any local file an authenticated caller named, so a request could trigger
remote code execution. Refuse the source.kind=='path' branch unless the operator sets
UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted
and unaffected. Document the requirement on the API field and add gate tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa
- apply_attention_backend now restores the native default when no backend is requested or a
kernel fails. diffusers keeps a process-wide active attention backend that
set_attention_backend updates, and a fresh transformer's processors follow it, so a load
that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile
load pinned, breaking the bit-identical/off guarantee.
- select_attention_backend drops flash3/flash4 up front when the CUDA capability is below
Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit
request on the wrong card set fine then crashed mid-generation; it now falls back to native.
- Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a
valid alias of native) is accepted instead of 422-rejected by Pydantic.
- Drop the dead replace('-','_') normalization (no alias uses dashes/underscores).
- perf_levers_probe.py output dir is now relative to the script, not a hardcoded path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF
- apply_step_cache now engages only via the transformer's native enable_cache (the diffusers
CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a
cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin
transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported
transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model
now runs uncached per the best-effort contract.
- GGUF transformers are quantized (the default Studio load path), so they now use the higher
quantized FBCache threshold when the caller leaves it unset, instead of the dense default
that could keep the cache from triggering.
- fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so
fullgraph=True failed warmup and silently measured an eager cached run); output dir is now
relative to the script, not a hardcoded path.
* Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder
_is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as
consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead
of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already
classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the
same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate.
Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0).
* Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config
Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just
TypeError on older torchao), which would break the mxfp8 config helper instead of falling
back to the default. Catch both so the fallback is robust.
quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds
CUDA memory during the per-row VRAM probe; output dir relative to the script.
* Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load
- snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform
missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the
whole snapshot. restore_backend_flags restores each flag independently so one failure can't
leave the others leaked process-wide.
- load_pipeline restores the flags (and clears the GPU cache) when the build fails after
apply_speed_optims mutated the process-wide flags but before _state captured them for unload
to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and
contaminated later off generations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output
Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs
without producing output (or without closing stdout) would never reach proc.wait and the
wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the
PROCESS, so the main thread always enforces the timeout and kills a hung process (which
closes the pipe and ends the reader). Add a test that times out even when stdout blocks,
and make the no-binary test hermetic so a host-installed sd-cli can't leak in.
* Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn
The filter callback can be invoked without a module name, so fqn.lower() would raise
AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens,
so the linear is kept) instead of crashing the quantise pass.
* Studio diffusion (Phase 16) review fixes: native engine robustness
- sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64);
a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds
aliased to the same image. Random seeds stay 53-bit (JS-safe).
- sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub
fall back to anonymous instead of failing auth on a blank token.
- sd_cpp_backend: a superseding load now cancels the in-flight generation, so the
old sd-cli can no longer return/persist an image from the previous model.
- diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a
slow 10+ GB free / CUDA sync does not block engine selection.
- diffusion_engine_router: probe sd-cli runnability (version()) before committing to
native, so a present-but-unrunnable binary falls back to diffusers at selection.
- diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a
CPU-only install can still reach the native sd.cpp engine instead of failing load.
- tests updated for the runnability probe + a not-runnable fallback case.
* Studio diffusion (Phase 9) review fixes: prequant safety + validation
- SECURITY: a request-supplied local pre-quant path is now unpickled only when it
resolves inside an operator-configured ALLOWLIST of directories
(UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in,
once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on
any path a load request named (arbitrary code execution). realpath() blocks symlink
escapes; a bare on/off toggle is no longer a wildcard.
- Validate the checkpoint's min_features against the runtime Linear filter, so a
checkpoint that quantised a different layer set is rejected instead of silently
loading a model that mismatches the dense path while reporting the same scheme.
- Tolerant base_model_id compare (exact or same final path/repo segment), so a local
path or fork of the canonical base is accepted instead of falling back to dense.
- _has_meta_tensors uses any(chain(...)) (no intermediate lists).
- prequant verify/probe scripts use repo-relative paths (+ env overrides), not the
author's absolute /mnt paths.
- tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts
- diffusion_memory: when group offload is unavailable and the plan falls back to
whole-module offload, enable VAE tiling (the group plan left it off, but the fallback
is the low-VRAM path where the decode spike can OOM). Covers both the group and
sequential fallback branches.
- perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a
balanced bit-identity regression actually fails the check instead of exiting 0.
- compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and
the load-progress poll has a 30 min deadline instead of looping forever on a hang.
- test for the group->model fallback enabling VAE tiling.
* Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path
- diffusion: a torchao-quantized transformer is committed only compiled. A dense model
resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF
it replaced), so when transformer_quant engaged and speed resolved to off, promote to
default (regional compile); warn loudly if compile still does not engage.
- diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the
CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK);
otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF.
- nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path).
- test asserts the eager-quant -> default-compile promotion.
* Studio diffusion (Phase 10) review fixes: attention gating + probe isolation
- diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere
NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation,
so auto now stays on native SDPA there.
- diffusion_attention: _active_attention_backend handles get_active_backend() returning an
enum/None (not a tuple); the old unpack always raised and was swallowed, so
the native-restore short-circuit never fired.
- perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS
on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul
so the inductor_flags variant doesn't leak into later compiled rows.
- tests for the SM80 cuDNN gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening
- install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit
timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket);
extract through a per-member containment check (Zip-Slip guard); expanduser the
--install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch
the separately-published cudart runtime DLL archive so sd-cli.exe can start.
- sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the
installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start
sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend
crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie.
- tests: Zip-Slip rejection, normal extraction, studio-home discovery.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs
Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes
the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of
the literal --output path. SdCppEngine.generate checked only the literal path, so
a batch generation would exit 0 and then raise 'no image' (or return a stale
file). generate now returns the literal path when present and otherwise falls
back to the numbered siblings; single-image behavior is unchanged.
Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected
without error.
* Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats
Codex review on the native engine arg builder:
- build_sd_cpp_command emitted --width/--height unconditionally, so an
img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of
the input. width/height are now Optional (None = unset): an image-conditioned
run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives
the size from the input image (set_width_and_height_if_unset); a plain txt2img
run with unset dims keeps the prior 1024x1024 default; explicit dims are always
honored. width/height are read only by the builder, so the type change is local.
- build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...)
that silently swallowed repeats=0 into sd-cli's default of one pass, turning an
explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError
and emits the flag for any explicit value != 1.
Tests: img2img unset dims omit width/height (init_img and ref_images), explicit
dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at
the default. (Two pre-existing binary-discovery tests fail only because a real
sd-cli is installed in this dev environment; unrelated to this change.)
* Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc
Codex review: the transformer_prequant_path field description still told operators
to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the
prior security fix made that variable a directory allowlist -- _allowed_prequant_roots
deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator
following the documented =1 would have every transformer_prequant_path request
silently refused. The description now states it must name one or more allowlisted
directories and that a bare on/off value is not accepted.
Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does
not say =1, and describes an allowlist/directory (guards against doc drift).
* Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset
Codex review on attention-backend selection:
- Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on
pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first
generation with no fallback. select_attention_backend now applies
_cudnn_attention_supported() to an explicit cuDNN request too.
- flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a
Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3
is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a
(min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+.
- apply_attention_backend's success path left diffusers' process-wide active
backend pinned to the kernel it set; a later component whose processors are
unconfigured (backend None) would inherit it. It now resets the global registry
to native after a successful per-transformer set (the transformer keeps its own
backend), best-effort. Also fixed _active_attention_backend: get_active_backend()
returns a (name, fn) tuple, so the prior code stringified the tuple and never
matched a name, defeating the native-restore short-circuit.
Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on
SM90; global registry reset after a successful set; _active_attention_backend
reads the tuple return.
* Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder
Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300
(Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So
_is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the
auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting
llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they
are treated as data-center class and keep the intended fp8-first behavior.
Test: extends the datacenter parametrize with 'NVIDIA B300' and
'NVIDIA GH200 480GB' (now _is_consumer_gpu False).
* Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder
Codex review: the M=1 modulation/embedder exclusion was wired only into the dense
runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called
make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint
quantised the AdaLN modulation and conditioning-embedder linears, and loading it
via transformer_prequant_path (the load path only loads already-quantised tensors,
it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes
for the runtime path.
Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 ->
the M=1 exclusion, every other scheme -> none) and use it in both the runtime
quantiser and the builder, so a prequant artifact's quantised-layer set always
matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion).
Test: int8_exclude_name_tokens returns the exclusion for int8 and () for
fp8/nvfp4/mxfp8.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race
Codex review on the native-engine routing:
- The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat)
unconditionally after engine selection. A native sd.cpp load on a pure-CPU host
never touches the GPU, so that needlessly tore down the resident chat model. The
handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a
CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it.
- sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when
_run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so
the setting was unverifiable. status now derives them from state.offload_flags
(still 'none' on CPU, where the flags are empty).
- _run_load committed the new state without cancelling/waiting on a generation that
started during the (slow) asset download, so a stale sd-cli run against the OLD
model could finish afterward and persist an image from the previous model once the
new load reported ready. The commit now signals the in-flight cancel and waits on
_generate_lock before swapping _state (taken only at commit, so the download never
serialises against generation), mirroring the diffusers load path.
Tests: CPU native load skips the arbiter while a GPU native load takes it; status
reports offload active when flags are set; _run_load cancels and waits for an
in-flight generation before committing.
* Studio diffusion (Phase 14) review round 2: align helper name with the stack
Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the
identical helper already present higher in the diffusion stack (Phase 16). The
helper definition, the runtime quantiser call, and the offline builder are now
byte-identical to that version, so the two branches no longer introduce a
divergent name for the same single-source-of-truth and the stack merges without
a conflict on this fix. No behavior change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion: eager patches + torch.compile cache speed phase
Adds the opt-in speed path for the GGUF diffusion transformer behind a
selectable speed mode (default off, so output is unchanged until a profile
is chosen):
- diffusion_eager_patches.py: shared eager fast-paths (channels_last,
attention/backend selection, fused norms and QKV) installed at load and
rolled back on unload or failed load.
- diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent
torch.compile cache and the GGUF-transformer compile wiring.
- diffusion_arch_patches.py: architecture-specific patches.
- diffusion_patch_backend.py: shared install/restore plumbing.
- diffusion_speed.py: speed-profile planning.
Tests for each module plus the benchmarking and probe scripts used to
measure speed, memory, and accuracy of the path.
* Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI
Backend:
- Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file
fp8 transformers, gated to the unsloth org plus a curated allowlist.
- Image-conditioned workflows built with Pipeline.from_pipe so they reuse the
loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint,
outpaint, and a hires-fix upscale pass.
- Instruction editing as its own family kind (Qwen-Image-Edit-2511,
FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and
multi-reference) plus klein inpaint.
- Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched
mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the
decoded image size and cap upscale output to avoid OOM on large inputs.
- Fixes: from_pipe defaulting to a float32 recast that crashed torchao
quantized transformers; image-conditioned calls forcing the slider size
onto the input image. Native sd.cpp engine rejects image-conditioned and
reference requests it cannot serve.
Frontend:
- Redesigned Images page with capability-gated workflow tabs (Create,
Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor,
client-side outpaint, and a multi-reference picker.
- Advanced options moved to a right-docked panel mirroring Chat: closed by
default, toggled by a single fixed top-bar button that stays in place.
sd.cpp installer: pin the release, verify each download's sha256, add a
download timeout, and make the source repo configurable for a future mirror.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: correct the Advanced panel comment (closed by default, fixed toggle)
* Studio diffusion: LoRA adapters for the Images workflow
Add community LoRA support across both diffusion backends, the single
biggest step toward broad image-workflow coverage.
Backend
- New shared module core/inference/diffusion_lora.py: adapter discovery
(local scan + curated catalog + owner/name[:file] Hub refs), download
via hf_hub_download_with_xet_fallback, alias sanitization, native
managed-dir materialization with collision-broken aliases, prompt-tag
injection (deduped against user-typed tags), and a supports_lora gate.
- Native sd-cli: resolve + materialize selected LoRAs into a per-run
managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with
--lora-apply-mode auto. The arg builder already emitted these flags.
- Diffusers: non-fused load_lora_weights + set_adapters manager, tracked
on the pipe so an unchanged selection is a no-op and a model swap
resets; cleared on unload. Never fuses (breaks quantized transformers
and blocks live weight tweaks).
- Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers,
and native Qwen-Image (no LoRA name-conversion branch upstream).
- Request contract: optional loras on DiffusionGenerateRequest; empty or
omitted is identical to today. supports_lora surfaced in status; chosen
LoRAs persisted in gallery recipe metadata.
- New GET /api/models/diffusion-loras for the picker (family-filtered).
Frontend
- Repeatable multi-LoRA picker (adapter select + weight slider 0..2 +
remove), gated by the loaded model's supports_lora and family, max 8.
Tests
- New test_diffusion_lora.py (14): helpers, request validation, native
tag/dir wiring, diffusers set_adapters manager, supports_lora matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion: ControlNet for the Images workflow (diffusers)
Add ControlNet conditioning, the #2 most-used diffusion workflow after
LoRA, on the diffusers backend for the families with ControlNet pipelines
(FLUX.1 and Qwen-Image), with Union models as the default picks.
Backend
- New core/inference/diffusion_controlnet.py: family-gated discovery
(curated Union models + local dirs + bare owner/name repos), resolution
to a loadable repo/dir, control-image preprocessing (passthrough +
a dependency-free canny edge map), and a supports_controlnet gate.
- diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the
(small) ControlNet model once via from_pretrained (cached by id) and
builds the family's ControlNet pipeline via Pipeline.from_pipe(base,
controlnet=model), reusing the resident base modules at their loaded
dtype (no reload, no recast). Passes the control image + conditioning
scale + guidance start/end at generate time; cleared on unload.
- Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image ->
QwenImageControlNetPipeline/Model. Others declare none (gated off).
- Gated off for the native engine, GGUF-via-diffusers, and torchao
fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only.
- Request contract: optional controlnet on DiffusionGenerateRequest;
supports_controlnet in status; the choice persisted in gallery meta.
- New GET /api/models/diffusion-controlnets for the picker.
Frontend
- A ControlNet control in the Images rail (model select + control-image
upload + control-type select + strength slider), gated by the loaded
model's supports_controlnet + family, shown for text-to-image.
Tests
- New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate
helpers, request validation, family wiring, and the diffusers pipe
manager (loads once, caches, from_pipe with controlnet, rejects
unsupported families).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img')
The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio.
* Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander
Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a
stray "gguf" tag on the Hub but ship no .gguf files. The model search classified
them as GGUF from the bare tag, so the picker rendered the GGUF variant expander,
which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when
the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata
(populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected.
* Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op
handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other
non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo
surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline
load when the id is unsloth-hosted or on-device (the backend infers the family + base repo
and gates loads to unsloth/* or local paths), and show a clear message otherwise instead
of silently ignoring the click. Curated and GGUF paths are unchanged.
* Studio Images: keep curated safetensors models in Recommended after download
The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's
Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet),
so they vanished from the picker after the first load and could only be found by typing an
exact search. The row already renders a downloaded badge, matching how GGUF Recommended
rows stay visible when cached. Drop the exclusion so the curated safetensors always list.
* Studio diffusion LoRA: sanitize dots out of adapter aliases
The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names
containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose
filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to
apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename
stem. Adds regression coverage for internal dots.
* Studio Images: clarify the GGUF transformer-quant Advanced control
Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode"
with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/
FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF,
falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only.
* Studio Images: list on-device unsloth diffusion models in the picker
The Images picker's On Device tab hid every non-GGUF cached repo whenever a
task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit
and FP8 safetensors) never showed up there. List cached repos that pass the
task gate, limited under a filter to unsloth-hosted ones so base repos (which
fail the diffusion load trust gate) don't appear only to dead-end on click.
Chat behavior is unchanged: the task gate still drops image repos there.
* Studio: hide single-file image checkpoints from the chat model picker
The chat picker treats a cached repo as an image model, and hides it, only
when it ships a diffusers model_index.json. Single-file, ComfyUI, and
ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a
Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models.
Fall back to resolving the repo id against the known diffusion families, the
same resolver the Images backend loads from, so these checkpoints are tagged
text-to-image and stay in the Images picker only.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: add the FLUX.2-dev model family
Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the
Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline)
resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline
+ Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated,
reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text
encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror.
text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for
dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from
klein's turbo defaults. Verified live: GGUF load resolves the family + gated base
repo and generates a real 1024x1024 image on GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: clearer error for an unsupported diffusion model
When a repo id resolves to no diffusion family the load raised 'Could not infer a
diffusion family... Pass family_override (z-image)', which points at an unrelated
family and doesn't say what is supported. Replace it with a message that lists the
supported families (from a new supported_family_names helper) and notes that video
models and image models whose diffusers transformer has no single-file loader are
not supported. Applies to both the diffusers and native sd.cpp load paths. Also
refreshes two stale family-registry comments that still called FLUX.2-dev omitted.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove stray async task scratch outputs committed by mistake
* ControlNet: reject filesystem-like ids and do not cache a model past an unload race
Two review findings on the ControlNet path:
- resolve_controlnet's bare-repo fallback accepted any id with a slash, so a
path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory.
Restrict the fallback to a strict owner/name HF repo id shape.
- _controlnet_pipe now re-checks the cancel event after the blocking
from_pretrained: an unload that raced the download had already cleared the
caches, so caching the late module would pin it past the unload.
* ControlNet: address review findings on the diffusers path
- resolve_controlnet enforces catalog family compatibility so a direct API call
cannot load a ControlNet built for another family through the wrong pipeline.
- Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError
to ValueError) instead of a generic 500.
- strength 0 disables ControlNet entirely, so a no-op selection never pays the
download / VRAM cost; the control image is decoded and validated BEFORE the
ControlNet is resolved or built, so a malformed image fails fast for the same reason.
- ControlNet loads use the base compute dtype (state.dtype is a display string,
not a torch.dtype, so it silently fell back to float32) and honor the base
offload policy via group offloading instead of forcing the module resident.
- Empty/malformed HF token coerced to anonymous access.
- Flux Union ControlNet control_mode mapped from the selected control type.
- resolve_controlnet drops the unused hf_token/cancel_event params.
- ControlNetSpec validates guidance_start <= guidance_end (clean 422).
- Images UI ControlNet Select shows its placeholder when nothing is selected.
Adds regression tests for family enforcement and the union control-mode map.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown
Address review findings on the LoRA path:
- resolve_one: normalise a blank/whitespace hf_token to None (anonymous access)
and reject a client-supplied weight file with traversal / absolute path.
- resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError
so the route returns 400 instead of a generic 500.
- _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs
foo.gguf) so each is uniquely addressable.
- inject_prompt_tags: the backend-validated weight now wins over a user-typed
<lora:ALIAS:...> for a selected adapter; unselected user tags are left alone.
- diffusers _apply_loras: reject a .gguf adapter with a clear error before touching
the pipe (diffusers loads safetensors only).
- _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is
dropped wholesale (freeing adapters), so the previous call could race an in-flight
denoise on the same pipe.
- Images page: use a stable LoRA key and clear the selection (not just the options)
when the catalog refresh fails.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Diffusion: guard trust check against OSError and validate conditioning inputs
- _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid
characters (or a bare owner/name id) can't raise OSError; treat any failure as
not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request
still raises the clear FileNotFoundError for a genuinely missing local pick.
- generate(): reject mask_image / upscale / reference_images supplied without an
input image, and reject reference_images on a family that does not support
reference conditioning, instead of silently degrading to txt2img / img2img.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Codex review findings on the image-workflows PR
Keep diffusion.py importable without torch: the compile/arch patch modules
import torch at module level, so import them lazily at their load/unload
call sites instead of at module load. This restores the torchless contract
so get_diffusion_backend() works on a CPU/native sd.cpp install.
Match family reject keywords and aliases as whole path/name segments, not
raw substrings, so an unrelated word like edited, edition, or kontextual no
longer misroutes or hides a valid base image model, while supported edit
families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same
segment matching in the picker task filter.
Route FLUX.2-dev native guidance through --guidance like the other FLUX
families rather than --cfg-scale. Reject native upscale requests that have
no input image. Read image header dimensions and reject over-limit inputs
before decoding pixels, so a crafted small-payload image cannot spike
memory. Reject an upscale that would shrink the source below its input
size. Validate the model_kind against the filename extension before the
GPU handoff. Estimate a local diffusers pipeline's size from its on-disk
weights so auto memory planning does not skip offload and OOM. Report
workflows: [txt2img] from the native backend status so the Create tab
stays enabled for a loaded native model. Clamp the outpaint canvas to the
backend's 4096px decode limit.
Adds regression tests for segment matching and kind/extension validation.
* Harden diffusion LoRA handling on the diffusers and native paths
Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max):
diffusers requires the adapter loaded before compilation, so applying one to
the already-compiled module fails with adapter-key mismatches. The status
gate now hides the picker and generate raises a clear message instead.
Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the
diffusion cancellation sentinel in resolve_specs, so an unload/superseding
load during resolution maps to a 409 instead of a generic server error.
Drop weight-0 LoRA rows before the native support gate so a request carrying
only disabled adapters stays a no-op on families where native LoRA is
unsupported, matching the diffusers path.
Reject duplicate LoRA ids in the request model: both apply paths suffix
colliding names, so a repeated id would stack the same adapter past its
per-adapter weight bound.
Strip all user-typed <lora:...> prompt tags on the native path (only the
selected adapters are materialized in the managed lora-model-dir, so an
unselected tag can never resolve), and restore saved LoRA selections from a
gallery recipe so restore reproduces a LoRA image.
* Harden ControlNet resolve, gallery metadata, and the control-type picker
Check cancellation immediately after a ControlNet from_pretrained and before
any device placement, so an unload/eviction that raced the download does not
allocate several GB onto the GPU after the load was already cleared.
Require a loadable weight or shard index (not just config.json) before a local
ControlNet folder is advertised, so an interrupted copy is hidden instead of
failing deep in from_pretrained as a generic 500.
Do not record a strength-0 ControlNet in the gallery recipe: it is treated as
disabled and skipped, so the image is unconditioned and the metadata must not
claim a ControlNet was applied.
Build the control-type picker from the selected ControlNet's advertised
control_types instead of a hardcoded passthrough/canny pair, so a union model
with a precomputed depth or pose map sends the correct control_mode.
* Address further Codex findings on the image-workflows PR
- Persist the actual output image size in the gallery recipe instead of the
request sliders: Transform/Inpaint/Edit derive the size from the uploaded
image, Extend grows the canvas, and Upscale resizes it, so the sliders
recorded (and later restored) the wrong dimensions for those workflows.
- Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file
name) in validate_load_request, so the unloadable pick fails before chat is
evicted rather than deep in from_pretrained.
- Only publish an image-conditioned from_pipe wrapper to the shared aux cache
when the load is still current: from_pipe runs under the generate lock but
not the state lock, so an unload racing its construction could otherwise
cache a wrapper over torn-down modules that a later load would reuse.
- Verify the Windows CUDA runtime archive checksum before extracting it, like
the main sd-cli archive, so a corrupt or tampered runtime is rejected rather
than extracted next to the binary.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict
Phase 1 of porting the richer diffusion stack onto the image-generation backend.
- Add a compartmentalized device/dtype policy module (diffusion_device.py)
resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA
capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or
fp32, never a silent fp16 that renders a black image.
- Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved
float16 to float32 for those families so they do not produce black images.
- Split the backend locks: a generation holds only _generate_lock, so status,
unload, and a new load are never blocked by a long denoise. Add per-generation
cancellation via callback_on_step_end so an eviction or a superseding load
preempts a running generation; a replacement load waits for it to stop before
allocating, so two pipelines never sit in VRAM at once.
- Validate a load request before the GPU handoff so an unloadable pick never
evicts a working chat model, and reject missing local paths up front.
- Add CPU-only tests for the device policy, dtype guard, lock split and
cancellation, and validate-before-evict, plus a GPU benchmark/regression
script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR
against a saved reference.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy
Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and
VAE tiling/slicing from measured free device memory vs the model's estimated
resident footprint, then applies it to the built pipeline. auto stays resident
when the model fits (byte-identical to the prior resident path), and falls to
whole-module offload when tight; fast/balanced/low_vram are explicit overrides.
Sequential submodule offload is unreliable for GGUF transformers on diffusers
0.38, so it falls back to whole-module offload and status reports the policy
actually engaged.
Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with
no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM
47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost.
73 prior + 35 new CPU tests pass.
* Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling
Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level,
use_stream) that keeps the transformer flowing through the GPU a few blocks at a
time while the text encoder / VAE stay resident, and fix VAE tiling to drive the
VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the
pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so
status never overstates either, and group falls back to whole-module offload when
the transformer can't be streamed.
Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group
cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 ->
2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names
now match that tradeoff: balanced = stream the transformer, low_vram = offload
every component. auto picks group when the companions fit resident, else model.
112 CPU tests pass.
* Studio diffusion (Phase 5): image quality-vs-quant accuracy harness
Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold
prompt + seed fixed, render a grid with a reference quant (default BF16), then render
each candidate quant and measure drift from the reference. Records mean PSNR + SSIM
(pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity
(transformers, --clip), plus file size, latency, and peak VRAM, then prints a
quality-vs-cost table and recommends the smallest quant within a quality budget.
--selftest validates the metrics on synthetic images with no GPU or model.
Verified on Z-Image (B200): the table degrades monotonically with quant size
(Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat
(~0.34) -- quantization erodes fine detail far more than prompt adherence.
* Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32)
Add a speed_mode knob (off by default, so the render path stays bit-identical):
default applies channels_last VAE + regional torch.compile of the denoiser's
repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional
compile is gated off for the GGUF transformer (dequantises per-op) and for families
flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image),
so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed
optims run before placement/offload, per the diffusers composition order. status now
reports speed_mode + the optims actually engaged.
Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last',
'tf32'], compile correctly skipped for GGUF; generation works in every mode.
121 CPU tests pass.
* Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting
Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3)
storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16
compute dtype while normalisations and embeddings stay full precision. Applied
before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder
dense). status reports which encoders were cast.
Verified on Z-Image (B200, balanced/group mode where the encoder stays resident):
generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload)
at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR
vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off
by default and documented as such, with the Phase 5 harness to size the cost.
127 CPU tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob)
Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant
(fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao
NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8
stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run
before placement; status reports the mode actually engaged. This is the lean
realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the
3045-line port.
Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the
bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE
option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and
both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default);
size it per model with the Phase 5 quality harness. diffusion_bench gains
--text-encoder-quant.
129 CPU tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac
Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the
chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm
/ XPU; this covers the hardware diffusers serves poorly, consuming the same
split GGUF assets Studio already curates.
- sd_cpp_args.py: pure sd-cli command builder. Maps the family to its
text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1
CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential)
to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu /
--vae-tiling / --diffusion-fa), so one user knob drives both engines.
- sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary()
with the same precedence as the llama finder (env override, then the Studio
install root, then in-tree, then PATH), an is_available/version probe, and a
one-shot subprocess generate that streams progress and returns the PNG.
runtime_env() prepends the binary's directory to the platform library path
so a prebuilt's bundled libstable-diffusion.so resolves.
select_diffusion_engine() is the pure routing decision (GPU backends to
diffusers, CPU/MPS to native when present).
- install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt
(macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the
Studio install root. resolve_release_asset() is a pure, unit-tested
host-to-asset matrix.
- scripts/sd_cpp_smoke.py: end-to-end native generation harness.
Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine,
routing, runtime env, and the installer resolver. Full diffusion suite 166
passing.
Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both
generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group
offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the
dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine
Builds on Phase 4's native stable-diffusion.cpp engine, extending it from
text-to-image to the wider feature surface, since sd.cpp supports all of these
through the binary already. Pure command-builder additions plus one engine
method, so the txt2img path is unchanged.
- sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img +
strength make a run img2img, adding mask makes it inpaint, ref_images drives
FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and
lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New
SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run
mode (input image + esrgan model, no prompt / text encoders).
- sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so
generate() (now carrying the conditioning flags) and a new upscale() reuse
the same streaming / error / output-check path.
- scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img /
--strength / --upscale-model / --upscale-repeats.
Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the
upscale builder and its validation, and the engine's img2img + upscale paths.
Full diffusion suite 176 passing.
Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the
init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale
(512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing
coherent images. Video and the diffusers-path feature wiring are deferred.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7): accuracy-preserving speed pass
Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy
bug and a dead-on-arrival speed path; this fixes both and adds the lossless /
near-lossless wins, all measured on a B200.
Correctness:
- TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32
process-wide and never restored them, so a later `off` load silently inherited
TF32 and was no longer bit-identical. Added snapshot_backend_flags /
restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer
runs and restored on unload. Verified: load max -> unload -> load off is now
byte-identical (PSNR inf) to a fresh off.
- sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and
only checked the timeout after EOF, so a child stuck in model load / GPU init
with no output ignored the timeout. Drained stdout on a reader thread with a
wall-clock deadline. Added a silent-hang regression test.
Speed (diffusers path), near-lossless, opt-in tiers:
- Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and
Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks
compiles and runs ~2.2x faster on the GGUF Z-Image transformer on
torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the
block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB
vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move
output quality. Gate relaxed; default tier delivers it.
- cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs).
- torch.inference_mode() around the pipeline call (lossless, strictly faster than
the no_grad diffusers uses internally).
Memory path:
- VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers;
the balanced (group) tier keeps exact slicing only, so it is now bit-identical to
the resident image (verified PSNR inf) and slightly faster.
- Group offload adds non_blocking + record_stream on the CUDA stream path to
overlap each block's H2D copy with compute (lossless; gated on the installed
diffusers signature so older versions still work).
Native (sd.cpp) path:
- native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a
near-lossless CUDA win that was previously only added on offload tiers; max also
-> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so
it is never auto-on. Engine generate() merges it, de-duped against offload flags.
Default profile: a GGUF model with no explicit speed_mode now resolves to the
`default` profile (resolve_speed_mode), since compile's perturbation sits below the
quantisation noise floor and so does not reduce quality versus the dense reference;
out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models
stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is
always honored, so the byte-identical path remains one flag away and is the
regression reference.
Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/
perf_verify.py (the B200 verification above), and diffusion_bench.py gains
--speed-mode so the speed tiers are benchmarkable.
Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore,
GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags +
the engine de-dup, and the sd-cli silent-hang timeout.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks
The opt-in `max` speed tier now compiles the repeated block with
mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode:
Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer
cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are
deliberately avoided -- both crash on the regionally-compiled block (its static
output buffer is overwritten across denoise steps), measured.
Adds two reproducible benchmarks used to validate the optimization research:
- scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head.
- scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes.
Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen;
coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune);
FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source)
Add an opt-in transformer_quant mode that loads the dense bf16 transformer and
torchao-quantises it onto the low-precision tensor cores, instead of the GGUF
transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a
B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's
0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor.
GGUF+compile stays the low-memory default and the fallback. The mode is gated on
CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB);
any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged
reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe
(Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips
the tiny projections that crash int8's torch._int_mm. New module mirrors
diffusion_precision.py; quant runs before compile before placement.
184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route
coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py
is the standalone torchao lever probe.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity
Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838
TFLOPS with FP16 accumulate; bf16 209), so:
- fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on;
pinning it guards consumer cards against a default change. On B200 it is identical
speed and slightly better quality (LPIPS 0.050 vs 0.091).
- the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate).
2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py):
2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the
cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile
(our main ~2x). Documented as a dead end, not shipped.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe
scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image
generation and reports max-abs + non-finite counts for use_fast_accum True vs False.
Confirms fast accumulation is an accumulation-precision knob, not an overflow one:
across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16),
0 non-finite elements and identical max-abs for both modes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override
Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want
fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and
prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match
on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40;
GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it.
Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s),
no overflow, quality below the quant noise floor. So the default leans to accuracy on
data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force)
lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off).
187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override
threading).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9
scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer.
Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with
use_triton_kernel=False (the default triton path needs the missing MSLK library), but
only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than
using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's
CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto
on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder
Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4
tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS,
2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the
diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both
slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166
vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is
correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add
scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 9): pre-quantized transformer loading
The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on
the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs
13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline
(scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton
on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True)
the quantized weights, so the dense bf16 never touches the GPU.
Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching
GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same
torchao config + min_features filter the runtime path uses, applied ahead of time.
New core/inference/diffusion_prequant.py (resolve_prequant_source +
load_prequantized_transformer, best-effort, lazy imports). diffusion.py
_load_dense_quant_pipeline tries the pre-quant source first and falls back to the
dense materialise+quantise path, then to GGUF, so the default is unchanged.
DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty
prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for
the resolver, the meta-init+assign loader, and the backend branch selection +
fallbacks; GPU verification via scripts/verify_prequant_backend.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 10): attention-backend selection
Add a selectable attention kernel via the diffusers set_attention_backend
dispatcher. Attention is memory-bandwidth bound, so a better kernel is an
end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV
matmuls torchao never touches) and composes with torch.compile.
auto picks the best exact backend for the device: cuDNN fused attention
(_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x
end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default
(below the compile/quant noise floor); native SDPA elsewhere and when speed=off
(so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/
xformers/aiter are honored, and an unavailable kernel falls back to the default
rather than failing the load.
New core/inference/diffusion_attention.py (normalize + per-device select + apply,
best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline;
attention_backend threads through begin_load / load_pipeline / status like the other
load knobs. New request field attention_backend + status field. Hermetic CPU tests
for normalize / select policy / apply fallback, plus route threading + 422. Measured
via scripts/perf_levers_probe.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder
Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core
throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public
benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm
is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada
consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the
arch tier to put int8 first on a consumer/workstation GPU (detected by the existing
_is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first.
Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression
on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation
(-> int8) and data-center Ada/Hopper/Blackwell (-> fp8).
* Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT
Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across
denoise steps a DiT's output settles, so once the first block's residual barely
changes the remaining blocks are skipped and their cached output reused. diffusers
ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the
standalone apply_first_block_cache hook as a fallback).
Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 ->
2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar.
OFF by default and a per-load opt-in: the win scales with step count, so it is for
many-step models (Flux / Qwen-Image) and pointless for few-step distilled models
(e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction
of the trajectory. It composes with regional compile only with fullgraph=False (the
cache's per-step decision is a torch.compiler.disable graph break), which the speed
layer now switches to automatically when a cache is engaged. Best-effort: a model
whose block signature the hook does not recognise is caught and the load proceeds
uncached.
- new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache
(enable_cache / apply_first_block_cache fallback; threshold auto-raised for a
quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import).
- diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph
when a cache is engaged.
- diffusion.py: apply_step_cache before compile; thread transformer_cache /
transformer_cache_threshold through begin_load -> load_pipeline and report the
engaged mode in status().
- models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and
transformer_cache_threshold request fields, engaged mode in the status response.
- hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold
selection, and best-effort failure handling, plus route threading + validation.
- scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM /
LPIPS vs the compiled no-cache baseline).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears)
The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with
'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant
goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN
modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2
*_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are
computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so
they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing
min_features filter did not exclude them.
Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a
modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed,
guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a
negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers
(M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and
quantises these layers fine, so the exclusion is int8-only. Sequence embedders
(context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note
'context_embedder' contains the substring 'text_embed', which is why the token is the
specific 'time_text_embed', not 'text_embed'.
Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every
supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's
FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s
(5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already
working) are unchanged.
- diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes
exclude_name_tokens; quantize_transformer passes it for int8 only.
- hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps
attention / FFN / sequence-embedder linears), while fp8 keeps them.
- scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's
Linear layers and derive the exclusion list.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears)
The prequant-checkpoint builder applied the dense quant filter without the int8-only
M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8
checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the
first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared
exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder
so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now
produces a working checkpoint on every supported model, giving int8 (the consumer-preferred
scheme) the same ~2x load-VRAM and download reduction fp8 already had.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine
When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the
native stable-diffusion.cpp engine instead of diffusers, with diffusers as the
guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM.
- diffusion_engine_router: centralised engine selection (built on the existing
select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason.
- sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface
backed by sd-cli, with lazy binary install, registry-driven asset fetch,
step-progress parsing, and cancellation.
- diffusion_families: per-family single-file VAE + text-encoder asset mapping.
- sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled).
- routes/inference + gpu_arbiter: drive the active engine via the router; the
API now reports the active engine and any fallback reason.
- tests for the backend, router, route selection, and cancellation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler
Address review feedback on #6724:
- engine router: unload the engine being deactivated on a switch, so the old
model is not left resident-but-unreachable (the evictor only targets the active
engine).
- generate route: sd.cpp execution errors (nonzero exit / timeout / missing
output) now map to 500, not 409 (which only means not-loaded / cancelled).
- native batch: return per-image seeds and persist the actual seed for each image
so every batch image is reproducible.
- Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the
stable-diffusion.cpp docs; other families keep sd-cli defaults.
- honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload
offload flags on the native load instead of hardcoding them off.
- fail the load when the sd-cli binary is present but not runnable (version()
now returns None on exec error / nonzero exit).
- size estimate: only treat the transformer asset as a possible local path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in
load_prequantized_transformer ends in torch.load(weights_only=False), which executes
arbitrary code from the pickle. The transformer_prequant_path load-request field reached
that unpickle for any local file an authenticated caller named, so a request could trigger
remote code execution. Refuse the source.kind=='path' branch unless the operator sets
UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted
and unaffected. Document the requirement on the API field and add gate tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa
- apply_attention_backend now restores the native default when no backend is requested or a
kernel fails. diffusers keeps a process-wide active attention backend that
set_attention_backend updates, and a fresh transformer's processors follow it, so a load
that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile
load pinned, breaking the bit-identical/off guarantee.
- select_attention_backend drops flash3/flash4 up front when the CUDA capability is below
Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit
request on the wrong card set fine then crashed mid-generation; it now falls back to native.
- Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a
valid alias of native) is accepted instead of 422-rejected by Pydantic.
- Drop the dead replace('-','_') normalization (no alias uses dashes/underscores).
- perf_levers_probe.py output dir is now relative to the script, not a hardcoded path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF
- apply_step_cache now engages only via the transformer's native enable_cache (the diffusers
CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a
cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin
transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported
transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model
now runs uncached per the best-effort contract.
- GGUF transformers are quantized (the default Studio load path), so they now use the higher
quantized FBCache threshold when the caller leaves it unset, instead of the dense default
that could keep the cache from triggering.
- fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so
fullgraph=True failed warmup and silently measured an eager cached run); output dir is now
relative to the script, not a hardcoded path.
* Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder
_is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as
consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead
of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already
classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the
same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate.
Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0).
* Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config
Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just
TypeError on older torchao), which would break the mxfp8 config helper instead of falling
back to the default. Catch both so the fallback is robust.
quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds
CUDA memory during the per-row VRAM probe; output dir relative to the script.
* Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load
- snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform
missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the
whole snapshot. restore_backend_flags restores each flag independently so one failure can't
leave the others leaked process-wide.
- load_pipeline restores the flags (and clears the GPU cache) when the build fails after
apply_speed_optims mutated the process-wide flags but before _state captured them for unload
to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and
contaminated later off generations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output
Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs
without producing output (or without closing stdout) would never reach proc.wait and the
wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the
PROCESS, so the main thread always enforces the timeout and kills a hung process (which
closes the pipe and ends the reader). Add a test that times out even when stdout blocks,
and make the no-binary test hermetic so a host-installed sd-cli can't leak in.
* Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn
The filter callback can be invoked without a module name, so fqn.lower() would raise
AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens,
so the linear is kept) instead of crashing the quantise pass.
* Studio diffusion (Phase 16) review fixes: native engine robustness
- sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64);
a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds
aliased to the same image. Random seeds stay 53-bit (JS-safe).
- sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub
fall back to anonymous instead of failing auth on a blank token.
- sd_cpp_backend: a superseding load now cancels the in-flight generation, so the
old sd-cli can no longer return/persist an image from the previous model.
- diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a
slow 10+ GB free / CUDA sync does not block engine selection.
- diffusion_engine_router: probe sd-cli runnability (version()) before committing to
native, so a present-but-unrunnable binary falls back to diffusers at selection.
- diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a
CPU-only install can still reach the native sd.cpp engine instead of failing load.
- tests updated for the runnability probe + a not-runnable fallback case.
* Studio diffusion (Phase 9) review fixes: prequant safety + validation
- SECURITY: a request-supplied local pre-quant path is now unpickled only when it
resolves inside an operator-configured ALLOWLIST of directories
(UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in,
once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on
any path a load request named (arbitrary code execution). realpath() blocks symlink
escapes; a bare on/off toggle is no longer a wildcard.
- Validate the checkpoint's min_features against the runtime Linear filter, so a
checkpoint that quantised a different layer set is rejected instead of silently
loading a model that mismatches the dense path while reporting the same scheme.
- Tolerant base_model_id compare (exact or same final path/repo segment), so a local
path or fork of the canonical base is accepted instead of falling back to dense.
- _has_meta_tensors uses any(chain(...)) (no intermediate lists).
- prequant verify/probe scripts use repo-relative paths (+ env overrides), not the
author's absolute /mnt paths.
- tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts
- diffusion_memory: when group offload is unavailable and the plan falls back to
whole-module offload, enable VAE tiling (the group plan left it off, but the fallback
is the low-VRAM path where the decode spike can OOM). Covers both the group and
sequential fallback branches.
- perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a
balanced bit-identity regression actually fails the check instead of exiting 0.
- compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and
the load-progress poll has a 30 min deadline instead of looping forever on a hang.
- test for the group->model fallback enabling VAE tiling.
* Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path
- diffusion: a torchao-quantized transformer is committed only compiled. A dense model
resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF
it replaced), so when transformer_quant engaged and speed resolved to off, promote to
default (regional compile); warn loudly if compile still does not engage.
- diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the
CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK);
otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF.
- nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path).
- test asserts the eager-quant -> default-compile promotion.
* Studio diffusion (Phase 10) review fixes: attention gating + probe isolation
- diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere
NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation,
so auto now stays on native SDPA there.
- diffusion_attention: _active_attention_backend handles get_active_backend() returning an
enum/None (not a tuple); the old unpack always raised and was swallowed, so
the native-restore short-circuit never fired.
- perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS
on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul
so the inductor_flags variant doesn't leak into later compiled rows.
- tests for the SM80 cuDNN gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening
- install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit
timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket);
extract through a per-member containment check (Zip-Slip guard); expanduser the
--install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch
the separately-published cudart runtime DLL archive so sd-cli.exe can start.
- sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the
installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start
sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend
crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie.
- tests: Zip-Slip rejection, normal extraction, studio-home discovery.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs
Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes
the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of
the literal --output path. SdCppEngine.generate checked only the literal path, so
a batch generation would exit 0 and then raise 'no image' (or return a stale
file). generate now returns the literal path when present and otherwise falls
back to the numbered siblings; single-image behavior is unchanged.
Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected
without error.
* Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats
Codex review on the native engine arg builder:
- build_sd_cpp_command emitted --width/--height unconditionally, so an
img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of
the input. width/height are now Optional (None = unset): an image-conditioned
run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives
the size from the input image (set_width_and_height_if_unset); a plain txt2img
run with unset dims keeps the prior 1024x1024 default; explicit dims are always
honored. width/height are read only by the builder, so the type change is local.
- build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...)
that silently swallowed repeats=0 into sd-cli's default of one pass, turning an
explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError
and emits the flag for any explicit value != 1.
Tests: img2img unset dims omit width/height (init_img and ref_images), explicit
dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at
the default. (Two pre-existing binary-discovery tests fail only because a real
sd-cli is installed in this dev environment; unrelated to this change.)
* Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc
Codex review: the transformer_prequant_path field description still told operators
to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the
prior security fix made that variable a directory allowlist -- _allowed_prequant_roots
deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator
following the documented =1 would have every transformer_prequant_path request
silently refused. The description now states it must name one or more allowlisted
directories and that a bare on/off value is not accepted.
Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does
not say =1, and describes an allowlist/directory (guards against doc drift).
* Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset
Codex review on attention-backend selection:
- Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on
pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first
generation with no fallback. select_attention_backend now applies
_cudnn_attention_supported() to an explicit cuDNN request too.
- flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a
Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3
is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a
(min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+.
- apply_attention_backend's success path left diffusers' process-wide active
backend pinned to the kernel it set; a later component whose processors are
unconfigured (backend None) would inherit it. It now resets the global registry
to native after a successful per-transformer set (the transformer keeps its own
backend), best-effort. Also fixed _active_attention_backend: get_active_backend()
returns a (name, fn) tuple, so the prior code stringified the tuple and never
matched a name, defeating the native-restore short-circuit.
Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on
SM90; global registry reset after a successful set; _active_attention_backend
reads the tuple return.
* Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder
Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300
(Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So
_is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the
auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting
llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they
are treated as data-center class and keep the intended fp8-first behavior.
Test: extends the datacenter parametrize with 'NVIDIA B300' and
'NVIDIA GH200 480GB' (now _is_consumer_gpu False).
* Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder
Codex review: the M=1 modulation/embedder exclusion was wired only into the dense
runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called
make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint
quantised the AdaLN modulation and conditioning-embedder linears, and loading it
via transformer_prequant_path (the load path only loads already-quantised tensors,
it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes
for the runtime path.
Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 ->
the M=1 exclusion, every other scheme -> none) and use it in both the runtime
quantiser and the builder, so a prequant artifact's quantised-layer set always
matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion).
Test: int8_exclude_name_tokens returns the exclusion for int8 and () for
fp8/nvfp4/mxfp8.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race
Codex review on the native-engine routing:
- The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat)
unconditionally after engine selection. A native sd.cpp load on a pure-CPU host
never touches the GPU, so that needlessly tore down the resident chat model. The
handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a
CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it.
- sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when
_run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so
the setting was unverifiable. status now derives them from state.offload_flags
(still 'none' on CPU, where the flags are empty).
- _run_load committed the new state without cancelling/waiting on a generation that
started during the (slow) asset download, so a stale sd-cli run against the OLD
model could finish afterward and persist an image from the previous model once the
new load reported ready. The commit now signals the in-flight cancel and waits on
_generate_lock before swapping _state (taken only at commit, so the download never
serialises against generation), mirroring the diffusers load path.
Tests: CPU native load skips the arbiter while a GPU native load takes it; status
reports offload active when flags are set; _run_load cancels and waits for an
in-flight generation before committing.
* Studio diffusion (Phase 14) review round 2: align helper name with the stack
Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the
identical helper already present higher in the diffusion stack (Phase 16). The
helper definition, the runtime quantiser call, and the offline builder are now
byte-identical to that version, so the two branches no longer introduce a
divergent name for the same single-source-of-truth and the stack merges without
a conflict on this fix. No behavior change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion: eager patches + torch.compile cache speed phase
Adds the opt-in speed path for the GGUF diffusion transformer behind a
selectable speed mode (default off, so output is unchanged until a profile
is chosen):
- diffusion_eager_patches.py: shared eager fast-paths (channels_last,
attention/backend selection, fused norms and QKV) installed at load and
rolled back on unload or failed load.
- diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent
torch.compile cache and the GGUF-transformer compile wiring.
- diffusion_arch_patches.py: architecture-specific patches.
- diffusion_patch_backend.py: shared install/restore plumbing.
- diffusion_speed.py: speed-profile planning.
Tests for each module plus the benchmarking and probe scripts used to
measure speed, memory, and accuracy of the path.
* Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI
Backend:
- Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file
fp8 transformers, gated to the unsloth org plus a curated allowlist.
- Image-conditioned workflows built with Pipeline.from_pipe so they reuse the
loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint,
outpaint, and a hires-fix upscale pass.
- Instruction editing as its own family kind (Qwen-Image-Edit-2511,
FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and
multi-reference) plus klein inpaint.
- Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched
mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the
decoded image size and cap upscale output to avoid OOM on large inputs.
- Fixes: from_pipe defaulting to a float32 recast that crashed torchao
quantized transformers; image-conditioned calls forcing the slider size
onto the input image. Native sd.cpp engine rejects image-conditioned and
reference requests it cannot serve.
Frontend:
- Redesigned Images page with capability-gated workflow tabs (Create,
Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor,
client-side outpaint, and a multi-reference picker.
- Advanced options moved to a right-docked panel mirroring Chat: closed by
default, toggled by a single fixed top-bar button that stays in place.
sd.cpp installer: pin the release, verify each download's sha256, add a
download timeout, and make the source repo configurable for a future mirror.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: correct the Advanced panel comment (closed by default, fixed toggle)
* Studio diffusion: LoRA adapters for the Images workflow
Add community LoRA support across both diffusion backends, the single
biggest step toward broad image-workflow coverage.
Backend
- New shared module core/inference/diffusion_lora.py: adapter discovery
(local scan + curated catalog + owner/name[:file] Hub refs), download
via hf_hub_download_with_xet_fallback, alias sanitization, native
managed-dir materialization with collision-broken aliases, prompt-tag
injection (deduped against user-typed tags), and a supports_lora gate.
- Native sd-cli: resolve + materialize selected LoRAs into a per-run
managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with
--lora-apply-mode auto. The arg builder already emitted these flags.
- Diffusers: non-fused load_lora_weights + set_adapters manager, tracked
on the pipe so an unchanged selection is a no-op and a model swap
resets; cleared on unload. Never fuses (breaks quantized transformers
and blocks live weight tweaks).
- Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers,
and native Qwen-Image (no LoRA name-conversion branch upstream).
- Request contract: optional loras on DiffusionGenerateRequest; empty or
omitted is identical to today. supports_lora surfaced in status; chosen
LoRAs persisted in gallery recipe metadata.
- New GET /api/models/diffusion-loras for the picker (family-filtered).
Frontend
- Repeatable multi-LoRA picker (adapter select + weight slider 0..2 +
remove), gated by the loaded model's supports_lora and family, max 8.
Tests
- New test_diffusion_lora.py (14): helpers, request validation, native
tag/dir wiring, diffusers set_adapters manager, supports_lora matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander
Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a
stray "gguf" tag on the Hub but ship no .gguf files. The model search classified
them as GGUF from the bare tag, so the picker rendered the GGUF variant expander,
which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when
the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata
(populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected.
* Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op
handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other
non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo
surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline
load when the id is unsloth-hosted or on-device (the backend infers the family + base repo
and gates loads to unsloth/* or local paths), and show a clear message otherwise instead
of silently ignoring the click. Curated and GGUF paths are unchanged.
* Studio Images: keep curated safetensors models in Recommended after download
The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's
Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet),
so they vanished from the picker after the first load and could only be found by typing an
exact search. The row already renders a downloaded badge, matching how GGUF Recommended
rows stay visible when cached. Drop the exclusion so the curated safetensors always list.
* Studio diffusion LoRA: sanitize dots out of adapter aliases
The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names
containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose
filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to
apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename
stem. Adds regression coverage for internal dots.
* Studio Images: clarify the GGUF transformer-quant Advanced control
Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode"
with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/
FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF,
falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only.
* Studio Images: list on-device unsloth diffusion models in the picker
The Images picker's On Device tab hid every non-GGUF cached repo whenever a
task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit
and FP8 safetensors) never showed up there. List cached repos that pass the
task gate, limited under a filter to unsloth-hosted ones so base repos (which
fail the diffusion load trust gate) don't appear only to dead-end on click.
Chat behavior is unchanged: the task gate still drops image repos there.
* Studio: hide single-file image checkpoints from the chat model picker
The chat picker treats a cached repo as an image model, and hides it, only
when it ships a diffusers model_index.json. Single-file, ComfyUI, and
ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a
Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models.
Fall back to resolving the repo id against the known diffusion families, the
same resolver the Images backend loads from, so these checkpoints are tagged
text-to-image and stay in the Images picker only.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: add the FLUX.2-dev model family
Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the
Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline)
resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline
+ Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated,
reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text
encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror.
text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for
dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from
klein's turbo defaults. Verified live: GGUF load resolves the family + gated base
repo and generates a real 1024x1024 image on GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: clearer error for an unsupported diffusion model
When a repo id resolves to no diffusion family the load raised 'Could not infer a
diffusion family... Pass family_override (z-image)', which points at an unrelated
family and doesn't say what is supported. Replace it with a message that lists the
supported families (from a new supported_family_names helper) and notes that video
models and image models whose diffusers transformer has no single-file loader are
not supported. Applies to both the diffusers and native sd.cpp load paths. Also
refreshes two stale family-registry comments that still called FLUX.2-dev omitted.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove stray async task scratch outputs committed by mistake
* Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown
Address review findings on the LoRA path:
- resolve_one: normalise a blank/whitespace hf_token to None (anonymous access)
and reject a client-supplied weight file with traversal / absolute path.
- resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError
so the route returns 400 instead of a generic 500.
- _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs
foo.gguf) so each is uniquely addressable.
- inject_prompt_tags: the backend-validated weight now wins over a user-typed
<lora:ALIAS:...> for a selected adapter; unselected user tags are left alone.
- diffusers _apply_loras: reject a .gguf adapter with a clear error before touching
the pipe (diffusers loads safetensors only).
- _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is
dropped wholesale (freeing adapters), so the previous call could race an in-flight
denoise on the same pipe.
- Images page: use a stable LoRA key and clear the selection (not just the options)
when the catalog refresh fails.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Diffusion: guard trust check against OSError and validate conditioning inputs
- _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid
characters (or a bare owner/name id) can't raise OSError; treat any failure as
not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request
still raises the clear FileNotFoundError for a genuinely missing local pick.
- generate(): reject mask_image / upscale / reference_images supplied without an
input image, and reject reference_images on a family that does not support
reference conditioning, instead of silently degrading to txt2img / img2img.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Codex review findings on the image-workflows PR
Keep diffusion.py importable without torch: the compile/arch patch modules
import torch at module level, so import them lazily at their load/unload
call sites instead of at module load. This restores the torchless contract
so get_diffusion_backend() works on a CPU/native sd.cpp install.
Match family reject keywords and aliases as whole path/name segments, not
raw substrings, so an unrelated word like edited, edition, or kontextual no
longer misroutes or hides a valid base image model, while supported edit
families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same
segment matching in the picker task filter.
Route FLUX.2-dev native guidance through --guidance like the other FLUX
families rather than --cfg-scale. Reject native upscale requests that have
no input image. Read image header dimensions and reject over-limit inputs
before decoding pixels, so a crafted small-payload image cannot spike
memory. Reject an upscale that would shrink the source below its input
size. Validate the model_kind against the filename extension before the
GPU handoff. Estimate a local diffusers pipeline's size from its on-disk
weights so auto memory planning does not skip offload and OOM. Report
workflows: [txt2img] from the native backend status so the Create tab
stays enabled for a loaded native model. Clamp the outpaint canvas to the
backend's 4096px decode limit.
Adds regression tests for segment matching and kind/extension validation.
* Harden diffusion LoRA handling on the diffusers and native paths
Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max):
diffusers requires the adapter loaded before compilation, so applying one to
the already-compiled module fails with adapter-key mismatches. The status
gate now hides the picker and generate raises a clear message instead.
Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the
diffusion cancellation sentinel in resolve_specs, so an unload/superseding
load during resolution maps to a 409 instead of a generic server error.
Drop weight-0 LoRA rows before the native support gate so a request carrying
only disabled adapters stays a no-op on families where native LoRA is
unsupported, matching the diffusers path.
Reject duplicate LoRA ids in the request model: both apply paths suffix
colliding names, so a repeated id would stack the same adapter past its
per-adapter weight bound.
Strip all user-typed <lora:...> prompt tags on the native path (only the
selected adapters are materialized in the managed lora-model-dir, so an
unselected tag can never resolve), and restore saved LoRA selections from a
gallery recipe so restore reproduces a LoRA image.
* Address further Codex findings on the image-workflows PR
- Persist the actual output image size in the gallery recipe instead of the
request sliders: Transform/Inpaint/Edit derive the size from the uploaded
image, Extend grows the canvas, and Upscale resizes it, so the sliders
recorded (and later restored) the wrong dimensions for those workflows.
- Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file
name) in validate_load_request, so the unloadable pick fails before chat is
evicted rather than deep in from_pretrained.
- Only publish an image-conditioned from_pipe wrapper to the shared aux cache
when the load is still current: from_pipe runs under the generate lock but
not the state lock, so an unload racing its construction could otherwise
cache a wrapper over torn-down modules that a later load would reuse.
- Verify the Windows CUDA runtime archive checksum before extracting it, like
the main sd-cli archive, so a corrupt or tampered runtime is rejected rather
than extracted next to the binary.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Fix external drive custom folder selection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/tests/test_linux_external_media_paths.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep legacy media scan validation strict
* Apply sensitive-dir denylist to legacy folder browser for PR #6799
The legacy /api/models browse endpoint gained the new /run/media mount
roots in its allowlist but not the credential/config guard that scan-folder
registration and the Hub browser already enforce. Filter sensitive names
during enumeration and reject them in _resolve_browse_target so .ssh, .aws,
.config, etc. under allowlisted roots stay unbrowseable, matching the Hub
browser. Add a public contains_sensitive_path_component helper and cover the
legacy resolver with a regression test.
* Trim redundant comments in PR #6799 changes
* Skip sensitive Linux media roots
* Reject sensitive dirs at exact browse roots for PR #6799
Both _resolve_browse_target functions only checked contains_sensitive_path_component
while walking descendant parts, so requesting an allowlisted root itself (empty
relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh,
~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to
the allowlist on upgrade and could then be browsed. Check the resolved target once
before returning in both the legacy and Hub browsers, and cover the root case in
both test suites.
* fix: avoid unused path helper reexports
* fix: import sensitive path helpers directly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict
Phase 1 of porting the richer diffusion stack onto the image-generation backend.
- Add a compartmentalized device/dtype policy module (diffusion_device.py)
resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA
capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or
fp32, never a silent fp16 that renders a black image.
- Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved
float16 to float32 for those families so they do not produce black images.
- Split the backend locks: a generation holds only _generate_lock, so status,
unload, and a new load are never blocked by a long denoise. Add per-generation
cancellation via callback_on_step_end so an eviction or a superseding load
preempts a running generation; a replacement load waits for it to stop before
allocating, so two pipelines never sit in VRAM at once.
- Validate a load request before the GPU handoff so an unloadable pick never
evicts a working chat model, and reject missing local paths up front.
- Add CPU-only tests for the device policy, dtype guard, lock split and
cancellation, and validate-before-evict, plus a GPU benchmark/regression
script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR
against a saved reference.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy
Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and
VAE tiling/slicing from measured free device memory vs the model's estimated
resident footprint, then applies it to the built pipeline. auto stays resident
when the model fits (byte-identical to the prior resident path), and falls to
whole-module offload when tight; fast/balanced/low_vram are explicit overrides.
Sequential submodule offload is unreliable for GGUF transformers on diffusers
0.38, so it falls back to whole-module offload and status reports the policy
actually engaged.
Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with
no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM
47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost.
73 prior + 35 new CPU tests pass.
* Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling
Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level,
use_stream) that keeps the transformer flowing through the GPU a few blocks at a
time while the text encoder / VAE stay resident, and fix VAE tiling to drive the
VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the
pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so
status never overstates either, and group falls back to whole-module offload when
the transformer can't be streamed.
Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group
cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 ->
2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names
now match that tradeoff: balanced = stream the transformer, low_vram = offload
every component. auto picks group when the companions fit resident, else model.
112 CPU tests pass.
* Studio diffusion (Phase 5): image quality-vs-quant accuracy harness
Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold
prompt + seed fixed, render a grid with a reference quant (default BF16), then render
each candidate quant and measure drift from the reference. Records mean PSNR + SSIM
(pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity
(transformers, --clip), plus file size, latency, and peak VRAM, then prints a
quality-vs-cost table and recommends the smallest quant within a quality budget.
--selftest validates the metrics on synthetic images with no GPU or model.
Verified on Z-Image (B200): the table degrades monotonically with quant size
(Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat
(~0.34) -- quantization erodes fine detail far more than prompt adherence.
* Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32)
Add a speed_mode knob (off by default, so the render path stays bit-identical):
default applies channels_last VAE + regional torch.compile of the denoiser's
repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional
compile is gated off for the GGUF transformer (dequantises per-op) and for families
flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image),
so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed
optims run before placement/offload, per the diffusers composition order. status now
reports speed_mode + the optims actually engaged.
Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last',
'tf32'], compile correctly skipped for GGUF; generation works in every mode.
121 CPU tests pass.
* Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting
Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3)
storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16
compute dtype while normalisations and embeddings stay full precision. Applied
before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder
dense). status reports which encoders were cast.
Verified on Z-Image (B200, balanced/group mode where the encoder stays resident):
generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload)
at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR
vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off
by default and documented as such, with the Phase 5 harness to size the cost.
127 CPU tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob)
Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant
(fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao
NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8
stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run
before placement; status reports the mode actually engaged. This is the lean
realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the
3045-line port.
Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the
bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE
option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and
both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default);
size it per model with the Phase 5 quality harness. diffusion_bench gains
--text-encoder-quant.
129 CPU tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac
Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the
chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm
/ XPU; this covers the hardware diffusers serves poorly, consuming the same
split GGUF assets Studio already curates.
- sd_cpp_args.py: pure sd-cli command builder. Maps the family to its
text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1
CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential)
to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu /
--vae-tiling / --diffusion-fa), so one user knob drives both engines.
- sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary()
with the same precedence as the llama finder (env override, then the Studio
install root, then in-tree, then PATH), an is_available/version probe, and a
one-shot subprocess generate that streams progress and returns the PNG.
runtime_env() prepends the binary's directory to the platform library path
so a prebuilt's bundled libstable-diffusion.so resolves.
select_diffusion_engine() is the pure routing decision (GPU backends to
diffusers, CPU/MPS to native when present).
- install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt
(macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the
Studio install root. resolve_release_asset() is a pure, unit-tested
host-to-asset matrix.
- scripts/sd_cpp_smoke.py: end-to-end native generation harness.
Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine,
routing, runtime env, and the installer resolver. Full diffusion suite 166
passing.
Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both
generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group
offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the
dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine
Builds on Phase 4's native stable-diffusion.cpp engine, extending it from
text-to-image to the wider feature surface, since sd.cpp supports all of these
through the binary already. Pure command-builder additions plus one engine
method, so the txt2img path is unchanged.
- sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img +
strength make a run img2img, adding mask makes it inpaint, ref_images drives
FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and
lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New
SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run
mode (input image + esrgan model, no prompt / text encoders).
- sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so
generate() (now carrying the conditioning flags) and a new upscale() reuse
the same streaming / error / output-check path.
- scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img /
--strength / --upscale-model / --upscale-repeats.
Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the
upscale builder and its validation, and the engine's img2img + upscale paths.
Full diffusion suite 176 passing.
Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the
init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale
(512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing
coherent images. Video and the diffusers-path feature wiring are deferred.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7): accuracy-preserving speed pass
Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy
bug and a dead-on-arrival speed path; this fixes both and adds the lossless /
near-lossless wins, all measured on a B200.
Correctness:
- TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32
process-wide and never restored them, so a later `off` load silently inherited
TF32 and was no longer bit-identical. Added snapshot_backend_flags /
restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer
runs and restored on unload. Verified: load max -> unload -> load off is now
byte-identical (PSNR inf) to a fresh off.
- sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and
only checked the timeout after EOF, so a child stuck in model load / GPU init
with no output ignored the timeout. Drained stdout on a reader thread with a
wall-clock deadline. Added a silent-hang regression test.
Speed (diffusers path), near-lossless, opt-in tiers:
- Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and
Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks
compiles and runs ~2.2x faster on the GGUF Z-Image transformer on
torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the
block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB
vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move
output quality. Gate relaxed; default tier delivers it.
- cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs).
- torch.inference_mode() around the pipeline call (lossless, strictly faster than
the no_grad diffusers uses internally).
Memory path:
- VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers;
the balanced (group) tier keeps exact slicing only, so it is now bit-identical to
the resident image (verified PSNR inf) and slightly faster.
- Group offload adds non_blocking + record_stream on the CUDA stream path to
overlap each block's H2D copy with compute (lossless; gated on the installed
diffusers signature so older versions still work).
Native (sd.cpp) path:
- native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a
near-lossless CUDA win that was previously only added on offload tiers; max also
-> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so
it is never auto-on. Engine generate() merges it, de-duped against offload flags.
Default profile: a GGUF model with no explicit speed_mode now resolves to the
`default` profile (resolve_speed_mode), since compile's perturbation sits below the
quantisation noise floor and so does not reduce quality versus the dense reference;
out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models
stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is
always honored, so the byte-identical path remains one flag away and is the
regression reference.
Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/
perf_verify.py (the B200 verification above), and diffusion_bench.py gains
--speed-mode so the speed tiers are benchmarkable.
Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore,
GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags +
the engine de-dup, and the sd-cli silent-hang timeout.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks
The opt-in `max` speed tier now compiles the repeated block with
mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode:
Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer
cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are
deliberately avoided -- both crash on the regionally-compiled block (its static
output buffer is overwritten across denoise steps), measured.
Adds two reproducible benchmarks used to validate the optimization research:
- scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head.
- scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes.
Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen;
coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune);
FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source)
Add an opt-in transformer_quant mode that loads the dense bf16 transformer and
torchao-quantises it onto the low-precision tensor cores, instead of the GGUF
transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a
B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's
0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor.
GGUF+compile stays the low-memory default and the fallback. The mode is gated on
CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB);
any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged
reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe
(Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips
the tiny projections that crash int8's torch._int_mm. New module mirrors
diffusion_precision.py; quant runs before compile before placement.
184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route
coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py
is the standalone torchao lever probe.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity
Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838
TFLOPS with FP16 accumulate; bf16 209), so:
- fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on;
pinning it guards consumer cards against a default change. On B200 it is identical
speed and slightly better quality (LPIPS 0.050 vs 0.091).
- the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate).
2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py):
2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the
cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile
(our main ~2x). Documented as a dead end, not shipped.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe
scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image
generation and reports max-abs + non-finite counts for use_fast_accum True vs False.
Confirms fast accumulation is an accumulation-precision knob, not an overflow one:
across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16),
0 non-finite elements and identical max-abs for both modes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override
Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want
fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and
prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match
on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40;
GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it.
Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s),
no overflow, quality below the quant noise floor. So the default leans to accuracy on
data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force)
lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off).
187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override
threading).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9
scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer.
Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with
use_triton_kernel=False (the default triton path needs the missing MSLK library), but
only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than
using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's
CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto
on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder
Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4
tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS,
2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the
diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both
slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166
vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is
correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add
scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 9): pre-quantized transformer loading
The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on
the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs
13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline
(scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton
on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True)
the quantized weights, so the dense bf16 never touches the GPU.
Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching
GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same
torchao config + min_features filter the runtime path uses, applied ahead of time.
New core/inference/diffusion_prequant.py (resolve_prequant_source +
load_prequantized_transformer, best-effort, lazy imports). diffusion.py
_load_dense_quant_pipeline tries the pre-quant source first and falls back to the
dense materialise+quantise path, then to GGUF, so the default is unchanged.
DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty
prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for
the resolver, the meta-init+assign loader, and the backend branch selection +
fallbacks; GPU verification via scripts/verify_prequant_backend.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 10): attention-backend selection
Add a selectable attention kernel via the diffusers set_attention_backend
dispatcher. Attention is memory-bandwidth bound, so a better kernel is an
end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV
matmuls torchao never touches) and composes with torch.compile.
auto picks the best exact backend for the device: cuDNN fused attention
(_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x
end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default
(below the compile/quant noise floor); native SDPA elsewhere and when speed=off
(so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/
xformers/aiter are honored, and an unavailable kernel falls back to the default
rather than failing the load.
New core/inference/diffusion_attention.py (normalize + per-device select + apply,
best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline;
attention_backend threads through begin_load / load_pipeline / status like the other
load knobs. New request field attention_backend + status field. Hermetic CPU tests
for normalize / select policy / apply fallback, plus route threading + 422. Measured
via scripts/perf_levers_probe.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder
Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core
throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public
benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm
is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada
consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the
arch tier to put int8 first on a consumer/workstation GPU (detected by the existing
_is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first.
Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression
on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation
(-> int8) and data-center Ada/Hopper/Blackwell (-> fp8).
* Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT
Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across
denoise steps a DiT's output settles, so once the first block's residual barely
changes the remaining blocks are skipped and their cached output reused. diffusers
ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the
standalone apply_first_block_cache hook as a fallback).
Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 ->
2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar.
OFF by default and a per-load opt-in: the win scales with step count, so it is for
many-step models (Flux / Qwen-Image) and pointless for few-step distilled models
(e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction
of the trajectory. It composes with regional compile only with fullgraph=False (the
cache's per-step decision is a torch.compiler.disable graph break), which the speed
layer now switches to automatically when a cache is engaged. Best-effort: a model
whose block signature the hook does not recognise is caught and the load proceeds
uncached.
- new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache
(enable_cache / apply_first_block_cache fallback; threshold auto-raised for a
quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import).
- diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph
when a cache is engaged.
- diffusion.py: apply_step_cache before compile; thread transformer_cache /
transformer_cache_threshold through begin_load -> load_pipeline and report the
engaged mode in status().
- models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and
transformer_cache_threshold request fields, engaged mode in the status response.
- hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold
selection, and best-effort failure handling, plus route threading + validation.
- scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM /
LPIPS vs the compiled no-cache baseline).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears)
The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with
'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant
goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN
modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2
*_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are
computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so
they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing
min_features filter did not exclude them.
Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a
modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed,
guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a
negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers
(M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and
quantises these layers fine, so the exclusion is int8-only. Sequence embedders
(context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note
'context_embedder' contains the substring 'text_embed', which is why the token is the
specific 'time_text_embed', not 'text_embed'.
Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every
supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's
FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s
(5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already
working) are unchanged.
- diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes
exclude_name_tokens; quantize_transformer passes it for int8 only.
- hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps
attention / FFN / sequence-embedder linears), while fp8 keeps them.
- scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's
Linear layers and derive the exclusion list.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears)
The prequant-checkpoint builder applied the dense quant filter without the int8-only
M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8
checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the
first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared
exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder
so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now
produces a working checkpoint on every supported model, giving int8 (the consumer-preferred
scheme) the same ~2x load-VRAM and download reduction fp8 already had.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine
When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the
native stable-diffusion.cpp engine instead of diffusers, with diffusers as the
guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM.
- diffusion_engine_router: centralised engine selection (built on the existing
select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason.
- sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface
backed by sd-cli, with lazy binary install, registry-driven asset fetch,
step-progress parsing, and cancellation.
- diffusion_families: per-family single-file VAE + text-encoder asset mapping.
- sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled).
- routes/inference + gpu_arbiter: drive the active engine via the router; the
API now reports the active engine and any fallback reason.
- tests for the backend, router, route selection, and cancellation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler
Address review feedback on #6724:
- engine router: unload the engine being deactivated on a switch, so the old
model is not left resident-but-unreachable (the evictor only targets the active
engine).
- generate route: sd.cpp execution errors (nonzero exit / timeout / missing
output) now map to 500, not 409 (which only means not-loaded / cancelled).
- native batch: return per-image seeds and persist the actual seed for each image
so every batch image is reproducible.
- Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the
stable-diffusion.cpp docs; other families keep sd-cli defaults.
- honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload
offload flags on the native load instead of hardcoding them off.
- fail the load when the sd-cli binary is present but not runnable (version()
now returns None on exec error / nonzero exit).
- size estimate: only treat the transformer asset as a possible local path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in
load_prequantized_transformer ends in torch.load(weights_only=False), which executes
arbitrary code from the pickle. The transformer_prequant_path load-request field reached
that unpickle for any local file an authenticated caller named, so a request could trigger
remote code execution. Refuse the source.kind=='path' branch unless the operator sets
UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted
and unaffected. Document the requirement on the API field and add gate tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa
- apply_attention_backend now restores the native default when no backend is requested or a
kernel fails. diffusers keeps a process-wide active attention backend that
set_attention_backend updates, and a fresh transformer's processors follow it, so a load
that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile
load pinned, breaking the bit-identical/off guarantee.
- select_attention_backend drops flash3/flash4 up front when the CUDA capability is below
Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit
request on the wrong card set fine then crashed mid-generation; it now falls back to native.
- Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a
valid alias of native) is accepted instead of 422-rejected by Pydantic.
- Drop the dead replace('-','_') normalization (no alias uses dashes/underscores).
- perf_levers_probe.py output dir is now relative to the script, not a hardcoded path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF
- apply_step_cache now engages only via the transformer's native enable_cache (the diffusers
CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a
cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin
transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported
transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model
now runs uncached per the best-effort contract.
- GGUF transformers are quantized (the default Studio load path), so they now use the higher
quantized FBCache threshold when the caller leaves it unset, instead of the dense default
that could keep the cache from triggering.
- fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so
fullgraph=True failed warmup and silently measured an eager cached run); output dir is now
relative to the script, not a hardcoded path.
* Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder
_is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as
consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead
of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already
classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the
same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate.
Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0).
* Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config
Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just
TypeError on older torchao), which would break the mxfp8 config helper instead of falling
back to the default. Catch both so the fallback is robust.
quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds
CUDA memory during the per-row VRAM probe; output dir relative to the script.
* Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load
- snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform
missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the
whole snapshot. restore_backend_flags restores each flag independently so one failure can't
leave the others leaked process-wide.
- load_pipeline restores the flags (and clears the GPU cache) when the build fails after
apply_speed_optims mutated the process-wide flags but before _state captured them for unload
to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and
contaminated later off generations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output
Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs
without producing output (or without closing stdout) would never reach proc.wait and the
wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the
PROCESS, so the main thread always enforces the timeout and kills a hung process (which
closes the pipe and ends the reader). Add a test that times out even when stdout blocks,
and make the no-binary test hermetic so a host-installed sd-cli can't leak in.
* Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn
The filter callback can be invoked without a module name, so fqn.lower() would raise
AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens,
so the linear is kept) instead of crashing the quantise pass.
* Studio diffusion (Phase 16) review fixes: native engine robustness
- sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64);
a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds
aliased to the same image. Random seeds stay 53-bit (JS-safe).
- sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub
fall back to anonymous instead of failing auth on a blank token.
- sd_cpp_backend: a superseding load now cancels the in-flight generation, so the
old sd-cli can no longer return/persist an image from the previous model.
- diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a
slow 10+ GB free / CUDA sync does not block engine selection.
- diffusion_engine_router: probe sd-cli runnability (version()) before committing to
native, so a present-but-unrunnable binary falls back to diffusers at selection.
- diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a
CPU-only install can still reach the native sd.cpp engine instead of failing load.
- tests updated for the runnability probe + a not-runnable fallback case.
* Studio diffusion (Phase 9) review fixes: prequant safety + validation
- SECURITY: a request-supplied local pre-quant path is now unpickled only when it
resolves inside an operator-configured ALLOWLIST of directories
(UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in,
once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on
any path a load request named (arbitrary code execution). realpath() blocks symlink
escapes; a bare on/off toggle is no longer a wildcard.
- Validate the checkpoint's min_features against the runtime Linear filter, so a
checkpoint that quantised a different layer set is rejected instead of silently
loading a model that mismatches the dense path while reporting the same scheme.
- Tolerant base_model_id compare (exact or same final path/repo segment), so a local
path or fork of the canonical base is accepted instead of falling back to dense.
- _has_meta_tensors uses any(chain(...)) (no intermediate lists).
- prequant verify/probe scripts use repo-relative paths (+ env overrides), not the
author's absolute /mnt paths.
- tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts
- diffusion_memory: when group offload is unavailable and the plan falls back to
whole-module offload, enable VAE tiling (the group plan left it off, but the fallback
is the low-VRAM path where the decode spike can OOM). Covers both the group and
sequential fallback branches.
- perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a
balanced bit-identity regression actually fails the check instead of exiting 0.
- compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and
the load-progress poll has a 30 min deadline instead of looping forever on a hang.
- test for the group->model fallback enabling VAE tiling.
* Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path
- diffusion: a torchao-quantized transformer is committed only compiled. A dense model
resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF
it replaced), so when transformer_quant engaged and speed resolved to off, promote to
default (regional compile); warn loudly if compile still does not engage.
- diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the
CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK);
otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF.
- nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path).
- test asserts the eager-quant -> default-compile promotion.
* Studio diffusion (Phase 10) review fixes: attention gating + probe isolation
- diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere
NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation,
so auto now stays on native SDPA there.
- diffusion_attention: _active_attention_backend handles get_active_backend() returning an
enum/None (not a tuple); the old unpack always raised and was swallowed, so
the native-restore short-circuit never fired.
- perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS
on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul
so the inductor_flags variant doesn't leak into later compiled rows.
- tests for the SM80 cuDNN gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening
- install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit
timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket);
extract through a per-member containment check (Zip-Slip guard); expanduser the
--install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch
the separately-published cudart runtime DLL archive so sd-cli.exe can start.
- sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the
installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start
sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend
crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie.
- tests: Zip-Slip rejection, normal extraction, studio-home discovery.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs
Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes
the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of
the literal --output path. SdCppEngine.generate checked only the literal path, so
a batch generation would exit 0 and then raise 'no image' (or return a stale
file). generate now returns the literal path when present and otherwise falls
back to the numbered siblings; single-image behavior is unchanged.
Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected
without error.
* Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats
Codex review on the native engine arg builder:
- build_sd_cpp_command emitted --width/--height unconditionally, so an
img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of
the input. width/height are now Optional (None = unset): an image-conditioned
run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives
the size from the input image (set_width_and_height_if_unset); a plain txt2img
run with unset dims keeps the prior 1024x1024 default; explicit dims are always
honored. width/height are read only by the builder, so the type change is local.
- build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...)
that silently swallowed repeats=0 into sd-cli's default of one pass, turning an
explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError
and emits the flag for any explicit value != 1.
Tests: img2img unset dims omit width/height (init_img and ref_images), explicit
dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at
the default. (Two pre-existing binary-discovery tests fail only because a real
sd-cli is installed in this dev environment; unrelated to this change.)
* Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc
Codex review: the transformer_prequant_path field description still told operators
to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the
prior security fix made that variable a directory allowlist -- _allowed_prequant_roots
deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator
following the documented =1 would have every transformer_prequant_path request
silently refused. The description now states it must name one or more allowlisted
directories and that a bare on/off value is not accepted.
Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does
not say =1, and describes an allowlist/directory (guards against doc drift).
* Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset
Codex review on attention-backend selection:
- Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on
pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first
generation with no fallback. select_attention_backend now applies
_cudnn_attention_supported() to an explicit cuDNN request too.
- flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a
Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3
is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a
(min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+.
- apply_attention_backend's success path left diffusers' process-wide active
backend pinned to the kernel it set; a later component whose processors are
unconfigured (backend None) would inherit it. It now resets the global registry
to native after a successful per-transformer set (the transformer keeps its own
backend), best-effort. Also fixed _active_attention_backend: get_active_backend()
returns a (name, fn) tuple, so the prior code stringified the tuple and never
matched a name, defeating the native-restore short-circuit.
Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on
SM90; global registry reset after a successful set; _active_attention_backend
reads the tuple return.
* Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder
Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300
(Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So
_is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the
auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting
llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they
are treated as data-center class and keep the intended fp8-first behavior.
Test: extends the datacenter parametrize with 'NVIDIA B300' and
'NVIDIA GH200 480GB' (now _is_consumer_gpu False).
* Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder
Codex review: the M=1 modulation/embedder exclusion was wired only into the dense
runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called
make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint
quantised the AdaLN modulation and conditioning-embedder linears, and loading it
via transformer_prequant_path (the load path only loads already-quantised tensors,
it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes
for the runtime path.
Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 ->
the M=1 exclusion, every other scheme -> none) and use it in both the runtime
quantiser and the builder, so a prequant artifact's quantised-layer set always
matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion).
Test: int8_exclude_name_tokens returns the exclusion for int8 and () for
fp8/nvfp4/mxfp8.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race
Codex review on the native-engine routing:
- The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat)
unconditionally after engine selection. A native sd.cpp load on a pure-CPU host
never touches the GPU, so that needlessly tore down the resident chat model. The
handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a
CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it.
- sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when
_run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so
the setting was unverifiable. status now derives them from state.offload_flags
(still 'none' on CPU, where the flags are empty).
- _run_load committed the new state without cancelling/waiting on a generation that
started during the (slow) asset download, so a stale sd-cli run against the OLD
model could finish afterward and persist an image from the previous model once the
new load reported ready. The commit now signals the in-flight cancel and waits on
_generate_lock before swapping _state (taken only at commit, so the download never
serialises against generation), mirroring the diffusers load path.
Tests: CPU native load skips the arbiter while a GPU native load takes it; status
reports offload active when flags are set; _run_load cancels and waits for an
in-flight generation before committing.
* Studio diffusion (Phase 14) review round 2: align helper name with the stack
Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the
identical helper already present higher in the diffusion stack (Phase 16). The
helper definition, the runtime quantiser call, and the offline builder are now
byte-identical to that version, so the two branches no longer introduce a
divergent name for the same single-source-of-truth and the stack merges without
a conflict on this fix. No behavior change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio diffusion: eager patches + torch.compile cache speed phase
Adds the opt-in speed path for the GGUF diffusion transformer behind a
selectable speed mode (default off, so output is unchanged until a profile
is chosen):
- diffusion_eager_patches.py: shared eager fast-paths (channels_last,
attention/backend selection, fused norms and QKV) installed at load and
rolled back on unload or failed load.
- diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent
torch.compile cache and the GGUF-transformer compile wiring.
- diffusion_arch_patches.py: architecture-specific patches.
- diffusion_patch_backend.py: shared install/restore plumbing.
- diffusion_speed.py: speed-profile planning.
Tests for each module plus the benchmarking and probe scripts used to
measure speed, memory, and accuracy of the path.
* Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI
Backend:
- Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file
fp8 transformers, gated to the unsloth org plus a curated allowlist.
- Image-conditioned workflows built with Pipeline.from_pipe so they reuse the
loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint,
outpaint, and a hires-fix upscale pass.
- Instruction editing as its own family kind (Qwen-Image-Edit-2511,
FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and
multi-reference) plus klein inpaint.
- Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched
mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the
decoded image size and cap upscale output to avoid OOM on large inputs.
- Fixes: from_pipe defaulting to a float32 recast that crashed torchao
quantized transformers; image-conditioned calls forcing the slider size
onto the input image. Native sd.cpp engine rejects image-conditioned and
reference requests it cannot serve.
Frontend:
- Redesigned Images page with capability-gated workflow tabs (Create,
Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor,
client-side outpaint, and a multi-reference picker.
- Advanced options moved to a right-docked panel mirroring Chat: closed by
default, toggled by a single fixed top-bar button that stays in place.
sd.cpp installer: pin the release, verify each download's sha256, add a
download timeout, and make the source repo configurable for a future mirror.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: correct the Advanced panel comment (closed by default, fixed toggle)
* Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander
Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a
stray "gguf" tag on the Hub but ship no .gguf files. The model search classified
them as GGUF from the bare tag, so the picker rendered the GGUF variant expander,
which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when
the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata
(populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected.
* Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op
handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other
non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo
surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline
load when the id is unsloth-hosted or on-device (the backend infers the family + base repo
and gates loads to unsloth/* or local paths), and show a clear message otherwise instead
of silently ignoring the click. Curated and GGUF paths are unchanged.
* Studio Images: keep curated safetensors models in Recommended after download
The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's
Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet),
so they vanished from the picker after the first load and could only be found by typing an
exact search. The row already renders a downloaded badge, matching how GGUF Recommended
rows stay visible when cached. Drop the exclusion so the curated safetensors always list.
* Studio Images: clarify the GGUF transformer-quant Advanced control
Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode"
with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/
FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF,
falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only.
* Studio Images: list on-device unsloth diffusion models in the picker
The Images picker's On Device tab hid every non-GGUF cached repo whenever a
task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit
and FP8 safetensors) never showed up there. List cached repos that pass the
task gate, limited under a filter to unsloth-hosted ones so base repos (which
fail the diffusion load trust gate) don't appear only to dead-end on click.
Chat behavior is unchanged: the task gate still drops image repos there.
* Studio: hide single-file image checkpoints from the chat model picker
The chat picker treats a cached repo as an image model, and hides it, only
when it ships a diffusers model_index.json. Single-file, ComfyUI, and
ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a
Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models.
Fall back to resolving the repo id against the known diffusion families, the
same resolver the Images backend loads from, so these checkpoints are tagged
text-to-image and stay in the Images picker only.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: add the FLUX.2-dev model family
Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the
Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline)
resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline
+ Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated,
reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text
encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror.
text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for
dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from
klein's turbo defaults. Verified live: GGUF load resolves the family + gated base
repo and generates a real 1024x1024 image on GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio Images: clearer error for an unsupported diffusion model
When a repo id resolves to no diffusion family the load raised 'Could not infer a
diffusion family... Pass family_override (z-image)', which points at an unrelated
family and doesn't say what is supported. Replace it with a message that lists the
supported families (from a new supported_family_names helper) and notes that video
models and image models whose diffusers transformer has no single-file loader are
not supported. Applies to both the diffusers and native sd.cpp load paths. Also
refreshes two stale family-registry comments that still called FLUX.2-dev omitted.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove stray async task scratch outputs committed by mistake
* Diffusion: guard trust check against OSError and validate conditioning inputs
- _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid
characters (or a bare owner/name id) can't raise OSError; treat any failure as
not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request
still raises the clear FileNotFoundError for a genuinely missing local pick.
- generate(): reject mask_image / upscale / reference_images supplied without an
input image, and reject reference_images on a family that does not support
reference conditioning, instead of silently degrading to txt2img / img2img.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Codex review findings on the image-workflows PR
Keep diffusion.py importable without torch: the compile/arch patch modules
import torch at module level, so import them lazily at their load/unload
call sites instead of at module load. This restores the torchless contract
so get_diffusion_backend() works on a CPU/native sd.cpp install.
Match family reject keywords and aliases as whole path/name segments, not
raw substrings, so an unrelated word like edited, edition, or kontextual no
longer misroutes or hides a valid base image model, while supported edit
families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same
segment matching in the picker task filter.
Route FLUX.2-dev native guidance through --guidance like the other FLUX
families rather than --cfg-scale. Reject native upscale requests that have
no input image. Read image header dimensions and reject over-limit inputs
before decoding pixels, so a crafted small-payload image cannot spike
memory. Reject an upscale that would shrink the source below its input
size. Validate the model_kind against the filename extension before the
GPU handoff. Estimate a local diffusers pipeline's size from its on-disk
weights so auto memory planning does not skip offload and OOM. Report
workflows: [txt2img] from the native backend status so the Create tab
stays enabled for a loaded native model. Clamp the outpaint canvas to the
backend's 4096px decode limit.
Adds regression tests for segment matching and kind/extension validation.
* Address further Codex findings on the image-workflows PR
- Persist the actual output image size in the gallery recipe instead of the
request sliders: Transform/Inpaint/Edit derive the size from the uploaded
image, Extend grows the canvas, and Upscale resizes it, so the sliders
recorded (and later restored) the wrong dimensions for those workflows.
- Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file
name) in validate_load_request, so the unloadable pick fails before chat is
evicted rather than deep in from_pretrained.
- Only publish an image-conditioned from_pipe wrapper to the shared aux cache
when the load is still current: from_pipe runs under the generate lock but
not the state lock, so an unload racing its construction could otherwise
cache a wrapper over torn-down modules that a later load would reuse.
- Verify the Windows CUDA runtime archive checksum before extracting it, like
the main sd-cli archive, so a corrupt or tampered runtime is rejected rather
than extracted next to the binary.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Add customizable RAG embedding model setting and reorganize settings tabs
Chat with files, project sources, and knowledge bases previously always
embedded with unsloth/bge-small-en-v1.5. This adds a Settings option to
pick any Hugging Face embedding model (or local path), with HF search
autocomplete, server-side verification that the repo is actually an
embedding model, and a save anyway escape hatch for offline or local
models. The setting persists in app_settings and applies at runtime to
both the sentence-transformers and llama-server GGUF embedder backends
without a restart.
Also reorganizes the General settings tab: Documents & RAG sits above
Uploads, Helper LLM moved above the danger zone, and Model auto-switch
(OpenAI API) moved to the bottom of the API tab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Support local model paths on the GGUF embedder and normalize default saves
Found by simulation testing of the embedding model setting:
Local paths saved as the embedding model now work on the llama-server
GGUF backend (the default backend on macOS and CPU). A path to a .gguf
file is used directly and a directory is scanned for a variant-matching
non-mmproj .gguf, with a clear error when none exists. Previously a
local path was sent to the HF hub API and failed with a repo lookup
error.
Saving the default model explicitly no longer stores an override, so
is_custom stays false and the UI does not show a reset button for the
default value.
* Address review: stale-vector handling, GGUF derivation, save-time guards
Review follow-ups, each verified by new tests:
Re-uploading a document after an embedding model change now re-indexes
instead of deduping by content hash. Documents record the embedder that
produced their vectors (lazy embedding_model column, NULL legacy rows
keep deduping) and a mismatch replaces the old document.
A vector width change no longer bricks the dense index. ensure_vec
drops and recreates chunks_vec when the dim changes (old vectors are in
a foreign space and only block inserts) and search_dense returns empty
on a width mismatch instead of surfacing a vec0 error, so lexical
search keeps working until documents are re-uploaded.
Saving a local sentence-transformers folder with no .gguf now returns
409 with a clear message when the install embeds via llama-server,
instead of failing at first index. force still saves.
A custom RAG_EMBEDDING_MODEL env without RAG_EMBED_GGUF_REPO now
derives the -GGUF companion repo instead of silently keeping the bge
GGUF on CPU and macOS installs.
The resolved GGUF path is tagged with the repo captured at entry, so a
setting change during a download cannot mark the old model as current.
GGUF repo detection matches gguf as a whole name segment rather than a
substring, hf_token is trimmed before verification, and the settings
combobox drops a redundant state mirror of its controlled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shrink embedding model font to 11px in the input and dropdown
The combobox wrapper applies className to the outer input group, so the
size utility must target the inner input element; the previous text-xs
never reached it and the field rendered at the browser default.
* Show curated unsloth embedding models when the search field is empty
The empty-query listing was the global top-downloads page, which holds
no unsloth mirrors for the unsloth-first float to reorder, so the
dropdown opened on third-party models. Match the model picker: curated
unsloth listing when empty, whole-Hub search once a query is typed.
* Address review: settings resilience and index consistency
Keep the last known embedding model on settings store errors, remove the
re-entrant dim lock in the llama-server backend, accept local GGUF saves
and verify GGUF availability for HF repos on that backend, match local
path embedders exactly in model list filters, drop same-width stale
vectors from dense search, pin the embedder per ingestion job, and only
replace completed documents after the re-index succeeds.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate the GGUF repo derivation tests
* Trim to a single core embedding-model test
* Address review: GGUF repo saves and cache race
Accept a GGUF-named HF repo on the llama-server backend by verifying GGUF
availability instead of the sentence-transformers metadata gate, and guard
the settings cache with a generation counter so a read overlapping a save
cannot repopulate it with the pre-save value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Backend:
- Sanitize a blank hf_token to None in begin_load and load_pipeline, so the
default empty Studio token loads anonymously instead of 401ing as an explicit
empty credential.
- Free the ACTIVE diffusion engine before LLM training and in the delete-cached
guard: on a native (sd_cpp) selection the diffusers singleton reports
unloaded, so training could start against a live sd-cli generation and
delete-cached could remove a GGUF the native engine is using. Both now go
through diffusion_engine_router.get_active_diffusion_engine().
- Refuse delete-cached while a background image load is downloading the repo
(or its companion base): status().loaded is False in that window, but the
delete would yank blobs from under the in-flight assembly. Both engines
expose the in-flight ids via a new loading_repo_ids().
- Cap request seeds at 2**53-1: seeds round-trip through JSON gallery recipes,
where JavaScript rounds larger integers, so a restored recipe generated a
different image. Random seeds were already masked to this range.
- Add the task field to CachedModelRepo: the handler sets it for cached
diffusers image repos but response_model silently dropped it, letting
image-only repos pass the chat picker's task gate.
Frontend:
- Offset sequential run seeds by the batch size: the native engine seeds image
j of a run at seed+j, so a +1 run offset regenerated the previous run's
batch-mates.
- Revert the optimistic quant selection when a load fails to start.
- Stop disabling the Images page on chat-only hosts: the native sd.cpp engine
exists exactly for the no-GPU route.
* add models for /update endpoint
* add logic for identifying out of date hf models
* add endpoint for updating hf models
* add relevant field to GgufVariantDetail
* make exception handling better
* add update_available flag for cached_models, and moved /update endpoint from inference -> models
* hook up /update endpoint on the frontend
* implement update scenarios for the model picker
* fix bug where downloaded flag for an older revision was being wrongly set to false
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix import and make hf calls async
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove has_vision from UpdateRequest
* fix ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* clear cancel event before updating gguf variant
* set _cancel_event back if it was set initially
* add hf_token to get_paths_info
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: harden model update endpoint and update checks
- update_hf_model: pass snapshot_download local_dir (local_path is not a
valid kwarg and 500s when updating bicodec audio models)
- get_gguf_variants: wrap the remote update check so a network, rate-limit,
gated, or offline failure degrades to "no update info" instead of failing
the whole variant listing, matching list_cached_models
- add regression tests for both paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: HF model update detection and Update action for cached models
Surface an "Update available" cue and a managed Update action for cached
on-device models. /api/hub/update-status compares each cached main GGUF
file's local blobs against the remote main revision using set membership
across all cached revisions, so a repo that was already updated (and still
holds the old snapshot alongside the new one) is not falsely flagged.
The Update action re-downloads through the download manager so it shows in
the Downloads panel with progress and cancel. The frontend wires the Update
button into the GGUF, on-device, and model-selector cards and keeps the
quant label fully visible when the action buttons crowd the row.
Adds regression tests for the multi-revision update check.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: accept force_download kwarg in hf_xet_fallback test double
The download seam now passes force_download to the attempt callable; the _FakeAttempt mock did not accept it, failing 6 tests with TypeError. Add the keyword (default False) so the scripted-results double matches the seam.
* Fix Studio model update regressions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Studio update review feedback
* Address Studio update edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Share GGUF update status helper
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF update detection and cache cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix cached GGUF update badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: persistent per-user trust_remote_code approval cache
The consent gate pins each approval to a content fingerprint (sha256 over every
repo .py), but nothing was persisted, so the dialog reappeared on every fresh
load of the same unchanged repo. This adds an on-disk, per-user approval cache
that lets the gate skip the dialog when the same user reloads the same code,
while keeping the safety guarantees intact.
Two-tier validation, both must hold or the user is re-prompted:
- Commit SHA (cheap, one HfApi.model_info().sha, no download): a match means a
byte-identical tree to the approved revision, so the scan/download is skipped.
- Content fingerprint (authoritative): used whenever the SHA is unavailable
(local path / offline) and always recomputed on a SHA miss. A new or edited
.py changes both the SHA and the fingerprint, so it is caught in every mode.
Safety:
- Keyed per subject; one user's approval never auto-runs code for another.
- CRITICAL is never stored or honored (guarded on both write and read), so a
hand-edited store cannot smuggle in an auto-approval.
- The malware (HF unsafe-file) gate stays unconditional.
- Fail-safe: a corrupt store, an unresolvable SHA, or any error degrades to
"ask again", never to "auto-approve". UNSLOTH_TRC_APPROVAL_CACHE_DISABLE=1
turns the cache off entirely.
New module utils/security/remote_code_approvals.py holds the store
(studio_root()/security/remote_code_approvals.json, atomic write, 0600, RLock)
plus the SHA resolvers. Recording happens at the single gate chokepoint when the
caller supplies the matching fingerprint, so subject is just threaded through
inference/training/export (orchestrators, routes, workers). The scan endpoint
returns already_approved so the frontend can skip the dialog on a cache hit.
Tests: new tests/test_trc_approval_cache.py covers cache miss, SHA-match skip,
SHA-moved re-scan, new-file re-consent, CRITICAL never cached (write + forged
read), disable flag, subject isolation, combined adapter+base key, corrupt
store, and no-subject bypass. Full security suite: 101 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: make the approval cache skip only the prompt, never the scan
Codex found that the SHA "no-scan" fast path could run untrusted code without
re-consent. Removed it; the gate now always re-scans and the cache only seeds the
authoritative fingerprint check, so it can skip the dialog but never the scan.
- CRITICAL is hard-blocked on every load (the scan always runs), so a hand-edited
store that downgrades a CRITICAL repo's severity can no longer auto-run it
(P2: do not trust editable severity for SHA approvals).
- The fingerprint covers external auto_map repos, so changed third-party code
always re-prompts even when the primary commit SHA is unchanged; there is no
longer a SHA path that bypasses the fingerprint (P1: external auto_map repos).
- resolve_commit_sha is resolved fresh on every call (no memoization), so a repo
whose default branch moves after approval re-prompts instead of reusing a stale
cached SHA (P1: revalidate mutable Hub SHAs). The SHA is now only a conservative
secondary gate: a fresh resolvable SHA must match the approved revision, else the
seed is withheld; a None (local/offline) falls back to the fingerprint.
- Approvals record the scanner ruleset version (SCAN_RULES_VERSION); the gate
ignores approvals from an older ruleset so reclassified bytes are re-scanned and
re-shown instead of silently auto-approved (P2: invalidate on scan-policy change).
Tests: test_trc_approval_cache.py rewritten around the prompt-skip semantics
(unchanged repo still scans; SHA move / changed code / scanner-version bump /
disable flag all re-prompt; forged downgraded severity still blocks CRITICAL).
105 passed with test_consent_gate.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments to be more succinct
* Keep run-owner subject out of persisted config; serialize approval writes
Threading subject (the run owner's username / API-key id) into the training
config meant _sanitize_db_config persisted it into config_json, which
training-history GET returns to any authenticated user, leaking who started a run
in multi-user installs. Filter subject alongside the token fields; the worker
still receives it from the live config.
The approval store's RLock only guards one process, but approvals are recorded
from separate inference/export/training subprocesses, so concurrent writers could
clobber each other on os.replace and drop an approval (re-prompt). Hold a
best-effort cross-process file lock around the read-modify-write.
* Fail safe on a malformed approval store
A store with the right version but a non-dict shape (e.g. a hand-edited
"subjects": []) passed _load()'s check, then lookup chained .get() on a list and
raised, breaking every remote-code load until the file was removed. Validate that
subjects is a dict in _load(), and tolerate a non-dict per-subject entry in
lookup/record/forget, so a corrupt store fails safe (re-prompt) instead.
* Keep subject out of the MLX W&B run config
_run_mlx_training uploads the whole training config to W&B minus a sensitive set
that only listed hf_token/wandb_token/s3_config, so the authenticated subject
(username / API-key id) was sent to W&B as run config even though DB history
already strips it. Add subject to the W&B-sensitive filter, mirroring
training._sanitize_db_config.
* Tighten the W&B subject-filter comment
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: redesign Select model dropdown to match Hub design
Make the chat Select model picker easier to scan by reusing the Hub
on-device card's visual language.
- Rows now split owner/name, add a param chip, a DotTag format pill,
a tabular size, and a Loaded marker on the active model.
- Hub models / Fine-tuned tabs reuse the Hub's exact .hub-tab-toggle
styling (selectors extended in hub.css to the selector menu).
- Add a Downloaded / Recommended / Custom section toggle on the Hub
tab to filter the list.
- Widen the popover and nudge the scrollbar toward the edge.
* Studio: move section toggle below search, size tabs to label
Put Downloaded / Recommended / Custom under the search bar in their own
row so Hub models / Fine-tuned no longer wrap. The section toggle uses a
smaller font and sizes each tab to its label instead of equal widths.
* Studio: extract pure row-meta helpers into their own module
Move splitRepoLabel, classifyMetaToken, and parseMetaTokens out of
pickers.tsx into row-meta.ts. No behaviour change; keeps the presentation
logic free of React/DOM deps so it is easy to test in isolation.
* Studio: content-size the source tabs and add section icons
Size the Hub models / Fine-tuned tabs to their labels (with side
padding) like the section toggle, instead of stretching full width. Add
a leading download, star, and folder icon to Downloaded, Recommended,
and Custom.
* Studio: stop source tabs stretching and hide empty Fine-tuned tab
The popover is a flex column, so the fit toggle stretched full width;
add w-fit/self-start so it sizes to its content. Also hide the
Fine-tuned tab when there are no fine-tuned models, defaulting to Hub
models.
* Studio: keep only fine-tuned models in the Fine-tuned tab
Local models (LM Studio, Ollama, custom folders) carry source "local"
and already show in the Hub tab's Downloaded / Custom sections, so
exclude them from the Fine-tuned tab and from its visibility count.
Extract the tab rules into source-tabs.ts.
* Studio: show local providers under Downloaded, Recommended first
Show LM Studio and other local provider models in the Downloaded
section in all modes (was chat-only). Put Recommended first and make it
the default section. Add a little more space below the search bar.
* Studio: make Recommended a sortable live Unsloth listing
Replace the static Recommended list (and its collapse chevron) with a
sort dropdown over Unsloth's own models: Recommended, Trending, Most
likes, Downloads, Recently updated. Recommended shows recently uploaded
GGUF/MLX models that fit the device (hidden if they do not); the other
sorts list all Unsloth models, badged but never hidden. Adds a sort
option to useHfModelSearch and a pure recommended-fit helper.
* Studio: size Recommended models from the repo name when metadata is missing
GGUF and MLX repos rarely expose safetensors metadata, so a large model
with no size could pass the Recommended fit check because unknown size was
treated as fitting. Parse the parameter count from the repo id, including
the Gemma E series, and hide anything we still cannot size.
* Studio: detect model capabilities and family from HF tags
Thread tags and the pipeline tag through the model search results and add a
pure helper that infers vision, reasoning and audio plus the architecture
family, falling back to repo-name keywords when tags are absent.
* Studio: add row details and inline section sorting to Select model
Give each model row more detail and make the Hub sections easier to scan:
- Show vision, reasoning and audio badges plus the architecture family tag
on each row, alongside the params, format and size.
- Drop the redundant unsloth/ prefix on the Recommended rows.
- Rename the Recommended section tab to Unsloth and enlarge the section tabs.
- Move the sort dropdown inline to the right of the tabs at a fixed width.
- Add Recent, Size and Downloaded sorting to the Downloaded and Custom tabs.
- Remove the header icons, pad the subheadings, and grow the list height.
* Studio: tune the Select model sort dropdown and trim row badges
- Recommended now lists the most recently created Unsloth repos.
- Narrow the sort dropdown, remove its border, and truncate long labels.
- Tighten the gap between the section tab icons and their labels.
- Remove the architecture family tag from rows since it repeats the name.
* Studio: extract the PillTabs toggle into a shared module
Move the segmented pill toggle out of the model selector into its own file so
the Hub picker can reuse it for a format filter without duplicating the markup.
* Studio: fix Recommended infinite scroll and add a format filter
- Re-attach the scroll observer on each loaded page so a filtered Recommended
list keeps paging until the viewport fills instead of spinning forever with
nothing new appearing.
- Add an All / GGUF / MLX / Safetensors toggle on the Unsloth listing that
filters every sort.
* Studio: default Recommended to Trending, rename Downloaded to On Device, and fade the scroll edge
Sort: default the Recommended view to Trending and add a Name option to
the On Device / Custom sort. Recent now orders by last load time while
Downloaded orders by file date, tracked in localStorage (model-usage.ts).
Formats: show the format filter on all three tabs (Unsloth, On Device,
Custom), exclude mobile GGUF builds from Recommended, and flag GGUF rows
that exceed the device with the same OOM badge as safetensors.
Polish: download-icon badge on already-downloaded Recommended rows, the
hugeicons view stroke-rounded vision badge, Search all models placeholder,
matched popover padding, and a top-edge mask fade once the list scrolls.
* Studio: size GGUF repos from gguf metadata so large ones flag OOM
Repos with no <n>B token in the name (Kimi, MiniMax) had no param count
and so never showed an OOM badge. Request the gguf expand field from
Hugging Face and read gguf.total, so those repos get a param chip and an
OOM badge when they exceed the device budget.
Keep the row name full contrast when over budget (the OOM badge already
signals the fit), shorten the format and sort dropdowns, narrow the
popover, and rename Recently updated to Recent and All formats to All.
* Studio: address selector review feedback
Add WAI-ARIA roving tabindex and Arrow Left/Right navigation to the pill
toggle so only the active tab is in the tab order. Keep the chat-only
GGUF/MLX filter for every Recommended sort, not just Recommended, so
chat-only users do not see unrunnable checkpoints under Trending. Feed
both listings' GGUF hints into repo detection so a tag-only GGUF in
Recommended expands variants instead of loading as a checkpoint.
* Studio: scope Select model search per tab and add an MLX tag
Search is now per section. The Unsloth tab searches the Unsloth HF
listing only, On Device filters downloaded and LM Studio models by name,
and Custom filters custom-folder models, each with its own empty state.
MLX repos get an MLX pill mirroring the GGUF tag. Downloaded quants in
the Unsloth and search lists get the same delete action as On Device.
Also: revert the model name to normal weight, narrow the popover to
558px so the format and sort dropdowns sit one gap-2 from the tabs,
tighten the dropdown menus to match the Projects activity Select, and
make the empty On Device state name the active format filter.
* Studio: show local ./models on the On Device tab so they stay selectable
Models under the local models directory (source models_dir) flow in as local
models but were dropped from every list: filtered out of Fine-tuned and never
re-added by the Hub picker, which kept only LM Studio and custom-folder
sources. Capture them in the local refresh and render a Local models group on
the On Device tab, with the same format, search, and chat-only GGUF rules as
the other local groups.
* Studio: add a Hub button beside the Select model search bar
Adds a Hub button next to the search bar that opens the full Hub Discover
page to browse more models. Styled like the section tabs (rounded, no
border, soft shadow with a faint top layer) and darkens on hover. Also
nudges the format and sort dropdown chevrons a touch toward the edge.
* Studio: align Select model padding and tighten the format pills
Sizes the popover to the tab cluster so the left and right padding match,
and drops the top row below the rounded corner so the Hub button lines up
with the Trending dropdown. Gives the Hub button a fixed width, lets the
list scrollbar sit inside the box, and shrinks the format pill dot with a
tighter dot-to-label gap.
* Studio: label the Hub button Search Hub and match the dropdown width
Renames the button to Search Hub, sets its width to the format and sort
dropdown width so it lines up above them, and tightens the icon gap.
* Studio: drop the vision and reasoning row badges to declutter
Removes the vision and reasoning capability icons from the model rows so
they read cleaner. Audio is kept.
* Studio: add a safetensors pill, hide diffusion models, eye on Vision
Gives safetensors rows a format pill and size so their meta matches GGUF
and MLX, drops image and video diffusion models from the listing since they
cannot run in chat, and shows an eye icon next to the Vision tag. Also
removes the em dashes from the Projects export and import labels.
* Studio: gate recommended folders on real weights and polish the selector
Only show a Recommended chip once the well-known dir actually holds
weights, so an empty LM Studio or Ollama scaffold no longer suggests
itself. _dir_has_downloaded_model checks for a GGUF/safetensors file or
a non-empty Ollama manifests store, with a bounded walk.
Selector polish: round the popover and option menus a touch more,
lighten the OOM badge in dark mode, soften the inner dropdown shadow,
even out the padding, and lift the toggle track and field triggers so
their edges read against the popover.
Also catch CogVideoX in the diffusion name fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align the dark Select model panel with the sidebar
Match the popover, fields, dropdowns, tab toggle and row states to the
sidebar surface and accent so the dropdown reads as one piece in dark
mode. The active tab pill and Search Hub button sit a touch lighter
than the track, and the inner option menus drop their drop shadow for a
flatter look. Light mode is unchanged.
* Studio: re-derive the Select model tab on open
The picker remounts each time the dropdown opens, but the source tab
state did not, so a persisted fine-tuned or connected selection that
only lands in its list after an async load would reopen on Hub. Reset
the active tab to the selection-derived default on the open edge, while
still letting the user switch tabs freely within a session.
* Studio: fold Custom into On Device and polish the picker
Merge the Custom tab into On Device so custom folders sit right below
the downloaded models, with a folder shortcut on the group header.
Rename the first Hub tab to Recommended, give the format dropdown
colored dots, even out the tab row spacing, and tighten the popover
width. Align the folder browser with the app dialogs (soft surface,
roomier padding, green confirm, grey hover).
* Studio: fix On Device controls and nudge the folder browser close
The Hub redesign merge dropped the old Search Hub button styling, so the
On Device search row rendered flat. Point the search input and Search
Hub button at the shared .field-soft surface so they match the rest of
the Hub controls, and lift the folder browser close button slightly.
* Studio: run the Select model search on the Hub search stack
Point the picker at the Hub's useHubModelSearch and useHubInfiniteScroll
instead of its own useHfModelSearch/useInfiniteScroll, scoped to unsloth
so the listing matches the old one. Both the search and the recommended
feed now share the Hub implementation, so there is one search path. The
Hub result folds GGUF params into totalParams, so the dead ggufParams
fallback is dropped.
* Studio: trim the recommended sort to Recommended, Trending, Recent
Drop Downloads and Most likes from the sort dropdown.
* Studio: give the section tabs room off the rounded edge
The fit-mode toggle wrapped the tabs with no inset, so On Device sat
tight against the rounded-full edge. Add a small horizontal inset and
widen the popover a touch to fit it.
* Studio: drop the legacy HF search hooks for the Hub ones
Migrate the training model and dataset sections, export page, onboarding
steps and recipe dataset combobox off useHfModelSearch, useHfDatasetSearch
and useInfiniteScroll onto the Hub equivalents, scoped to unsloth so the
listings match. The picker reads recommended param counts off the search
results it already has instead of a separate fetch. Removes the duplicate
search stack: use-hf-model-search, use-hf-dataset-search,
use-hf-paginated-search, use-infinite-scroll, use-recommended-model-vram
and the old lib/hf-cache.
* Fix model selector section toggle proportions
Remove the fit-mode track inset so the active pill sits flush to the
track edge, matching the Hub's segmented controls.
* Tighten model selector width and tab padding
Reduce the popover width so the right edge aligns with the row, and
widen the fit-mode tab padding so On Device clears the track edge.
* Refine Recommended formats, sort width and tab padding
Recommended now suggests GGUF anywhere and MLX only on Mac, never
safetensors. Size the sort dropdown to its label so Recommended no
longer truncates, and match the On Device trailing gap to the active
pill's leading inset.
* Flush section toggle and match dropdown font to Search Hub
Drop the trailing track pad so the active pill fits the track exactly
at either end. Size the sort and format dropdown text to text-xs like
the Search Hub button, and clip long labels without an ellipsis.
* Fix sort menu checkmark overlap and lock dropdown widths
Keep the option's right padding so the selected checkmark no longer
overlaps the label, and let the open menu expand to fit it. Set the
format and sort triggers to a fixed width matching the Search Hub
button so they always line up.
* Keep section toggle and dropdowns on one row
Drop the wrap and size the Search Hub button, format and sort dropdowns
to a shared 100px so they stay equal width and fit on one row without
widening the box.
* Studio: pre-load inference settings dialog with native context
Add a gear on downloaded GGUF quant rows that opens a settings dialog
to adjust inference parameters before loading a model:
- Context length, KV cache dtype, speculative decoding and tensor
parallelism, all written to the runtime store the load call reads.
- Settings can be remembered per model in localStorage.
- The context slider ceiling and "Model supports up to N tokens" come
from the model's native context, read from GGUF metadata and returned
by /api/models/gguf-variants once a variant is downloaded.
Also drop models Studio can't run for chat (diffusion, image, video)
from the recommended feed and Hub search, plus minor selector polish
on row hover padding, Search Hub and dropdown widths, and tab spacing.
* Studio: model selector polish and memory-aware load warning
Search and listing:
- Drop the "Recommended" and "Hugging Face" section labels while
searching so results read as one list; keep the format and sort
dropdowns visible so search results can still be sorted and filtered.
- Request gguf metadata in the Hub listing so GGUF repos report a
parameter count, restoring the OOM badge for repos without a size
token in the name (Kimi, MiniMax, GLM).
Load settings dialog:
- Warn when weights plus the KV cache at the chosen context exceed
available memory. The KV size is sized by the backend's
architecture-aware estimator via a new kv-cache-estimate endpoint;
the budget uses VRAM plus system RAM. Best-effort, no warning on
failure or on auto context.
- Context Length placeholder reads "auto"; dark background slightly
lighter.
Other:
- Clicking the Custom Folders header opens the folder browser; its
title now reads "Select folder to detect models".
- On Device sort lists Downloaded last.
- Smaller chat template editor font; rounded wrapper clips the prompt
and template editor scrollbars so the right corners stay round.
* Studio: fix load dialog memory warning budget and KV dropdown width
- The memory warning never fired without a discrete GPU. useGpuInfo
returned zero system RAM in that case, so the budget was always zero.
Surface system RAM even when no GPU is present (Mac unified memory),
and have the load dialog read memory directly instead of through props.
- Give the dialog fields shrink-0 so the KV Cache Dtype value (e.g.
q8_0) is not squeezed and clipped by the row.
* Studio: fold fine-tuned models into On Device tab
Remove the Hub models and Fine-tuned source tabs. Fine-tuned models now
show as a section in the Hub tab's On Device view, above Custom Folders,
with the Train icon and a collapse toggle. The section only appears when
the user has fine-tuned models. With no external providers the lone Hub
tab hides its own toggle.
Also: tick-circle Show hidden checkbox and drop the divider above Eject;
keep run settings load params (KV cache dtype, speculative, tensor
parallel) from being clobbered by a mid-load status poll.
* Studio: stage load settings in the sidebar with a Load on selection toggle
Replace the pre-load settings popup with a staging flow in the Run settings
sidebar. The gear on a downloaded quant row now stages the model and opens
Run settings with Load model and Cancel buttons, so options like context
length, KV cache, speculative decoding and tensor parallelism are set before
the model loads. A "Remember these settings" tick reuses them next time.
Add a global Load on selection toggle in Settings, Chat tab (default on).
On: Unsloth auto-picks the best settings for your hardware and loads on
selection. Off: picking a model stages it in Run settings to customize first.
The gear always stages, regardless of the toggle.
Other polish in this change:
- Fine-tuned models live under the On Device tab, with a train icon on the
header that jumps to the Fine-tuned section.
- Default to the On Device tab when downloads exist, otherwise the last used
section.
- Standard Unsloth tooltips on the train, folder and gear icons.
- Request the gguf param count on every Hub listing fetch so Kimi, MiniMax
and GLM show a size badge.
- Search Hub hover state, scrollbar position and minor spacing fixes.
Remove the old inference load settings dialog.
* Studio: always show the fine-tuned shortcut and smooth out the picker
- Fine-tuned section and its train shortcut now always show on On Device,
with an empty state when no fine-tuned models exist yet.
- Folder icon on the header jumps to Custom Folders instead of opening the
browse popup, matching the train shortcut.
- Folder browser keeps the list mounted and dims it while refetching, so
toggling Show hidden or changing folders no longer flashes.
- Drop the tooltip hover grace area in the picker so moving between the
train, folder and gear icons switches the tooltip at once.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add quantization display options and drop the fine-tuned empty text
- Settings, Chat: 'Expand quantizations' toggle. On expands every On Device
GGUF model's quantizations by default; off keeps them behind a click
(default).
- Settings, Chat: 'Show all quantizations' toggle. On lists every quant
including ones not downloaded (default); off shows downloaded only.
- Remove the empty-state line under the Fine-tuned header; the header still
shows on its own.
* Studio: let expanded quantizations collapse on click and split the On/Off help
- With Expand quantizations on, clicking an On Device model now collapses or
re-expands its quantizations. The collapse state is in memory only, so it
resets on reload and when the setting is toggled.
- Put the Off sentence on its own line in the quantization setting descriptions.
* Studio: reorder chat settings and rename the model section
- Rename the Models section to Select model settings and move it above the
Chat menu section.
- Trim the section and Load on selection descriptions.
* Studio: tighten the On/Off lines in the model setting descriptions
Use a line break instead of separate spans so the On and Off lines sit on
consecutive lines without the extra paragraph gap.
* Studio: top-align the Load on selection toggle
Add an alignTop option to SettingsRow and use it so the toggle sits at the top
of the row next to the label, not centered against the tall description.
* Studio: put the gear hint and example chip on one line
Move the gear example chip inline with its label so it reads as a single line
instead of wrapping onto its own row.
* Studio: move the New badge from API keys to Chat settings
Add the New badge to the Chat settings tab and drop it from API keys.
* Studio: line the Load on selection toggle up with the first description line
Offset the top-aligned control past the label row so it sits next to the On
line instead of the label.
* Studio: label the chat menu item Chat with Files (RAG)
Rename the Chat with Files entry in the chat menu settings to clarify it is RAG.
* Studio: drop the pill around the gear example so it fits on one line
Remove the background and padding from the gear example chip so it sits inline
with its label at a lower height.
* Studio: fold the gear example into the description line spacing
Render the gear example inline in the same text block so its line spacing
matches the On and Off lines instead of an extra flex gap.
* Studio: scope Show all quantizations to On Device only
Gate the downloaded-only filter on an onDevice flag so Recommended and other
browse lists always show every quant, and note On Device in the setting copy.
* Studio: tidy On Device GGUF rows
- Drop the redundant Quantizations subheading under On Device models.
- Relay GGUF vision support up to the model name as a Vision badge instead.
- Drop the repo size from On Device GGUF model rows since the quants already
show their size.
* Studio: pin the eject button and tidy General settings
- Move Eject loaded model out of the scrollable list into a centered footer so
it stays in view no matter how far the list is scrolled.
- Space out and center the gear example in the Load on selection description.
- General: drop the duplicate Unsloth version section, move llama.cpp
notifications above Helper LLM, and note new models in its description.
* Studio: add left padding before the gear example
Nudge the gear example away from its label with a small left margin.
* Studio: make the eject footer a sticky bar over the list
Pin Eject loaded model to the bottom of the scroll area with the menu
background so rows scroll under it, and drop the divider line.
* Studio: drop the eject footer background, keep it a sticky button
Make the sticky eject a centered transparent button so it coexists with the
rows scrolling behind it. The wrapper ignores pointer events so only the button
is clickable.
* Studio: give the eject button a solid background
Add the menu background, a border and a soft shadow to the sticky eject button
so it reads as a floating button over the list.
* Studio: restore the eject footer block, keep hover on the button only
Bring back the full-width menu background behind the sticky eject footer, but
keep the button compact and centered so the hover stays on the button.
* Studio: show the vision badge on On Device rows without expanding
- cached-gguf listing reports has_vision (mmproj present), so the badge shows
on the model name without opening the quantizations.
- Make the vision badge icon-only with a tooltip: "This model can process
image inputs". Falls back to the expander-reported value on older backends.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make LM Studio and Local models sections collapsible
* Fade the eject footer instead of a solid block
* Wrap the vision badge in a bordered pill
* Taller model list with the eject footer pinned to the bottom
* Use purple for the vision badge to set it apart from GGUF
* Reduce the model list height
* Make the eject button inline with no background block
* Match the vision badge color to the Hub indigo tone
* Shorten the model list and square off the format tags
* Pin the eject button so it floats at the bottom of the list
* Give the floating eject button a tinted background
* Add bottom clearance so the list ends on white space under the eject button
* Match eject button to the menu background and unify the settings gear icon
* Move eject below the list and match its shadow and dark background
* Drop the min height so short model lists leave no white space
* Remove the eject button fill so it never covers the list
* Nest dropdown hover radius inside the menu corners
* Float the eject pill again and fix sort dropdown hover radius
* Make the eject button opaque in both themes on hover and dark
* Trim the model menu bottom padding so it stops clipping the last row
* Match dark eject background to the Search Hub button and pad row indicators
* Fade the model list bottom edge while rows sit below the fold
* Lift the eject button and trim the section toggle right padding
* Nudge the model list taller and run the bottom fade to the box edge
* Nudge the model list slightly taller
* Remove the eject button shadow
* Align the eject button to the right
* Widen the Search Hub and dropdowns and right-align them
* Seat the eject button at the base and restore On Device right padding
* Reduce the Search Hub and dropdown width by 4px
* Widen the model menu so the section toggle keeps its padding
* Make the eject button an icon-only button with shadow
* Tighten section tab padding to cut the grey between tabs
* Revert section tab padding back to px-3
* Remove the section toggle trailing padding
* Add an eject button beside the model selector trigger
* Shrink the in-list eject button to a smaller proportional size
* Raise the in-list eject button
* Make the trigger eject a bare icon next to the dropdown arrow
* Revert eject back to the labeled button on the right
* Place the format and sort dropdowns next to the section toggle
* Raise the eject button and shorten its label to Eject model
* Widen the gap between the toggle and dropdowns slightly
* Align Search Hub with the last dropdown via a shared-width grid
* Narrow the model menu for symmetric padding
* Stretch the search row so Search Hub lines up with the last dropdown
* Inset the list so the right padding matches the left
* Right-align dropdowns and full-width search so Search Hub meets the last dropdown
* Pack section toggle and dropdowns with a uniform gap
* Inset search row so Search Hub aligns with the Trending dropdown
* Trim model menu right padding to match the left
* Nudge model list scrollbar inward
* Move eject button to the bottom left with a light shadow
* Shorten show all quantizations description
* Keep eject button right-aligned, nudged in from the edge
* Move Connected into the section toggle as a cloud-icon tab
* Align eject button with the format tag edge
* Right-align Connected layout so Search Hub meets Trending
* Download selected models through the Hub download manager
* Add Other models section for non-Unsloth downloads
* Add directions icon and shortcut for Other models section
* Space out subheadings and gate Other models on non-Unsloth downloads
* Use direction-right icon for Other models
* Use flag icon for Other models
* Widen Connected menu so dropdowns align with Search Hub
* Model selector: truncate long quant labels and tidy layout
- Hub GGUF card: truncate long file-path quant labels with an ellipsis
instead of overflowing the row.
- Connected layout: left-pack the dropdowns and size the box so the last
dropdown's right gap matches the pill's left gap, with Search Hub on its edge.
- On Device: show MLX/Safetensors with the size on non-GGUF rows.
- Connected list rows use the same grey hover as the tabs; the selected
section tab no longer shows a hover change.
* Model selector: drop stale custom section on restore
A persisted custom section value no longer maps to a tab, so restoring it
opened the picker to an empty view. Fall back to recommended instead.
* Model selector: align the non-connected search bar with the All dropdown
Nudge the non-connected box width so the search bar's right edge meets the
All dropdown, which lands Search Hub on the last dropdown's edge.
* Studio chat model selector: remember last tab, route non-GGUF downloads through Hub, stack overlays
- Restore the last Hub section (Recommended / On Device) on every open instead of always snapping to On Device when downloads exist.
- Route uncached non-GGUF repos (safetensors / MLX) through the Hub download manager via a snapshot download, so every model download shows in the bottom-right indicator and follows Load on selection like GGUF.
- Allow safetensors in Recommended on Mac (they run locally there now), and honor the Safetensors format filter instead of dropping it via the recommendation default.
- Stack bottom-right overlays in one column so the download panel and banners never overlap.
- Add evenly spaced divider lines between the On Device subheadings.
- Pad the bottom of the list so the floating Eject pill never covers the last row.
* Studio downloads panel: widen left padding on header and rows
Bump the left inset to pl-4 while keeping pr-3 so the collapse and cancel buttons stay put.
* Studio: update cached-gguf route tests for the has_vision field
list_cached_gguf now returns has_vision per row (vision badge on On Device);
the expected dicts were missing it. True for the mmproj vision repo, False elsewhere.
* Studio: keep MLX/safetensors selectable in chat-only Mac search
The empty Recommended view allows GGUF plus MLX/safetensors on Mac, but the
curated and HF search lists dropped non-GGUF in chat-only via a GGUF-only filter,
so typing a query hid runnable Mac models. Reuse isRecommendableFormat in both
lists so search matches the empty view (chat-only non-Mac stays GGUF-only).
* Model selector: restore global model search and fix GGUF/device-fit regressions
- Search: training, export and onboarding pickers searched only the unsloth org
on a typed query. Restore the prior behavior (global Hub search with unsloth
floated first when a query is typed, curated unsloth listing when empty).
- Recommended browse: the GGUF/MLX-only gate ran before the format filter, so
the Safetensors filter and the Trending/Recent sorts always came back empty.
Apply that gate only for the Recommended sort and chat-only mode.
- GGUF metadata: request the gguf expand field through listModels so repos with
no size token in the name (Kimi, MiniMax, GLM) report a param count for the
size and OOM badge.
- Local GGUF: custom-folder and standalone ./models/*.gguf files now load
directly with the GGUF marker instead of dead-ending in the variant expander,
and scanned GGUF folders are classified via a backend model_format hint.
- Device fit: use system RAM in the budget on unified-memory hosts, and keep MLX
rows selectable on chat-only Macs.
- kv-cache-estimate: resolve the quant from the snapshot-relative path, skip MTP
drafter files, and prefer the most complete snapshot (mirrors the variant
scanner). Bound the Ollama manifest walk.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Model selector: classify suffixless local GGUF folders consistently
Complete the model_format plumbing so a GGUF folder is detected and loaded
through the same GGUF path that the format filter already uses:
- _scan_models_dir: a config.json no longer disqualifies a folder whose only
weights are .gguf, so HF GGUF repos shipping a config still classify as GGUF.
- _scan_lmstudio_dir: emit model_format for every GGUF row (LM Studio dirs
rarely carry a -GGUF suffix), via a shared _dir_model_format helper.
- Custom Folders and LM Studio rows: use localModelIsGguf (the same helper the
filter uses) so the row label, expand-vs-direct-load, and isGguf flag agree;
a suffixless GGUF folder no longer filters as GGUF but loads as non-GGUF.
Adds tests/test_local_model_format.py covering the classification rule.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio model selector: tighten section spacing
Trim each subheading's gap to its rows (pb-1.5 to pb-1) and pull the On Device
heading block tight to the controls while Recommended keeps a little top room.
* Hub: format filter fix, sort defaults, avatar and layout polish
- Format dropdown now filters the feed's Latest list too, so the default
GGUF hides fp8/safetensors and picking a format changes the rows.
- Latest Unsloth Models sorts by newest created, not recently updated.
- Sort dropdown order: Newest, Trending, Most downloads, Recently
updated, Most likes.
- Unsloth uploads with no upstream provider logo show the Unsloth avatar
instead of a colored initial.
- Owner scope pill gets a little more room before the chevron.
- README detail column lines up with the top bar (both-edges gutter).
- Long file-path quant labels truncate instead of overflowing the row.
- Model list keyboard nav no longer clips the focus ring.
- Run settings sheet: restore the Remember settings toggle and larger
Load/Cancel buttons on the staged load flow.
* Hub: hide the RAG embedding model from browse previews
The Hub discover feed and chat model selector pull from the Hugging Face
listing on the client, which the backend _is_hidden_model filter never
touches, so the RAG embedder (unsloth/bge-small-en-v1.5-GGUF) and the
llama.cpp validation probe leaked into the lists.
Added isHiddenModelId mirroring the backend needles and filtered it out of
the discover rows, the trending feed, and the selector's recommended and
Hugging Face search lists. Per-repo file and download views are untouched,
so the model is never deleted and a reinstall still shows it as already
downloaded.
* Studio: skip hidden dirs when checking a folder for downloaded models
_dir_has_downloaded_model walked the tree with rglob("*") bounded by
max_entries. rglob yields entries in arbitrary order and counts every one, so a
model directory that also holds a large hidden subtree (.git/.cache/venv) could
exhaust the budget before reaching the real weights and falsely report no model,
hiding a valid Recommended-folder chip. Replace the generic-weights pass with a
bounded BFS that skips hidden directories so their entries can't starve the walk.
Adds a regression test (50-entry .git beside the weights, max_entries=10).
* Fix/adjust model selector handling for PR #6364
* Studio: address codex review on the staging/recommended-folder paths
- chat-page auto-load: selectModel only clears pendingSelection on success, so a
failed auto-load left the hidden stage (and its edited load knobs) behind.
Abandon the stage when it still matches the failed pick.
- model picker: count fine-tuned rows in the On Device empty check so a
fine-tuned-only tab no longer shows a false 'No models on device' message
above the Fine-tuned section.
- general settings: add the remembered per-model load settings key to PREFS_KEYS
so 'Reset all local preferences' actually clears it.
- recommended-folders: recognize PyTorch .bin weights (gated by the scanner's
weight-name prefixes) so a .bin-only model folder still earns a chip; add tests.
* Studio: name-gate .bin weight detection and complete selector preference reset
Follow-up to the codex review on the model_format/recommended-folder paths:
- _dir_model_format and _scan_models_dir treated any .bin (incl. tokenizer.bin)
as a non-GGUF weight, so a suffixless GGUF folder shipping a companion .bin was
misclassified as a plain checkpoint and routed through the wrong load path.
Factor the scanner's weight-name gating into shared _is_weight_bin /
_has_non_gguf_weights helpers and use them everywhere (also in
_dir_has_downloaded_model).
- PREFS_KEYS was missing the new 'Select model settings' keys (load on selection,
expand/show-all quantizations), so 'Reset all local preferences' left them set.
- On Device cached search dropped the active format filter while a query was
typed; keep matchesFormatFilter applied so the format dropdown stays consistent.
Adds tests for the tokenizer.bin vs weight-.bin classification.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: validate Ollama blobs, gate staged context, honor RAM budget on no-GPU hosts
- recommended-folders: only count an Ollama dir once its manifest resolves to an
on-disk model blob, so a failed/pruned pull no longer surfaces an empty chip
- GGUF variant click: only seed the staged contextLength for already-downloaded
picks, so choosing an undownloaded quant from a partially cached repo still
starts its download (the staging effect short-circuits on a known context)
- device fit: classify GGUF variants against the system-RAM budget on no-GPU /
unified-memory hosts instead of reporting everything as fits, and pass
systemRamGb to every variant expander regardless of gpu.available
* Studio: scope Hub search to Recommended, fix staged non-GGUF settings, keep local MLX on Mac
- model picker: only run the Hub search hooks on the Recommended section. On
Device / Connected render local data, so typing there no longer fires HF
requests or a spinner and the local/offline flow is preserved
- chat settings: when a pick is staged, decide the GGUF-only controls from the
staged model's type, not the currently loaded model's. A staged non-GGUF Hub
repo no longer inherits a loaded GGUF's context/KV/speculative controls
- On Device: keep local MLX builds in ./models selectable on Mac (chat-only ran
GGUF/MLX only, but the filter dropped MLX before the format toggle)
---------
Co-authored-by: shimmyshimmer <info@unsloth.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Show model provider/org in the trust_remote_code consent dialog
The consent dialog showed only the trailing model name (modelName.split('/').pop()),
dropping the owner. The HF org/owner is the 'who do I trust' signal the prompt is
asking about, so render it: 'NVIDIA-Nemotron-3-Nano-4B from "unsloth"'. A null
provider for local paths and bare names leaves those renders unchanged. Applies to
the enable/blocked/malware variants (shared description block).
* Only show consent provider tag for a confident single Hub repo
Tighten parseModelDisplay so the 'from "<provider>"' tag is shown only for a
canonical owner/repo Hub id (exactly one slash, both segments non-empty) that is
not a local path and not part of a multi-repo scan. This avoids misattributing a
relative local directory name (models/llama/7b) or a LoRA base/external repo's
finding to the wrong publisher in a trust decision. Extract a ProviderSuffix
component so both description branches render the clause identically via &&.
* Tighten consent provider-tag comments
* Source the consent provider tag from the backend
The dialog inferred the provider client-side from the model id, using
scanCreatedRepos (a cleanup-only list) to detect multi-repo scope and a
regex that missed bare relative paths like a local owner/model dir. Both
could attribute the scanned code to the wrong publisher.
Move the decision to the backend, where locality and scan scope are
known: _consent_provider returns the owner only for a single, non-local,
canonical owner/repo Hub id, and the route returns it as payload[provider].
The frontend now renders scan.provider directly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Suppress consent provider tag when external auto_map code is scanned
A single Hub repo can declare an auto_map that loads code from another repo
(owner/other--module.Class). The scanner fingerprints that external repo's
Python, but security_targets still held only the primary, so the dialog
attributed the custom code to the primary publisher. Pass the external refs
collected during the scan to _consent_provider and return no provider when any
are present, so attribution is shown only for genuinely self-contained repos.
* Trim comments to be more succinct
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: scale export GGUF size estimates from the real model size
The Export page showed hardcoded, model-independent GGUF quant size
labels (Q8_0 ~8.2 GB, BF16 ~14.2 GB, ...) calibrated for an ~8B model.
For a 35B MoE model like Qwen3.6-35B-A3B (67 GiB bf16, Q8 ~34 GiB) the
picker wrongly reported Q8 ~8.2 GB. Only the displayed estimate was
wrong; the actual export via save_pretrained_gguf was always correct.
Add GET /api/models/export-size, which returns a model's MoE-aware
fp16/bf16-equivalent size and total params using the existing
estimate_fp16_model_size_bytes (safetensors -> config -> local -> vllm).
The result is memoized and degrades to nulls so a size hint can never
break the Export page.
The Export picker now scales each quant from that size
(bytes ~= fp16_bytes * bits_per_weight / 16, GiB units to match the
model selector), and renders no size when it is unknown rather than a
misleading fixed number. The Est. size summary in the page and dialog
is restored now that the value comes from the backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export-size: address review feedback
- Run the size estimate off the event loop with asyncio.to_thread so a slow
Hugging Face request cannot stall other API or SSE endpoints.
- Cache only successful estimates; a transient failure (offline, gated before
credentials) is no longer pinned as unavailable until restart.
- Forward the HF token so private and gated models can be sized, and refetch
when the token changes.
- Clamp the size formatter index so sub-1-byte values cannot pick an
out-of-range unit.
* Studio export-size: address second review pass
- Send the HF token in an X-HF-Token header instead of the query string, so
it never lands in URLs, logs, or browser history.
- Key the estimate cache by model id only (the fp16 size is token independent),
so HF tokens are never retained in the cache.
- Restrict local-path sizing to known Studio roots (outputs/exports/cache/home)
so an authenticated caller cannot trigger a scan of an arbitrary directory.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export-size: fix CI (import-hoist + isolated-load test stubs)
- Import ExportSizeResponse from models.models in routes/models.py instead of
re-exporting it through models/__init__.py, so the import-hoist lint does not
flag a newly added but un-loaded re-export (models/__init__.py is unchanged).
- Add Header and ExportSizeResponse to the stubbed fastapi / models.models in
test_export_absolute_paths.py, which loads routes/models.py in isolation.
* Studio: validate export-size local path before filesystem access
CodeQL flagged the export-size local-path guard as path injection: the
user-provided model path was resolved and stat-ed before it was checked
for containment under a Studio data root. Decide containment by lexical
normalization (normpath/abspath/expanduser, no filesystem access) and
only touch the filesystem once the path is proven to sit under a trusted
root, so an unvalidated value never reaches a filesystem call. Add a
direct containment unit test (under-root, root itself, missing, /etc,
and '..' traversal).
* Studio: trim export-size comments to be more concise
Shorten docstrings and comments on the export-size endpoint, helpers, tests,
and frontend size utilities; drop comments that just restate the code. Verified
code-identical (comments only) via AST/TS-compiler check. No behavior change.
* Studio: harden export-size local-path handling
Address review feedback on the export-size endpoint's local sizing:
- Resolve symlinks and re-verify containment in _is_sizable_local_path so a
symlink inside a Studio root can't point the sizer outside it.
- Re-validate the resolved LoRA base before sizing, so a crafted adapter
whose base_model points outside the roots can't redirect the scan.
- Skip nested checkpoint-*/global_step* snapshots when summing local weight
sizes so a run dir's intermediate checkpoints don't inflate the estimate.
- Size the checkpoint directory for full fine-tune checkpoint exports (whose
base may be a local/custom path), keeping base-model sizing for adapters.
Adds tests for the adapter-base escape, symlink escape, and nested-checkpoint
exclusion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Harden model fetching: consent gate for trust_remote_code
Add a load-path consent gate that scans a model's auto_map repository code
before it executes and blocks CRITICAL/HIGH findings unless the user pins
approval of that exact code version. Capability detection stays code-free,
reading raw config.json instead of AutoConfig.
- Scan config.json and tokenizer_config.json auto_map, nested local helpers,
and external owner/name--module repos; fail closed on partial downloads.
- Gate inference, training, and export workers, including the MLX path and a
LoRA's base model, and report requires_trust_remote_code from the raw config
so chat and auto-load surface the dialog.
- Verify trusted-org auto-enable against the Hub with the request token and key
the verdict cache by token; reject local-path and spoofed names.
- Add a consent dialog showing the flagged file, line, and surrounding code.
- Thread hf_token through the scan and load paths for gated repos.
* Address review: token handling, tokenizer/LoRA scan coverage, rollback
- Send the HF token for remote-code scans in the POST body, not the URL, so it
never lands in a log or browser history.
- Collect tokenizer_config.json auto_map files directly instead of relying only
on the repo file listing.
- Resolve a LoRA's base model for the validate flag and the scan endpoint so the
dialog scans the code the workers actually gate.
- Pass the request token to the training YAML trusted-org auto-enable.
- Resend a previously approved fingerprint when rolling back to a custom-code
model after a failed switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consent UX: drop legacy chat toggle, fix decline copy, purge declined downloads
The per-model consent dialog is now the single approval path for custom
(auto_map) code in chat, so three leftovers from before it existed are removed:
- Remove the "Enable custom code" switch from Chat Settings and stop persisting
trust_remote_code, so a previously saved blanket-on cannot linger and load a
model without going through per-version review. The flag stays as an internal
YAML/preset default (e.g. first-party auto-enable); the load path still gates
every custom-code load on a fingerprint only the dialog produces.
- Reword the decline message and the auto-load toast to describe approving the
model's code from the dialog, not a missing settings toggle.
- On decline, purge the repo the scan downloaded so untrusted code is not left
on disk. A new /api/models/discard-remote-code endpoint deletes only a
metadata-only cache entry the scan created; it refuses local paths, loaded
models, and any repo with weight files cached, so a model the user already had
or pre-downloaded is always left untouched. The frontend only calls it when
the scan reported created_by_scan.
Adds discard-endpoint tests (delete metadata-only, refuse on weights/gguf,
refuse local, no-op when not cached) and a created_by_scan payload assertion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Export: remove the user-facing trust remote code toggle
The Export page kept a "Trust remote code" switch (default on) next to the HF
token field. Like chat, custom (auto_map) code should be approved per model
through the load-time review dialog, not a persistent blanket switch, so the
toggle is removed. The export load path already routes through the same consent
dialog: an HF source now starts with trust_remote_code off and only enables it
when the user approves the scanned code in the dialog (a local checkpoint the
user exported stays trusted by default). With the dialog unreachable and no
approval, an HF source loads with trust_remote_code off, which fails closed
rather than running unreviewed code.
* Block loads of repos with unsafe files using Hugging Face's security scan
The trust_remote_code consent gate covers one load-time RCE vector (a repo's
auto_map Python). It does not cover the other: a malicious pickle inside a weight
file (pytorch_model.bin, *.pkl, *.dat) deserializes during from_pretrained even
with trust_remote_code False, so a repo with a normal config plus a poisoned
pickle slips past the existing gate.
Add a metadata-only malware gate that uses Hugging Face's own scan (picklescan +
ClamAV), read via model_info(securityStatus=True).security_repo_status. It never
downloads, opens, or unpickles the flagged files; it only reads the Hub's verdict
and surfaces the flagged file names. New evaluate_file_security runs
unconditionally (independent of trust_remote_code) in every load path (inference,
training SFT/MLX, export), blocking the load when a file is flagged
unsafe/suspicious/malicious. The /remote-code-scan preflight and the validate
endpoint also report the result so the consent dialog opens as a hard block (no
override) listing the flagged files, even for a repo with no custom code.
Policy: hard block with no user override; fail open when the scan is unavailable
(offline/unscanned) so legitimate loads are not broken; no first-party exemption
(a poisoned pickle in a compromised trusted repo still blocks); local paths and
GGUF are skipped (no Hub scan, non-pickle format). Blocking does not gate on
scansDone, since that is often false for clean repos and a file already flagged
unsafe is unsafe regardless.
Adds test_file_security.py covering the block/allow/fail-open/skip matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scan list-form tokenizer auto_map, gate unsafe files on all load paths
Fixes from a 10-reviewer pass on the model-fetching hardening:
- The remote-code scanner skipped tokenizer auto_map encoded as a [slow, fast]
list (transformers' standard tokenizer shape, e.g.
{"AutoTokenizer": ["owner/repo--tokenization_x.Slow", null]}). External
tokenizer code in that form was never fetched, scanned, or fingerprinted, so an
AutoTokenizer(trust_remote_code=True) load could run it. _auto_map_refs now
flattens string, list, and nested values. Adds a regression test.
- Compare-mode chat loads and background auto-load only gated on
requires_trust_remote_code, so a repo flagged unsafe by the Hub scan but with no
custom code skipped the hard-block dialog. Both now also gate on
requires_security_review, matching the main chat path.
- The /remote-code-scan and /validate routes collapsed a LoRA adapter to its base
before the malware scan, so unsafe files in the adapter repo itself were missed
in the pre-load review (the workers already scan both). Both routes now run the
file-security scan over the adapter and the base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Require approval for all HIGH remote code, fail closed when unscannable
Tighten the load-time security gates based on review:
Consent gate
- HIGH-severity auto_map code now requires explicit, per-version approval for
every repo, including first-party unsloth/nvidia. The org is no longer a
blanket bypass: a compromised first-party repo with HIGH code still warrants
review. CRITICAL stays a hard block; clean code still loads after the consent
prompt.
- Fail closed when auto_map code is present but cannot be fully fetched or
listed to scan (gated, offline, transient, or a repo-listing failure that
could hide an imported helper). We cannot fingerprint code we cannot see, so
this is a non-approvable block, retryable once the repo is reachable.
- Scan auto_map from every config that can carry one (model, tokenizer, image
and feature processor, processor, video processor), not just config.json and
tokenizer_config.json, so a custom-processor model is not missed. The file
list is the single source of truth in remote_code_scan and is pinned to the
transformers filename constants by a guard test.
- Distinguish a genuine 404 (config truly absent) from a transient error: only
the latter forces a scan, so a repo with no config is correctly a no-op.
Malware gate
- Scan a remote repo even when its name ends in .gguf; only local paths skip the
Hub scan, so a repo cannot dodge the scan by naming itself "*.gguf".
- Correct the docstring: a file already flagged unsafe blocks regardless of
scansDone; the only fail-open path is an unavailable scan.
Coverage
- Resolve a remote LoRA adapter's base model (not just local directories) so the
base, where the code and weights actually execute, is scanned in validate,
the scan route, and the training and export workers.
- Gate the embedding training path (FastSentenceTransformer) with the malware
and consent checks, matching the other load paths.
Tests updated and added for each change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope malware gate to the load-path vector; stop false-blocking first-party models
Follow-up hardening from a second review pass + a broad live model matrix
(unsloth/* , nvidia/* , third-party, and the eicar malware repo).
Malware / unsafe-file gate
- Scope the block to the actual RCE vector: a root-level file in a code-executing
format. from_pretrained deserializes weight files at the repo ROOT, so a flag is
only a load-path pickle vector there. Two exclusions, because neither is loaded:
inert formats (safetensors is tensor-only, gguf is non-pickle, configs/text/
images) and files in subdirectories. This keeps eicar blocked (its *.pkl/*.dat/
eicar_test_file sit at the repo root) while no longer false-blocking legitimate
first-party repos: nvidia/Nemotron-H-8B-Base-8K ships root safetensors plus NeMo
pickle checkpoints under nemo/ that the loader never touches, and the Hub flags
both; the gate previously hard-blocked it.
- Unknown / future non-"safe" levels now fail closed (block) instead of being
silently allowed, so Hub schema drift cannot introduce a bypass; in-progress
("pending"/"scanning"/"error") levels stay non-blocking to avoid false blocks.
Consent gate
- Ignore a STALE own-repo auto_map target that is absent from the repo listing (an
older config pointing at a file the repo no longer ships) instead of failing the
whole repo closed as unscannable. The present .py are still fully scanned, which
is the stronger coverage, and a file that is not there cannot execute. This
unblocks first-party models like unsloth/PaddleOCR-VL (its tokenizer_config.json
names processing_ppocrvl.py while the repo ships processing_paddleocr_vl.py). A
referenced .py that IS present but cannot be fetched, and a repo-listing failure,
still fail closed.
Remote LoRA base resolution
- Distinguish a genuine 404 (not a LoRA / repo absent -> None) from a transient
error: the transient case is retried once, then logged as a WARNING (a missed
base is scanned by neither gate) rather than silently skipped.
Discard endpoint
- Treat .onnx and .ckpt as weights so a repo whose only heavy artifact is one of
those is never eligible for the declined-download purge.
Tests added for each: load-path scoping (safetensors/subdir/Nemotron-H shapes,
unknown-level fail-closed, pending non-block), stale own-repo auto_map ref, remote
LoRA transient retry, and the empty-config-list (all-404 -> []) semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make LoRA-base transient-warning test robust to logging backend
Assert on the logger object directly instead of capsys, so the test does not
depend on whether the real structlog logger or the module-stub logger is active
(which varies with test collection order).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allow a repo with auto_map but no executable code (e.g. GGUF) instead of blocking
A config can declare an auto_map yet the repo ship NO executable .py -- most
commonly a GGUF repo whose config.json carries an auto_map copied from the original
model (e.g. unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF references
modeling_decilm.py, which the GGUF-only repo does not contain). A GGUF model loads
through llama.cpp, which never executes auto_map, and transformers cannot run a file
that is not present, so there is nothing to scan and trust_remote_code is a no-op.
The fail-closed change treated this empty result the same as "code is present but we
could not fetch it" and hard-blocked the load. Distinguish the two: repo_remote_code_files
now RAISES RemoteCodeUnscannable when code is present but cannot be fully fetched or
listed (offline / gated / transient / a present .py that 404s / a listing failure),
and returns an empty dict only when the listing succeeded and the repo genuinely ships
no executable .py. The consent gate blocks on the exception (fail closed) and allows the
empty case as a no-op. Real unscannable code still hard-blocks; eicar and CRITICAL/HIGH
custom code are unaffected.
Verified against all 37 unsloth/*Nemotron* models (two GGUF repos were false-blocked,
now load) and the existing matrix (eicar still blocks; DeepSeek-OCR / NVLM-D-72B still
prompt approvable consent). Tests updated to expect the raise for unscannable cases and
added for the no-executable-code no-op.
* Ignore vestigial auto_map in GGUF repos (llama.cpp never runs it)
A GGUF repo's config.json is often copied verbatim from the original
transformers model, auto_map and all, but a GGUF load goes through
llama.cpp which never executes auto_map, so the config is inert. Treat
a direct .gguf reference, and a repo that ships .gguf weights with no
.safetensors, as having no remote code so the consent flow is never
triggered. A mixed repo with both .gguf and .safetensors is still gated,
since the safetensors variant would load through transformers where
auto_map does run. The check sits behind the existing auto_map-present
gate so normal models pay no extra repo listing.
* Add scanner-result copy to the remote-code consent dialog
Make the consent dialog state the scan outcome in plain language for
every model. When the static scan finds nothing, reassure the user with
'Our automatic scanner did not flag any worrying files, but please
double check.' (shown only for the clean, approvable case). When the
scan flags custom code or unsafe files, label the list with 'Our
automatic scanner flagged issues including:'. The Hugging Face
attribution for unsafe files stays in the dialog description.
* Close GGUF-suffix consent bypass for repo ids ending in .gguf
The .gguf short-circuit in _config_has_auto_map skipped the scan for any
model name ending in .gguf, including a bare two-segment repo id like
'evil/model.gguf'. Such a repo can still ship safetensors plus auto_map
Python that transformers would execute, so skipping the scan was an
asymmetric bypass (file_security already scans those repos). Restrict the
short-circuit to genuine direct GGUF file references via
_is_direct_gguf_file_ref: a local .gguf path, or a remote repo_id plus
filename (three or more segments). A two-segment repo id named *.gguf now
falls through to the config scan and _is_gguf_repo file inspection, so it
only skips consent when it actually ships .gguf weights and no safetensors.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align consent dialog body with the title and fix narrow-width overflow
The scan results (the 'Our automatic scanner...' label, finding/unsafe
cards, and the clean-scan reassurance) sat at the dialog's left padding
while the title and description were indented past the status icon, so
the body did not line up under the description. Move the title,
description and results into one column to the right of the icon so they
share a left edge, and let that column fill its width so the description
no longer wraps early.
Also stop a wide code snippet from pushing the dialog off-screen on
narrow viewports: AlertDialogHeader is a grid with place-items-center,
which sized the content row to its content; give the row w-full so it
fills the track, and add min-w-0 down the results chain so the snippet
scrolls inside its card instead of widening the dialog. Verified aligned
and contained from mobile portrait through ultrawide.
* Treat a repo as GGUF-only only when it ships no transformers weights
_is_gguf_repo excluded only .safetensors, so a repo with a .gguf and a
pytorch_model.bin (or .pt/.pth/.h5/.msgpack/.onnx/.ckpt) and no
safetensors was treated as GGUF-only and skipped the consent scan, even
though transformers can load that weight set and execute the repo's
auto_map code. Require the absence of ANY transformers-loadable weight
before treating the repo as a llama.cpp-only GGUF load. A genuine
GGUF-only repo (only .gguf) is still inert; a mixed repo with any pickle
or safetensors weight is gated. Adds a regression test across all the
non-safetensors weight formats.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Block flagged subdir weight shards referenced by a root index
The malware gate treated every subdirectory file as non-loadable, but
from_pretrained deserializes a subdir shard a root index references
(pytorch_model.bin.index.json -> shards/...-00001-of-00002.bin). Read the
root weight indexes and block a flagged subdir pickle the weight_map
points at; a flagged subdir pickle no index lists (NeMo nemo/*.distcp)
stays non-blocking, and an inconclusive index lookup fails closed.
* Pass hf_token to the export checkpoint load
ExportBackend.load_checkpoint scanned with hf_token in the worker but
loaded the weights unauthenticated, so a gated/private checkpoint passed
preflight then 401'd at from_pretrained. Add hf_token to load_checkpoint
and forward token to every from_pretrained branch; the worker passes the
command's hf_token.
* Scope created_by_scan to every HF cache the discard searches
created_by_scan used get_cache_path (active HF_HUB_CACHE only) while
/discard-remote-code deletes across active, legacy, and default caches. A
repo the user already had in a legacy/default cache was marked
scan-created and deleted on decline. Check all three caches for the repo
dir before declaring the scan created it.
* Scan the full .py closure of external auto_map repos
An auto_map cross-repo ref (owner/name--module.Class) only had its entry
file downloaded, but transformers also fetches that file's relative
imports from the same repo, so a dangerous helper.py was left outside the
scanned fingerprint. List each external repo's .py and scan the whole set
(plus the referenced entry files); fail closed if the repo cannot be
listed or fetched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail closed when a weight index cannot be fully read
_indexed_shard_paths treated a partial result as definitive: if one weight
index read cleanly but another failed transiently, it returned the shard
paths it did see. A flagged subdirectory pickle listed only by the index we
could not read would then be classed as "not a load input" and skipped,
re-opening the very fail-open this guard was added to close.
Return None whenever any index read is inconclusive, even if another read
cleanly, so the caller blocks the already-flagged subdir pickle. A repo that
ships no index files raises EntryNotFoundError for each (never inconclusive)
and still returns an empty set.
* Match cached repos case-insensitively in the created_by_scan guard
_repo_in_any_hf_cache resolved casing only against the active cache and then
probed every cache with an exact directory name. A case-variant already
present in a legacy or default cache (models--Unsloth--Foo for a scan of
unsloth/foo) was missed, so the repo was marked created_by_scan and deleted
on decline -- but discard_remote_code_download deletes case-insensitively,
so that delete would hit the user's pre-existing cache entry. Detect
case-insensitively too, mirroring the deletion path.
* Skip remote-code and security review for selected GGUF variants
validate_model ran the trust_remote_code and Hugging Face security-scan
preflight against the repo even when the selected artifact is a .gguf. A
GGUF loads through llama.cpp, which never executes the repo's auto_map
Python and never deserializes root pickle weights, so repo-level Transformers
artifacts (a config.json with auto_map, or an unsafe pytorch_model.bin next
to the .gguf in a mixed repo) are inert for that load. Gating the GGUF on
them is a false positive. Run both preflights only for non-GGUF loads.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the malware gate to actual load roots and serialized files
Two fixes to evaluate_file_security so it neither misses a load-path pickle nor
false-blocks an inert file:
- Honor subdirectory load roots. Spark-TTS / BiCodec call from_pretrained on the
snapshot's LLM subdirectory, so a flagged pickle directly under it is a
root-level load artifact there. A new load_subdirs parameter (set from the
model's audio type via security_load_subdirs) reclassifies those files relative
to the load root and looks for weight indexes under it, so a flagged shard in
that subdir is no longer skipped as "not root-level".
- Exempt source files. A root .py is never deserialized by from_pretrained;
executable repo code runs only through auto_map, which the remote-code consent
gate scans. Flagging a Python helper here would false-block a repo that merely
ships a build or train script.
* Scan a LoRA adapter and base as one consent unit, and gate MEDIUM code
A LoRA load runs both the adapter's and the base's repo code. The consent gate
scanned them separately and pinned one fingerprint per repo, so an adapter that
shipped its own auto_map code was either never shown in the dialog (which only
saw the base) or impossible to approve with the base's fingerprint.
evaluate_remote_code_consent_for_targets now scans all of a load's repos as a
single combined unit and pins ONE fingerprint over the union of their code, so
approving the load approves every repo's code together. evaluate_remote_code_consent
becomes a thin single-target wrapper, and an unscannable target fails the whole
load closed.
Also gate MEDIUM findings: like HIGH they now block pending pinned approval, so a
direct API caller cannot run flagged code by setting trust_remote_code=True
without consenting. Only a clean scan loads without a fingerprint.
* Preflight a LoRA load's adapter and base as one combined consent scan
scan_model_remote_code rewrote a LoRA adapter to its base and scanned only the
base for remote code, so the dialog never surfaced an adapter's own auto_map
code. Scan the adapter and base together through
preflight_remote_code_consent_for_targets, which pins one combined fingerprint
the worker gate accepts. The malware preflight is also scoped to each target's
load subdirectories.
* Apply combined consent and subdir-aware malware scan in load workers
Each load worker (inference, export, training) evaluated remote-code consent
once per target with a single shared fingerprint, so a LoRA adapter that ships
its own auto_map code could not be approved by the base's fingerprint. They now
scan the adapter and base together via evaluate_remote_code_consent_for_targets,
which pins one combined fingerprint over the union of their code. The malware
scan in each worker is also scoped to the model's load subdirectories so a
flagged pickle under a from_pretrained load subdir is not missed.
* Report a consistent trust_remote_code requirement after a model loads
validate_model reports requires_trust_remote_code from the YAML default OR the
raw auto_map, but the load, already-loaded, and status responses reported only
the YAML default. A custom-code model approved and loaded via auto_map was then
reported as not requiring trust_remote_code, so the frontend stored false and a
later retry or rollback sent trust_remote_code=false and failed.
A shared resolver reports the same requirement for a loaded model (a value
stored at load time, else the trust_remote_code the load used, else the YAML
default, else the raw auto_map check), and the load response persists it so the
status and already-loaded paths stay consistent. The selected-GGUF security
review is also scoped to the model's load subdirectories.
* Run the consent gate on training resume and for YAML-only trust_remote_code
Three frontend gaps left a model loading without the trust_remote_code it needs:
- The shared consent helper returned early when the scan found no auto_map and no
unsafe files, dropping a requirement that comes from a model's Studio YAML
default (e.g. GLM-4.7-Flash). It now grants the caller's requirement with an
empty pin instead of sending trust_remote_code=false.
- Resume-from-history called startTraining directly with no consent gate, so a
resumed run whose model needs custom code (or an old run with no approved
fingerprint) hit the worker block with no dialog. It now runs the same gate as
a fresh start.
- HF export passed requiresTrustRemoteCode=false for every HF source, so a
YAML-only model could not flip the flag before export. It now signals the
requirement for HF sources.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover both LoRA repos in validate, report GGUF as inert, purge all declined repos
Three follow-on gaps from the combined adapter+base consent work:
- validate_model resolved requires_trust_remote_code from the base alone, so a
LoRA adapter that ships its OWN auto_map code (with a plain base) was reported
as not needing trust_remote_code and the consent dialog never opened. It now
checks the [adapter, base] target set, matching the scan route and the workers
(which already gate both) and the security review already running over both.
- The already-loaded, loaded, and status responses for a selected GGUF reported
requires_trust_remote_code from the model's YAML default. A GGUF loads through
llama.cpp, which never executes the repo's auto_map Python, so the requirement
is inert for that load. They now report False, matching validate_model (which
already skips both gates for GGUF) so a status refresh cannot flip the flag
back on.
- The remote-code scan downloads both the adapter's and the base's config, but
created_by_scan tracked only the primary, so a base the scan was first to pull
into the cache was left on disk when the user declined. The scan now reports
scan_created_repos (every repo it newly cached) and the decline cleanup purges
each; created_by_scan stays for older clients. The frontend falls back to the
primary flag when the list is absent.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan the repo the load fetches, purge external code on decline, harden consent pins
Six follow-on hardening fixes from a fresh review pass over the gate:
- The malware gate scanned the literal "Spark-TTS-0.5B/LLM" alias, but the trainer
downloads it as unsloth/Spark-TTS-0.5B and loads LLM/, so the alias 404'd and
failed open, missing a flagged LLM/ pickle. evaluate_file_security now resolves
the alias to the repo the loader fetches and scans LLM/ as a load root.
- security_load_subdirs relied only on tokenizer detection, which fails on an
unresolved alias or offline; it now also honors the Studio YAML audio_type
default, so a BiCodec LLM/ load root is not missed.
- The remote-code scan downloads external auto_map repos (owner/name--module.Class),
but the decline cleanup tracked only the model/adapter/base, leaving the external
untrusted code cached. The scan now enumerates external auto_map repos and reports
the ones it created in scan_created_repos, so a decline purges them too.
- External auto_map refs failed the whole load closed on a stale or mis-derived
dotted ref (sub.mod.py vs the real sub/mod.py) even though the actual file was
present and scanned. They now drop such refs when the repo listing is real, exactly
like the own-repo path; an empty/incomplete listing still fetches and fails closed.
- The combined consent fingerprint keyed code by the raw target string, so the scan
endpoint's canonicalized casing and a worker's raw user input produced different
pins for identical code, rejecting a valid approval. Hub repo ids are now folded to
lowercase in the key (local paths stay case-sensitive), so the pin tracks the code.
- Export threaded hf_token into the weight load but not into detect_audio_type /
is_vision_model, so a gated multimodal base 404'd in detection and fell through to
the text loader. Both probes now use the same token.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Thread the token through check-vision and guard the gate's parallel sites
The /check-vision endpoint classified a model without the hf_token, so a gated or
private vision model 404'd in the probe and was reported as a plain text model --
the same dropped-token shape as the export probes, at a sibling site. It now passes
the token like the neighboring /check-embedding endpoint.
Add deterministic consistency guards (tests/test_security_gate_consistency.py) that
enumerate the gate's parallel sites mechanically instead of relying on a review to
spot a missed sibling: every is_vision_model / is_embedding_model / detect_audio_type
caller under routes/ and core/ must thread the token, every GGUF response must report
trust_remote_code via the resolver or False (never the raw YAML default), and every
load worker that runs the malware or consent gate must resolve the LoRA base. A new
site that drops the token or mis-reports the requirement now fails CI directly.
* Narrow the LLM alias rewrite and make audio detection token-aware
Three fixes from the confirmatory review, one a regression from the previous round:
- _load_scan_target rewrote EVERY remote repo ending in "/LLM" to unsloth/<parent>,
so a real third-party repo named "<owner>/LLM" was scanned as unsloth/<owner>
while the loader still fetched the real repo -- a fail-open hole introduced when
the Spark-TTS alias handling was added. It now rewrites only a registry-known
bicodec alias; every other "/LLM" repo is scanned as itself.
- detect_audio_type cached results under the bare model name, so an unauthenticated
probe of a gated/private repo cached None and poisoned a later authenticated call
with the token. The cache is now keyed by (normalized_name, token_fingerprint),
matching the vision cache.
- The training fallback /check-vision call dropped the hf_token, misclassifying a
gated/private VLM when the config endpoint failed. It now passes the token, like
the getModelConfig call it falls back from; checkEmbeddingModel takes the token too.
Extend the consistency guards: every capability cache must be keyed by a tuple
including the token, so a cache re-declared as Dict[str, ...] fails CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Document the broad .py scan as deliberate and enforce it with a test
The remote-code scanner scans every .py in a repo once an auto_map exists, not
just the auto_map entry's static import closure. This is intentional: the entry
module can reach a sibling via an absolute import, importlib, or exec, none of
which a static relative-import closure follows, so closure-only scanning would be
a real bypass of a load-time RCE gate. The broad scan never under-scans; the cost
is that an unrelated benign script can over-block, which is the safe failure
direction (HIGH stays approvable; only CRITICAL hard-blocks).
Spell this out at both the local and remote scan sites so the choice reads as
deliberate, and add a test asserting an unrelated, never-imported .py is still
scanned -- so a future narrowing to the static closure fails CI.
* Purge a declined remote LoRA adapter the scan downloaded
scan_model_remote_code probed the created-by-scan state AFTER resolving the base,
but get_base_model_from_lora_identifier downloads a remote adapter's own
adapter_config.json, so the adapter looked already-cached and was dropped from
scan_created_repos. On decline the adapter -- including the auto_map .py the
preflight fetched -- was left on disk, defeating the "untrusted code is not left
on disk" guarantee for the adapter itself.
Snapshot the primary's cache state BEFORE base resolution and use it when marking
the adapter scan-created; on any probe error treat it as pre-existing so a decline
never deletes it. The base and external repos are unaffected (their configs are not
downloaded before their own probe). Add a test that models the mid-scan download
side effect, which the prior static-stub tests did not.
* Clear remote-code approval when the training model changes
Switching the training model from an approved custom-code model to a clean one
kept the previous model's trust_remote_code=true and approved fingerprint in the
store: setSelectedModel reset visionImageSize on a true switch but not the
remote-code approval. The clean model then trained with trust_remote_code=true,
which bypasses the compiler and disables fused cross-entropy.
Reset trustRemoteCode and approvedRemoteCodeFingerprint on a true model switch.
The new model's own YAML default is re-applied by loadAndApplyModelDefaults, and a
custom-code model still re-opens the consent dialog before training starts, so the
only change is that a clean model no longer inherits a stale approval.
* Trim verbose comments across the model-fetching hardening changes
Condense the explanatory comments and docstrings introduced across the
trust_remote_code consent gate, the malware/unsafe-file gate, the remote-code
scanner, the load workers, the model routes, and the security frontend into
fewer, tighter lines while preserving every security rationale (fail-open vs
fail-closed direction, the deliberate broad-scan anti-bypass note, the
empty-vs-unscannable distinction, stale-ref handling, and the alias-rewrite
spoof guard).
Comments and docstrings only. No code, logic, identifiers, or test behaviour
changed; verified comment-only via the AST/TypeScript checker (40/40), with the
backend test suite and frontend tsc green.
* Do not cache transient audio-detection failures
detect_audio_type cached _detect_audio_from_tokenizer's result
unconditionally, so a transient read failure (network error or 5xx,
returned as None) poisoned the cache and the later successful probe never
ran. Mirror the vision cache: _detect_audio_from_tokenizer now returns
(audio_type, definitive) and the caller caches only definitive results.
A read that succeeds with no audio tokens, or clean 404s for every
tokenizer path, stays a cacheable None; only a genuine transient failure
(connection error, timeout, 5xx, malformed body) skips the cache so the
next call retries.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: hide the llama-server install validation probe from model pickers
The ggml-org/models / stories260K probe (install_llama_prebuilt validates
the prebuilt binary against it) can land in the HF cache and, being a 260K
toy model, sorts smallest and gets auto-selected for chat. Hide it in
_is_hidden_model alongside the RAG embedding model so it never surfaces.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: assert the validation probe is hidden across all model pickers
_is_hidden_model gates the model list, local, cached-GGUF and cached-models
endpoints that feed the picker search, so cover both the repo-id and the
on-disk snapshot-path forms callers pass.
* Studio: match the validation probe by exact filename, not a bare substring
Use the probe's repo id (ggml-org/models) plus its exact filename
(stories260K.gguf) so the picker filter does not also hide unrelated repos
that merely reference stories260K, e.g. user/stories260K-finetune-GGUF.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Downloaded model list disappearing and order it by last download
The chat model picker scan for cached GGUF and safetensors models aborted
whenever an auxiliary Hugging Face cache dir (such as ~/.cache/huggingface/hub)
was unreadable, returning an empty list. That hid the Downloaded section and
let already downloaded models appear under Recommended. Isolate each cache
probe so an inaccessible directory is skipped instead of failing the scan.
Also order Downloaded newest-first using cached blob mtimes (multi-quant repos
group by their most recent quant), keep the section visible while searching,
and make the per-quant downloaded check per-snapshot and mmproj aware so a
Recommended quant is never falsely marked downloaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden gguf-variants scan and dedupe by newest timestamp
Guard f.stat() per file so a broken symlink or unreadable file in a
snapshot no longer aborts the downloaded check early, and match quant
labels case-insensitively. When the same repo is present in multiple
caches with equal size, keep the newest last_modified so Downloaded
ordering reflects the most recent copy.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply cache-scan guards to sibling endpoints found in review
Extend the inaccessible-cache guard and mmproj/stat hardening to the
parallel HF cache code paths flagged in review:
- list_local_models and the Hub inventory scan now skip an unreadable
auxiliary cache instead of returning 500.
- The GGUF download-progress endpoint excludes mmproj adapters and
guards f.stat() so one bad file does not zero a repo's progress.
- The offline snapshot scanner guards its is_dir() probes.
- The chat-only picker no longer renders a blank list when a search
matches only cached non-GGUF models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController.
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* Studio: stop leaking internal exceptions to API clients; harden sandbox path
Security hardening for the FastAPI backend.
Error exposure (CodeQL py/stack-trace-exposure): many route handlers returned
raw caught-exception text to clients via HTTPException detail / response bodies,
which can leak internal filesystem paths and stack detail. Add shared helpers in
utils/utils.py (safe_error_detail, log_and_http_error) that log the full
exception server-side and return a generic message, and sweep the route layer
(inference, models, export, training, datasets, chat_history, providers,
mcp_servers, settings, data_recipe/{jobs,seed,validate,mcp}) to use them.
Intentionally user-facing validation messages, the existing _friendly_error SSE
paths, and upstream-service body passthrough (llama-server / OpenAI) are kept;
absolute server paths echoed in models.py browse/read errors are redacted.
Path injection (CodeQL py/path-injection): serve_sandbox_file already does
basename + realpath containment; add a strict filename allowlist
(^[A-Za-z0-9._-]{1,255}$) before the path is built as defense-in-depth and to
give the analyzer a clear sanitizer.
No behavior change beyond error-message text; status codes preserved.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: keep curated error messages, fix remaining load leak
- inference.py /load non-native path: redact str(e) instead of leaking it
(matched the native branch which already redacted).
- llama_extra_args validation: return the curated, path-redacted message
instead of the generic fallback so users see the offending flag.
- sandbox file serving: allowlist now forbids only separators/control chars
via fullmatch, so generated images like 'loss curve.png' render again
while traversal is still blocked by basename + extension + realpath.
- Add safe_curated_detail() for domain/validation exceptions whose message
is intentionally user-facing; apply it to data_recipe job/validate,
chat conflict, provider test, and MCP probe paths (these were collapsing
to 'An internal error occurred', and 'connection' even mis-mapped to an
upstream-service message). Generic Exception paths keep safe_error_detail.
- log_and_http_error: tolerate stdlib loggers (no structlog kwargs).
- delete_openai_container: log transport errors with exc_info like list/create.
- Drop helper/HTTPException imports this change left unused.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* log_and_http_error: log original error traceback on stdlib-logger fallback
* Tidy error-helper and sandbox comments for PR #6072
* Trim redundant comments in studio error-hardening routes for PR #6072
* Re-trigger CI now that unsloth-zoo #727 is merged (Core pulls zoo main)
* Address PR #6072 review feedback
- inference.py: keep the actionable NativePathLeaseError detail (path-redacted)
instead of collapsing it to the generic message, matching the other curated
validation paths in this file.
- utils.py: log via a single formatted log.error(exc_info=error) call that works
for structlog and stdlib loggers; drop the now-unneeded try/except helper.
- models.py: use Path.name instead of os.path.basename(str(current)).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix /recommended-folders 500 on unreadable model dirs (Python 3.12+)
get_recommended_folders probed candidate paths with a bare
Path(p).is_dir(). On Python <= 3.11 that returned False for an
unreadable path, but on Python 3.12+ is_dir() propagates
PermissionError (EACCES) instead, so a stock root-owned ollama
install at /usr/share/ollama/.ollama/models (mode 700) made the
endpoint 500 through the entire middleware stack.
Move the directory-accessibility check into a stdlib-only helper
utils.fs_access.is_accessible_dir that swallows OSError and keeps
the existing os.access(R_OK|X_OK) filter, restoring the pre-3.12
"unreadable path is simply not a candidate" behaviour. Add a
dependency-free regression test.
* Address review: inline helper, no new file, cover all probe sites
- Drop studio/backend/utils/fs_access.py and the cross-module import;
the guard is now a small module-level _safe_is_dir() in models.py
(also moots the import-placement / ModuleNotFoundError feedback,
since there is no longer an import to misplace).
- Apply _safe_is_dir to every system-location probe with the
vulnerable bare is_dir() pattern, not just /recommended-folders:
_build_browse_allowlist._add and the /browse-folders _add_sug
helper, so the same Python 3.12+ PermissionError cannot 500 those
endpoints either. Each site keeps its exact prior semantics
(recommended-folders retains its os.access(R_OK|X_OK) filter);
the only behavioural change is "no longer crashes".
- Rewrite the regression test to extract the real _safe_is_dir from
source via ast, keeping it dependency-free without standing up the
FastAPI app, and correct the mode-000 case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* feat: enable deleting fine-tuned chat models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten fine-tuned model delete guards
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden fine-tuned model deletion edge cases
* fix: reject gguf export delete without variant; surface 503 on backend probe failure
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: narrow loaded-model delete guard by gguf_variant
* fix: clear inference state when cancelling model load
* fix: verify deletion completed and prune empty parent dirs
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Daniel Han