The memory plan's bf16_components_gb held 50.4 GB for the LTX text
encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download),
not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4
GB resident. The 26 GB over-estimate pushed the auto plan toward offload
on cards that fit the real footprint. Comments and the size-table test
now pin the resident semantics.
The generic quantize_text_encoders pass only covers text_encoder.._3, so
HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed
dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path
now lives there: when the requested TE quant is layerwise fp8 and the
device/family qualify, TE4 prefers the hosted pre-cast checkpoint
(unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then-
cast; a mid-pass cast failure reloads a fresh dense encoder instead of
shipping partial state. The pre-cast loader and builder gain
config_subfolder/config_overrides for standalone encoder repos whose
config sits at the root and whose pipeline needs forward flags
(output_hidden_states/attentions).
Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches),
hosted checkpoint engages through the real backend (marker + status fp8),
load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs
the dense-TE render (gate 0.25), non-black frames.
Hook presence alone cannot distinguish a legitimately pre-cast text
encoder from leftover hooks after a cast that failed mid-pass, so the
early return now requires the completion marker _cast_fp8 sets once the
hooks are fully installed. Leftover partial state keeps failing closed.
Also tolerates non-Module encoder doubles in the hook probe and the
dtype override.
Two more findings from the hosted-TE GPU smokes:
- Module.dtype reports the first floating parameter, which after the
layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt
embed and latent dtypes from encoder.dtype and feeds them to
randn_tensor, which has no fp8 kernel, so ANY flux.2 load with
text_encoder_quant=fp8 crashed at generation (pre-existing, runtime
cast included). The cast now swaps in a subclass whose dtype property
reports the compute dtype; forward behaviour is unchanged.
- The dense transformer_quant fast path assembles companions through
_assemble_pipe, which never received the pre-cast TE injection, so the
hosted encoder engaged on full-pipeline and GGUF builds but not on the
fast path. Threaded through like the other two branches.
Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker
with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE,
render non-black, and report text_encoder_quant=fp8 truthfully.
qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a
hosted pre-cast fp8 text encoder from their unsloth -FP8 repos:
- unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB
- unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB
- unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB
Every checkpoint verified bit-identical to dense-load-then-cast
(729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the
real backends with the repo engagement marker. Tests cover the wired
entries, the resolver filenames, builder metadata weights_only survival,
and the idempotent re-cast.
Two bugs found while building the hosted checkpoints:
- The builder recorded torch.__version__ (a TorchVersion object) in the
checkpoint metadata, so torch.load(weights_only=True) rejected every
artifact and the loader silently fell back to the dense download.
Record plain strings.
- Re-applying the layerwise fp8 cast to an injected pre-cast encoder
raised on the duplicate hook registration, making quantize_text_encoders
report the engaged cast as failed (status showed no TE quant while the
encoder ran fp8). _cast_fp8 now returns early when the hooks are
already installed.
Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B.
Wire te_prequant_pipe_kwargs into the three pipeline assembly sites:
the diffusion full-pipeline branch, the diffusion transformer-only and
GGUF branch (where the companion TE is the big remaining download), and
the shared video assembly path before the pipeline/component split.
Injection is gated exactly like the runtime cast (mode normalized to
fp8, device supported, family not denied), so it can never engage where
quantize_text_encoders would not; the later quantize_text_encoders call
re-applies the cast idempotently and keeps status reporting truthful.
With no hosted checkpoint configured the call returns {} and assembly
loads the dense encoder as before.
The runtime text_encoder_quant=fp8 path downloads the full bf16 text
encoder and layerwise-casts it in place on every fresh load. For the
heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B
~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load
time on a fresh machine.
diffusion_te_prequant.py loads a pre-cast fp8-storage state dict
instead: meta-init the encoder skeleton from the checkpoint's te_class,
load_state_dict(assign=True), rebuild on CPU if non-persistent buffers
stay on meta, then re-apply the same layerwise cast to install the
upcast hooks. The cast is a deterministic storage transform, so the
loaded encoder is bit-identical to dense-load-then-cast by construction.
v1 hosts the layerwise fp8 storage scheme only: its state dict is plain
tensors (torch.load(weights_only=True), no pickle execution). The
dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao
subclass wrappers at runtime and are deliberately not hosted.
Checkpoints validate format, scheme, component and base_model_id before
use and any problem falls back to the dense download and cast. Local
path overrides reuse the DiT prequant allowlist env var. Families opt in
via a new te_prequant_repos (scheme, component, repo_id) field on both
DiffusionFamily and VideoFamily; the field defaults empty so nothing
changes until a gate-validated artifact is wired.
Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291
int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means
0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly
quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0).
Uploaded to unsloth/HiDream-I1-Full-FP8.
Two live-test findings on the video progress endpoints:
- load-progress downloaded_bytes froze mid-download: the counter used
scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the
last completed blob for the whole multi-GB shard pull while the disk kept
filling. Count the repo's cache directory directly (completed plus incomplete
blobs, snapshot symlinks skipped so nothing is double-counted).
- generate-progress reported total_steps=null / fraction=0 while step advanced:
the video API only carried the native total field while the image API exposes
total_steps and fraction, so one poller could not work against both. Derive
the image-compatible aliases in generate_progress and declare them on the
response model; the native total stays for back-compat.
Two live-test findings on the images load path:
- transformer_quant with baked LoRAs, when the dense quantized build is
declined for memory or fails: the load completed as a plain GGUF with the
adapters silently dropped (HTTP success, supports_lora=false after the
fact) -- wrong output with no signal. The load now fails with the recovery
options (drop the adapters, free VRAM, or pick a smaller model). Weight-0
adapters still count as no bake request, and the plain no-LoRA decline
keeps its silent GGUF fallback.
- A fresh GGUF load on a small GPU prefetched the base repo's full bf16
transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch
widening only checked scheme viability, not whether the device could ever
hold the candidate resident. Gate the widening on total device capacity
(reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is
certain to decline the dense build never pays the download; capable devices
keep the prefetch.
Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors
(456 quantized) dequantize equal between the loaded checkpoint and a fresh
quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed
LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this
family's own run-to-run nondeterminism (identical weights and seed reproduce a
17/255 mean pixel delta through the 50-step guider pipeline); per-case hard
checks pass and the drift is compositional, reviewed visually. Uploaded to
unsloth/HunyuanImage-2.1-FP8.
A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text
encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open
Full / Dev / Fast repos (same arch); per-variant generation defaults follow the
upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28
and Fast 16 guidance-free).
The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not
ship its weights; the official example passes the gated meta-llama repo in by
hand. The loader instead assembles the component from the open unsloth mirror
(byte-identical weights, already inside the non-GGUF trust gate), injected at
the three pipeline from_pretrained sites, with output_hidden_states matching
the official example. Memory planning counts the assembled TE4: 34.2 GB DiT +
28.8 GB encoders, ~63 GB bf16-resident.
The hunyuanvideo-community diffusers mirror carries the full stack in
standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text
encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and
guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers
0.39 loads natively, so the generic from_pretrained pipeline path covers
everything with no per-component assembly.
Family notes:
- The call's guidance knob is distilled_guidance_scale (there is no
guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG
runs inside the repo's guider at its baked scale. Defaults follow the
card recipe: 50 steps, 3.25.
- 2K-native: verified live at both 1024 and 2048.
- Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no
diffusers pipeline and stays excluded with its stated reason).
- int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed
bf16); a short prompt does not trip the int8 torch._int_mm minimum on
this arch, so no family exclude entry is needed.
- bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB.
Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero
failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 /
SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing
checkpoint repo conventions.
Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B
encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the
generic from_pretrained pipeline path loads it as a new lumina-2 family:
- Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to
lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next
checkpoints are a different arch and must stay unknown rather than crash
mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image.
- Trust the official repo for non-GGUF loads; bf16 component table entry
(ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident).
- Generation defaults 50 steps / guidance 4.0 per the model card, and the
generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated,
signature-gated): the pipeline default (1.0) runs the CFG double-forward on
every step and oversaturates output.
- Catalog group with the single ungated bf16 pipeline artifact (11 GB resident)
plus routing assertions; images page defaults row.
- No GGUF artifact: none exists upstream (only finetune/LLM quants), so the
dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now.
Offline probes of the future prequant campaign: int8 and fp8 both engage and
render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory
drift with intact quality), so neither scheme is family-denied.
One family entry covers several published variants whose weights differ
(flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on
(family, scheme) alone, so only the default base could ever be served: the
loader's baked base_model_id validation correctly refused the schnell
checkpoint for dev and Krea-dev bases and every such load paid the dense
download plus on-the-fly quantise.
Add an optional prequant_variant_repos table on DiffusionFamily as
(base_repo, scheme, repo_id) triples and thread the resolved base repo
through resolve_prequant_source / usable_prequant_source and their three
call sites (load fast path, memory-plan probe, auto-policy candidate). A
base without its own entry keeps returning the family default, preserving
the existing refuse-then-dense behavior exactly.
Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8
checkpoints (built in the earlier campaign but never reachable) and the
new unsloth/FLUX.1-Krea-dev-FP8.
Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it
runs under the existing flux.1 family unchanged. Wire it up end to end:
- Catalog group with the gated official bf16 pipeline and the open QuantStack
GGUF quants; the gated artifact is skipped on auto-routing when undownloaded.
- Trust the official repo for non-GGUF from_pretrained loads, next to the other
black-forest-labs bases.
- Generation defaults: 28 steps at guidance 4.5 per the model card. The generic
"krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the
id, which would have produced garbage output; the new flux.1-krea key precedes
it on both the backend table and the images page table.
- The flux.1 prequant checkpoints are schnell-based; the loader's baked
base_model_id validation refuses them for the Krea-dev base, so int8/fp8
requests dense-quantize instead (covered by existing prequant tests).
Adapters are baked at load time: they attach to the dense transformer,
then quantize_ converts only the frozen base linears (the lora_ side
path is excluded by name), then the loader compiles. Post-quant PEFT
injection is not possible on a manually quantized module, so the
prequant shortcut is skipped for a baked load and the memory plan is
sized for the dense build (force_dense on the quant candidate).
At generation time the baked topology is frozen: weight tweaks and
disabling (scale 0 reproduces the quantized base exactly) go through
set_adapters, while adding or removing adapters returns a clean 400
telling the client to reload with the new selection.
supports_lora now returns True for int8/fp8 diffusers loads (checked
before the gguf-kind early return, since the quant fast path keeps the
picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The
load request model takes an optional loras list, threaded through
begin_load on both engines (native ignores it and keeps applying LoRA
at generation).
Verified end to end on GPU: Z-Image GGUF picker + int8 + trained
adapter loads through the API, bake marker logged, weight 1.0 vs 0
renders differ visibly, weight 0.5 accepted live, unknown adapter
rejected as 400. Affected suites: 296 passed.
Register flux.2-klein and flux.2-dev in the DiT trainer following the
upstream DreamBooth references: latents train patchified and batch-norm
normalized from the VAE posterior mode, the packed forward reuses
step-invariant position ids, and the guidance vector (3.5) is gated on
the variant's guidance_embeds config. Conditioning stacks load per
variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline
for Klein) and are encoded and freed before the transformer lands on the
device. The fused single-stream to_qkv_mlp_proj joins the attention
projections in the LoRA targets; the single-stream out projection stays
dense because its to_out suffix would also match the double-stream
ModuleList container.
Wire both families through the training registry (family set, labels,
VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them
trainable with train base repos in the family registry, add FLUX.2-dev
to the gated-repo token check, and trust both official bases for
training downloads.
Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev
int8 (52.0s) runs train with finite decreasing loss and the saved
adapters apply on the bf16 base pipeline (weight 0 reproduces the base
image exactly, weight 1 visibly restyles it).
A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model
(companions exceed budget) and silently served the GGUF as-is; the identical
retry went resident and engaged the hosted prequant. The plan arithmetic was
byte-identical across both loads (required 90,228 MiB, resident needs free of
about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is
device-wide and instantaneous: a transient foreign CUDA context briefly held
about 100 GB at the first snapshot, and the planner trusted that single read.
Three changes:
- settled_snapshot_device_memory: on cuda, synchronize + empty_cache
(best-effort) and take the MAX free over up to 3 spaced reads. A transient
can only shrink free, so the max rejects transient undercounts while a
persistent tenant still caps every read. _plan_memory now uses it.
- plan_fits_total_capacity + one replan retry: when the dense/prequant
candidate fits TOTAL device capacity under the standard reserve and the 0.85
resident margin, an offload verdict can only stem from the free reading, so
the loader re-snapshots and replans once before declining the fast path.
Explicit balanced/low_vram modes skip the retry (they offload by mode).
- diffusion.transformer_quant_declined log line with required/budget/free and
the plan reasons, so the next decline is diagnosable from the server log
(previously silent).
Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the
hosted prequant resident (offload=none).
Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the
Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short
prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its
M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than
16, but got 13' (measured on B200 through the Studio images tab).
Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for
qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme,
family) and the prequant checkpoint validation, so a checkpoint baked under the old
token list is rejected and re-quantised instead of loaded crashing. The text stream
runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the
rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069).
_assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo
ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer
dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the
assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take
(Krea2Transformer2DModel has no from_single_file). Assemble per-component via
load_krea2_pipeline like the pipeline-kind and single-file paths already do.
Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and
render through the Studio images tab.
Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image
(int8 only there; fp8 is family-denied), z-image and krea-2 at the
unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8
transformer checkpoints, so the fast quant path loads the small
pre-quantized file instead of materialising the dense bf16 transformer
and quantising on device. Measured on FLUX.2-dev int8: build peak drops
from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical
30.7 GB resident after either path since loading a checkpoint is
bit-identical to on-the-fly quantisation.
The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source
now derives that model-name filename from the repo id (scheme suffix
stripped case-insensitively) and carries the legacy transformer_<scheme>.pt
as a fallback the resolver tries when the primary 404s, keeping older
repos loadable.
Wiring a repo also exposed a fallback hazard: with a prequant source
present, the dense-fit preflight used to be skipped entirely, so a failed
prequant download would fall through to the dense bf16 load the memory
plan never budgeted, OOMing after eviction. The preflight now always runs
and gates an allow_dense_fallback flag through _load_dense_quant_pipeline:
a dense misfit still skips the fast path when no prequant exists, but with
one it proceeds and a prequant failure raises to the GGUF build instead of
loading dense. The same flag is set when the auto-policy replans an
offloaded GGUF against a prequant-sized transient.
Tests updated to the new filename convention plus new coverage for the
derivation and the legacy-name fallback; the prequant-skips-refit test now
asserts the re-check runs and forbids the dense fallback. Verified end to
end on GPU: z-image int8 resolves the hosted repo, downloads the
model-name file and renders (6.8s load, 5.9 GB peak).
An all-zero activation token row makes the dynamic per-row fp8 scale 0,
which turns the quantized data to NaN and the render to black frames on
torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels
clamp zero rows internally, so the bug only reproduces on machines without
them, which is most user environments. Zero rows are real inputs, not a
corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and
Qwen-Image regenerate zero rows inside their transformer blocks every step.
Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig
whenever the installed torchao supports the kwarg (Float8Tensor rework,
0.13+), checked via inspect.signature so older torchao keeps exactly the
current behaviour; the existing Float8MMConfig fallback chain is unchanged.
Verified on GPU: with the forced plain-torch kernel path a zero-row input
NaNs without the floor and stays finite with it, and end to end on
HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal
render (LPIPS 0.225); on Wan the floor matches the condition_embedder
exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels
present are unaffected, and pre-quantized fp8 checkpoints stay valid since
weight scales are untouched.
Gallery clear/delete now scope to Studio-owned files: image_gallery and
video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or
orphan file the listing already hides), so clear() and a guessed-id delete no
longer destroy files the gallery never surfaced.
Remote ControlNets now force use_safetensors: a bare owner/name reaches
from_pretrained without the base trust gate, and the Hub scan fails open when
unavailable, so requiring safetensors closes the pickle deserialization vector.
POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root
before removing the tree (marker-gated), mirroring the Windows stop-before-delete
scan; a live native server no longer survives unlinking its binary.
Diffusion dataset containment: the training-start read path and the discovery
picker route bare names through the protected resolver, so a symlinked dataset
is rejected / not advertised like the caption/delete routes already do. Uploads
gain the inference decode guard (oversized real images 400 before OOMing the
trainer) and dataset upload/caption/delete/import are blocked with 409 while a
diffusion run is active.
JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8
instead of raising AttributeError / 500.
LoRA family compatibility is enforced in the shared resolver, not only the
picker, so a direct API client cannot apply a mismatched-family adapter.
GPU arbiter gains release_if so the image/video unload idle-check and release
are atomic against a concurrent same-owner load's registration. Native batch
recipes persist the base batch_seed and restore replays from it, so a native
batch_index>0 image no longer advances its seed twice.
FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B,
9B -> Qwen3-8B) instead of the single family default.
Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path.
Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once.
Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker.
install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the
install created the target directory or it was empty. Adopting a pre-existing,
unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made
it eligible for the uninstaller's recursive delete.
routes/training upload: make the multi-file promotion transactional. Back up
each displaced original and roll every destination back on any failure, so a
mid-loop rename error can no longer partially overwrite the live dataset.
routes/training _resolve_dataset_folder: reject a symlinked dataset directory
and prove the resolved folder stays under the datasets root, so image
read/caption/delete cannot escape the root through a link.
routes/training delete: escape glob metacharacters in the thumbnail filename so
deleting an image named like [ab].png removes only its own thumbnails.
image_gallery / video_gallery listing: filter records against the response
schema inside the pager via a valid callback, so offset/limit/has_more all count
over accepted records. A leading schema-invalid record no longer returns an
empty page with has_more=true and stalls infinite scroll at offset 0.
image_gallery / video_gallery save: publish via a temp file plus atomic rename
(the sidecar is the video pair's commit marker) and clean up on failure, so a
partial write never surfaces a truncated PNG or strands an orphan MP4.
diffusion_train_common discovery: treat an empty caption sidecar as a metadata
tombstone that still falls through to the dreambooth instance prompt, so
clearing every metadata caption no longer fails with no captioned images found.
diffusion backend unload: wait for an in-flight denoise to exit before tearing
down process-wide patches and state, mirroring the load path.
diffusion_engine_router: serialize the whole check/unload/publish transition so
a concurrent selection cannot return the engine being unloaded.
uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a
user's own sd-server is not terminated for a directory we then keep.
generate() assigned self._gen only at the pipe() call, after deferred
compile, LoRA resolution/application, and ControlNet download/build had
run. Across that setup window generate_progress() reported inactive even
though _generate_lock was held, so a reloaded page's mount probe showed
idle and let a second generate queue behind the first.
Publish an active step-0 _GenState the moment the generation lock is
acquired, before the setup work, and clear it in the outer finally so a
setup-time error cannot leave the UI stuck active. Mirrors the video
backend's queued phase and the training start guard.
Register the dims the forward actually compiled with: image-conditioned
workflows (img2img, inpaint, upscale, edit) run at the input image's size,
not the slider's, so recording the slider values marked never-compiled
shapes as covered and warm restarts kept paying compile for the real one.
Validate a request-supplied transformer_prequant_path (existence plus the
UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH allowlist) before treating prequant as
available at the resident-fit re-check: an unusable path skipped the dense
fit check up front and then fell back to materializing dense bf16 after
the previous pipeline was evicted, recreating the post-eviction OOM path.
Shared as usable_prequant_source, also used by the auto-policy planner.
diffusion_compile_cache: auto mode now saves the Mega-cache bundle after the
first compiled generation (UNSLOTH_DIFFUSION_COMPILE_CACHE_SAVE=0 opts out), so
users get warm restarts without the distributor env; a bundle hit starts clean
(no pointless rewrite of the just-loaded artifacts) and explicit mode 1/on keeps
the distributor-style re-save. New register_shape + manifest shape coverage: a
STATIC compile produces new artifacts per (width, height, batch), so the
generate path registers each generation's shape and an uncovered shape
re-dirties the context, growing the bundle to cover every shape the session
used. Measured (B200, real backend): Qwen-Image deferred gen-3 hitch 29.1 ->
22.2 s warm with bit-identical output (7.9 MB bundle, ~0.5 s save); SDXL gen-3
115.7 -> 24.7 s and a mid-session 768px recompile 65.8 -> 12.6 s (bundle 63.6 ->
98.7 MB after the 768 re-save).
diffusion_speed: U-Net denoisers (UNet2DConditionModel; no _repeated_blocks, so
the regional compile never reached them) now get a whole-module STATIC
torch.compile on the default tier, plus fused QKV projections and a compiled VAE
decode. Measured on SDXL (30 steps / 7.0 / 1024px, 4 prompts, LPIPS vs the
bit-exact reference): 6.16 -> 3.14 s end to end (1.96x) at LPIPS 0.035, steady
state 0.70-0.88 s/image through the real backend. Rejected on measurement:
dynamic=True whole-module (366 s compile for 39.3 ms/step vs static's 73 s for
26.9), regional BasicTransformerBlock only (45.0 ms/step; ResNet convs stay
eager), max-autotune + inductor flags (25.9 ms/step for a 445 s warmup),
channels-last UNet alone (neutral). DiT tiers unchanged: fused QKV measured
exactly neutral under the regional compile (Qwen-Image 6.53 vs 6.52 s), so it
stays max-only there, and the DiT VAE decode stays eager (a few % of a DiT
generation). compiled_shapes_are_static tells the cache layer which loads are
per-shape (max tier, U-Net whole-module).
diffusion: register each generation's shape with the compile cache before the
save, pass pipe.unet to the cache fingerprint when the pipe has no transformer,
and correct the transformer_quant resolved reason on dense loads (it claimed a
GGUF transformer was loaded on every non-quantized pipeline load).
Tests: 333 passing across the related suites (speed 42, compile_cache 27, cache
40, precision 20, backend, base_precision, transformer_quant, memory); ruff
clean. Full measurement record: outputs/image_optim_round2_audit.md.
Applies the video round-2 accuracy findings to the image diffusion stack and fixes
two real image-path bugs found while measuring. All numbers B200, production
settings (family default steps/guidance, 1024px, seed 42, 4 fixed prompts), LPIPS
(AlexNet) via the new scripts/image_speedmem_bench.py, which drives the production
lever functions in the loader's own order.
- inductor precision parity: emulate_precision_casts=True on the regional-compile
path (fused pointwise kernels keep fp32 intermediates where eager rounds to bf16
between ops). Pairwise LPIPS of the compiled tier vs the same-stack eager tier:
Qwen-Image 0.019 to 0.006 at identical speed (72.4 vs 72.5 ms/step), FLUX.1-dev
0.046 to 0.029 at +2% step time (69.8 vs 68.3, reproduced), FLUX.2-klein-4B
0.018 to 0.017 at identical speed. Snapshot/restored with the other process-wide
backend flags so an off load never inherits it.
- cache x compile composition: re-point each cache hook's fn_ref.original_forward
at a torch.compile'd wrapper of the same bound method (armed only where the
speed layer compiled the block; restored before every disable_cache and before
the partial-hook cleanup). Qwen-Image FBCache computed steps 91.8 to 71.2 ms
(back at the uncached compiled rate), 1.21x end to end (7.36 to 6.06 s per 4
images); FLUX.1-dev already traced through its FBCache hook and is measured
neutral (same-process armed vs unarmed latents bit-identical). Skip counts
within noise (13 vs 11 of 76; pairwise LPIPS 0.005).
- FBCache mid-session toggle crash: diffusers 0.39 caches the HookRegistry child
list on first cache_context use, so an uncached generation followed by a
20+-step generation (the auto toggle path) enabled hooks the context never
reached and crashed with "No context is set" (reproduced live on FLUX.1-dev).
Invalidate the stale child cache after every enable_cache.
- TE fp8_dynamic zero-row guard: torchao per-row fp8 derives a per-output-channel
scale from the row amax, so an all-zero weight row is 0/0 = NaN. SDXL's
text_encoder_2 (OpenCLIP bigG) ships exactly such a row, and every explicit
fp8_dynamic SDXL render came out black; keep zero-row Linears dense (LPIPS
0.976 black to 0.096 working). Other families' encoders have no such rows and
are byte-identical.
- No AUTO TE quant exists on the image branch (text_encoder_quant defaults dense,
explicit-only), so the video round's auto-dense retune has no image analogue;
the explicit lever's cost is now measured (TE fp8_dynamic alone, LPIPS vs
bit-exact: Qwen-Image 0.038, FLUX.1-dev 0.084, SDXL 0.096; no speed win, VRAM
-6.5 GB on Qwen-Image) for the docs.
Tests: 96 passing across the cache/speed/precision suites (11 new arming, 2
child-registry, 2 zero-row, 4 inductor-flag); ruff clean.
POST /video/generate previously held the response open for the whole
generation (multi-minute for 720p), so in --secure mode the Cloudflare
quick tunnel's ~100s origin-response cap returned a 524 while the server
kept generating, and the frontend treated the run as failed.
Generation now follows the same return-at-once pattern as /video/load:
begin_generate validates synchronously (409 on no model or on a second
concurrent generate via a new busy sentinel) and runs the existing
generate + gallery-persist pipeline, with the route's exact error
mapping, on a daemon thread. GET /video/generate-progress gains optional
terminal fields: phase completed carries the saved gallery record, phase
failed a client-safe error; active only drops together with a terminal
phase. The cancel event is registered before the worker starts so
/video/generate/cancel keeps working across the whole job.
VideoGenerateResponse becomes an accepted acknowledgement (status
started, video kept as an always-null compat field). The video page
fires the POST, then drives completion off the progress poll it already
runs (completed prepends the clip, failed surfaces the error, the
cancelled sentinel stays toast-free). The API-key training-start guards
now also probe the video backend for an in-flight background clip, since
it is no longer visible as an in-flight HTTP request to the keep-warm
counter.
Route tests keep the fake backend for load/generate/status but inherit
the real job machinery, covering immediate accept, concurrent 409, the
terminal completed record, sanitized/ValueError/cancelled failures, and
cancel of a running job.
* Studio: resolve the repo-root MTP drafter after the MTP/ GGUF rename
The Gemma 4 QAT GGUF repos renamed the higher-precision MTP/ subdir
copies from gemma-4-...-<quant>-MTP.gguf to mtp-gemma-4-...-<quant>.gguf,
so their basenames now start with the same mtp- prefix as the small
repo-root drafter (mtp-gemma-4-E4B-it.gguf).
The drafter selectors filtered candidates by a mtp- basename prefix and
took the first in sort order. With the new names the MTP/ copies also
match, and because MTP/ (uppercase) sorts before the lowercase root file,
selection flipped to the large BF16 copy under MTP/ instead of the root
drafter both functions document they should pick.
Restrict both selectors, and the companion byte estimate, to root-level
mtp-*.gguf so the MTP/ copies stay explicit-selection only:
- core/inference/llama_cpp.py _pick_mtp (loader auto-download)
- hub/utils/gguf_plan.py preferred_mtp_sibling (Hub variant plans)
- routes/inference.py _remote_gguf_companion_bytes (VRAM headroom)
Also reuse a drafter already in the local cache before downloading, so a
device that already holds a copy on disk does not re-fetch it.
Old-scheme names keep working (they have no root-level mtp- sibling to
mis-select). Adds regression tests for the new naming, both selection
paths, and the on-disk reuse.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate MTP drafter cache reuse to offline mode
Reuse the cached drafter only when HF is offline. Online, route back
through _download_companion_gguf/hf_hub_download so the current revision
is checked (etag) and a changed drafter is refetched, matching the
offline-only cross-snapshot reuse already used for the main GGUF. This
avoids pairing freshly downloaded weights with a stale cached draft.
Make the reuse tests offline and add an online-skips-reuse test.
* Studio: prefer a root MTP drafter across all cached snapshots
Offline reuse scanned snapshots one at a time and returned the first
snapshot that held any drafter, only preferring root within it. A newer
partial snapshot with just the MTP/ copy could shadow the small root
drafter in an older snapshot. Collect drafters across all snapshots and
prefer any repo-root file before an MTP/ copy.
* Studio: keep newest-first snapshot order when reusing cached drafters
Collecting root candidates and sorting by absolute snapshot path could
pick a drafter from an older snapshot. _iter_hf_cache_snapshots yields
newest first and the main GGUF is resolved in that order, so preserve it
(root still preferred over MTP/ copies) to avoid pairing a fresh main
weight with a stale drafter revision.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Vulkan llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address gemini's feedback
* Studio: move the Vulkan VRAM probe into a standalone script
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve Vulkan probe error reporting
* Resolve llama-server symlink so Vulkan build is detected
* Drop unreachable Vulkan fallback in GPU free-memory dispatcher
* Skip the Intel GPU probe when NVIDIA or ROCm is present
* Reserve host RAM headroom for Vulkan integrated GPUs
* Add a `UNSLOTH_FORCE_VULKAN` environment variable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor GGML_VK_VISIBLE_DEVICES, reserve discrete Vulkan VRAM headroom, and clear Intel GPU on --cpu-fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route Intel and forced-Vulkan hosts to the upstream Vulkan prebuilt, add arm64 Vulkan, keep Vulkan out of RAG auto-detect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the fork release pin when routing a Vulkan host to the upstream repo
* Gate auto-Vulkan routing on no physical NVIDIA so hidden CUDA devices aren't used
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin Vulkan launches with --device Vulkan<i> instead of the raw GGML_VK_VISIBLE_DEVICES index space
* Let user --device override the Vulkan pin, and gate direct Vulkan asset picks on no physical NVIDIA
* Update RAG auto-backend test mocks for the _resolve_auto binary and Vulkan probes
* Keep the add_dll_directory handle alive through the Vulkan probe DLL loads
* Revert RAG auto Vulkan guard, guard multi-backend Vulkan detection, and preserve forced Vulkan across updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use getattr for RTLD_GLOBAL in the Vulkan probe CDLL mode
* Skip CUDA/ROCm APU and datacenter GPU tuning on Vulkan builds
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not
CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and
_apply_datacenter_env were reading the wrong device. On a mixed AMD APU
plus discrete GPU host that could raise a spurious system-RAM shortfall
and block a valid discrete-GPU load. Gate all three call sites on
not is_vulkan_backend; the Vulkan path already reserves iGPU host
headroom and the backend ignores GGML_CUDA_* anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Vulkan-guard comment in load_model
* Reduce comments in Vulkan support to be more succinct
* Resolve shell-wrapper llama-server entrypoint to the real lib dir
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install
root when it cannot symlink into build/bin. _find_llama_server_binary
returns that root entrypoint, but Path.resolve() does not follow a shell
wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend
missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device
pin on an otherwise valid Vulkan install. Follow the wrapper's exec target
to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
