Commit graph

6,566 commits

Author SHA1 Message Date
Daniel Han
e0786a98be Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 05:35:54 +00:00
Daniel Han
1978b2a844 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 05:35:53 +00:00
Daniel Han
1919491665 Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 05:35:52 +00:00
Daniel Han
dfe15c8141 Merge branch 'image-generation' into diffusion-train-perf 2026-07-05 05:35:50 +00:00
Daniel Han
fc35c40c18 Return 404 for malformed diffusion run records in the detail route
The run detail route built DiffusionTrainingRunDetail(**rec) unguarded, so a
valid-JSON-but-wrong-shape record (hand-edited or an older schema) would 500
instead of reading as absent. Catch ValidationError and 404, matching how the
list route skips malformed records.
2026-07-05 05:34:18 +00:00
Daniel Han
7a05d8655b Stream every DiT through group offload, not just the primary transformer
A dual-DiT pipeline (Ideogram 4's unconditional tower) placed its second
denoiser resident under the group tier, which defeats the tier since the
pair rarely fits where one alone did not. Stream transformer_2 and
unconditional_transformer alongside the transformer and keep only the
smaller companions resident.
2026-07-05 05:30:58 +00:00
Daniel Han
d6795ed077 Restore pre-Ampere bf16 fail-fast in the DiT trainer
The perf rewrite dropped the bf16 capability guard, so a pre-Ampere CUDA
device (T4/V100/RTX 20xx) would die deep in model load with an opaque dtype
error instead of a clear message. Restores parity with the SDXL trainer.
2026-07-05 05:27:58 +00:00
Daniel Han
d13c1d37c9 Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 05:01:12 +00:00
Daniel Han
017ef14877 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 05:01:10 +00:00
Daniel Han
54159ac64b Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 05:01:09 +00:00
Daniel Han
34a8459f2f Merge branch 'image-generation' into diffusion-train-perf 2026-07-05 05:01:08 +00:00
pre-commit-ci[bot]
237d197b5d [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:58:38 +00:00
pre-commit-ci[bot]
5732e3adc5 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:58:06 +00:00
pre-commit-ci[bot]
c475f7da00 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:57:35 +00:00
pre-commit-ci[bot]
09ff9e24f9 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:57:03 +00:00
pre-commit-ci[bot]
bcf25ca569 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:56:29 +00:00
Daniel Han
10d01cbc92 Merge branch 'diffusion-train-tab-2' into diffusion-krea2
# Conflicts:
#	studio/backend/core/inference/diffusion.py
2026-07-05 04:56:19 +00:00
Daniel Han
45f5ee7628 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 04:55:07 +00:00
Daniel Han
83a0f15c07 Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 04:55:05 +00:00
Daniel Han
e0f7ac68a4 Merge branch 'image-generation' into diffusion-train-perf 2026-07-05 04:55:04 +00:00
Daniel Han
e3ad2263fb Merge branch 'image-generation' of https://github.com/unslothai/unsloth into image-generation 2026-07-05 04:53:43 +00:00
Daniel Han
c53a6cb65e Address review findings: dataset preflight, sd.cpp unload barrier, caption tombstone, ControlNet cache race
- Run the trainer's caption discovery in the start route BEFORE freeing GPU
  residents, so a missing or uncaptionable dataset 400s without evicting the
  loaded chat/Images model.
- sd.cpp unload now waits out a cancelled one-shot generation on the generate
  lock before reporting the device free, matching the diffusers backend.
- Clearing a caption that came from metadata.jsonl writes an empty sidecar
  tombstone instead of unlinking (both readers treat an existing sidecar as
  authoritative), so the cleared label cannot resurface.
- The ControlNet wrapper pipe is only cached while its load is still current,
  closing the unload race the model cache already handled.
2026-07-05 04:53:37 +00:00
pre-commit-ci[bot]
f50e40c9d6 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:43:59 +00:00
Daniel Han
176a544e1c Merge branch 'diffusion-train-tab-2' into diffusion-krea2
# Conflicts:
#	studio/backend/core/inference/diffusion.py
2026-07-05 04:43:41 +00:00
pre-commit-ci[bot]
546dc9dce1 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:43:25 +00:00
pre-commit-ci[bot]
694178a35e [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:42:51 +00:00
pre-commit-ci[bot]
919661ddf0 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 04:42:18 +00:00
Daniel Han
40723c3465 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 04:41:58 +00:00
Daniel Han
7e3abefddd Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 04:41:57 +00:00
Daniel Han
39f9fa0fee Merge branch 'image-generation' into diffusion-train-perf 2026-07-05 04:41:56 +00:00
Daniel Han
62cae5fe71 Load image pipelines from the prefetched snapshot instead of re-sweeping the hub
The prefetch already scopes the file list (no packaged root singles, no
dtype-variant twins, no ONNX/Flax exports), but from_pretrained was then
called with the hub id, and its own snapshot sweep re-downloaded the
skipped files anyway: 24 GB per FLUX.1 repo and 65 GB on FLUX.2-dev, as
found in the blob cache. Return the snapshot dir from the prefetch (keyed
on the pipeline manifest) and hand it to every pipeline-assembly
from_pretrained site; any prefetch failure keeps the hub id and the old
behavior.
2026-07-05 04:41:43 +00:00
Daniel Han
740ff8454e Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 02:38:59 +00:00
Daniel Han
86b12b2213 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 02:38:57 +00:00
Daniel Han
aaa9007c37 Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 02:38:56 +00:00
Daniel Han
ad8213ba70 Merge branch 'image-generation' into diffusion-train-perf 2026-07-05 02:38:55 +00:00
pre-commit-ci[bot]
20a3650108 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 02:12:27 +00:00
Daniel Han
dc36983562 Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 02:11:25 +00:00
Daniel Han
a99b951c33 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2
# Conflicts:
#	studio/backend/tests/test_diffusion_training.py
2026-07-05 02:11:15 +00:00
pre-commit-ci[bot]
91d7297d41 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 02:10:58 +00:00
Daniel Han
32a77623ba Merge branch 'diffusion-train-perf' into diffusion-train-precision
# Conflicts:
#	studio/backend/core/training/diffusion_train_common.py
2026-07-05 02:10:23 +00:00
Daniel Han
c2ab1a0e61 Merge branch 'image-generation' into diffusion-train-perf
# Conflicts:
#	studio/backend/core/training/diffusion_dit_trainer.py
#	studio/backend/core/training/diffusion_train_common.py
2026-07-05 02:09:39 +00:00
pre-commit-ci[bot]
f6f198fd5f [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 02:07:49 +00:00
Daniel Han
25d9cf9604 Stub diffusers.hooks too in the no-diffusers cache test 2026-07-05 02:04:16 +00:00
Daniel Han
e605075508 Fix diffusion training validation and honor lr_scheduler and batch size in the DiT trainer 2026-07-05 01:51:34 +00:00
Daniel Han
098809d2fe Reject sd-cli batch runs and clear stale output targets before a run 2026-07-05 01:49:19 +00:00
Daniel Han
76eee534ea Gate explicit attention kernels on NVIDIA CUDA and roll back partial FBCache hooks 2026-07-05 01:48:12 +00:00
Daniel Han
f1d9c88606 Validate load modes before eviction and wait out a cancelled denoise on unload 2026-07-05 01:47:02 +00:00
Daniel Han
6a8b0b47e7 Fix review findings on image generation: failed-load VRAM, API defaults, preflights
- Free reserved VRAM in the diffusion load worker's failure path: a load-time OOM
  never commits _state and the next load's _unload_locked early-returns, so nothing
  else reclaimed the half-built pipeline's memory
- Use a monotonic clock for the denoise ETA rate
- Sync _GENERATION_DEFAULTS with the UI table: kontext, flux.2-dev, sdxl-turbo and
  SDXL base rows so /v1/images/generations stops falling back to 9 steps / CFG 0
- 400 (not sanitized 500) when /v1/images/generations hits an edit-only model
- Fail fast on pre-Ampere CUDA in the DiT trainer instead of dying in model load
- Run the trainer trust gate in the diffusion training route before freeing GPU
  residents so an untrusted base cannot tear down loaded chat/Images models
- Protect native sd.cpp companion VAE/text-encoder repos from cache deletion while
  a load is downloading them
- Exempt the task-scoped Images picker from the chat-only GGUF/MLX format gate so
  local diffusers pipelines stay selectable on no-GPU hosts
2026-07-05 01:00:47 +00:00
Daniel Han
99eb248607 Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 00:31:33 +00:00
Daniel Han
71c20ded19 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 00:31:31 +00:00