Commit graph

297 commits

Author SHA1 Message Date
Daniel Han
2d974219bf Deploy Krea adapters on Turbo and use its distilled recipe over the API
- DiffusionFamily gains deploy_base_repo (krea/Krea-2-Turbo): deploying a LoRA
  trained on Raw now previews it on Turbo, not the non-distilled Raw checkpoint.
  Scoped to a same-precision override so it never turns an nf4 train base into a
  larger bf16 deploy load; exposed through family_train_infos -> the Train UI's
  onDeployClick / historical-run deploy resolve the deploy base.
- _GENERATION_DEFAULTS gains a Krea entry (8 steps, 0 CFG) so the OpenAI
  /v1/images/generations route matches the Create UI's documented distilled recipe
  instead of falling through to the generic (9, 0.0).
2026-07-06 11:16:25 +00:00
pre-commit-ci[bot]
d0f7dad7ec [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-05 07:58:51 +00:00
Daniel Han
4b93558ad4 Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 07:56:47 +00:00
Daniel Han
69f32b9586 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 07:56:20 +00:00
Daniel Han
3925aea07f Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 07:56:14 +00:00
Daniel Han
8c00f81a5d Size-gate the automatic diffusion latent cache
The latent cache holds two fp32 posterior tensors per crop/flip variant per
image, pinned on CUDA hosts, so datasets with thousands of images can exhaust
host or pinned memory with no fallback. Estimate the cache size from the first
real encoded latent and fall back to per-step VAE encoding when it exceeds a
4 GiB budget. UNSLOTH_DIFFUSION_FORCE_LATENT_CACHE bypasses the gate; the
existing UNSLOTH_DIFFUSION_NO_LATENT_CACHE opt-out is unchanged.
2026-07-05 07:53:12 +00:00
Daniel Han
da3a79468e Use permutation-cycle index sampling in diffusion trainers and guard non-object run records
Replace the with-replacement per-batch index draw in the SDXL and DiT LoRA
trainers with a shared PermutationBatchSampler that visits every image once per
cycle before repeating, so short runs cover the whole dataset. The sampler
reshuffles from the run's rng so the index stream stays seed-deterministic.

Guard the diffusion run detail route against a valid-JSON non-object record,
which previously raised TypeError and returned a 500; it now 404s like the list
path's shape check.

Add regression tests for both.
2026-07-05 07:49:30 +00:00
Daniel Han
e0786a98be Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 05:35:54 +00:00
Daniel Han
1978b2a844 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 05:35:53 +00:00
Daniel Han
1919491665 Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 05:35:52 +00:00
Daniel Han
d6795ed077 Restore pre-Ampere bf16 fail-fast in the DiT trainer
The perf rewrite dropped the bf16 capability guard, so a pre-Ampere CUDA
device (T4/V100/RTX 20xx) would die deep in model load with an opaque dtype
error instead of a clear message. Restores parity with the SDXL trainer.
2026-07-05 05:27:58 +00:00
Daniel Han
dc36983562 Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 02:11:25 +00:00
Daniel Han
a99b951c33 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2
# Conflicts:
#	studio/backend/tests/test_diffusion_training.py
2026-07-05 02:11:15 +00:00
Daniel Han
32a77623ba Merge branch 'diffusion-train-perf' into diffusion-train-precision
# Conflicts:
#	studio/backend/core/training/diffusion_train_common.py
2026-07-05 02:10:23 +00:00
Daniel Han
c2ab1a0e61 Merge branch 'image-generation' into diffusion-train-perf
# Conflicts:
#	studio/backend/core/training/diffusion_dit_trainer.py
#	studio/backend/core/training/diffusion_train_common.py
2026-07-05 02:09:39 +00:00
Daniel Han
e605075508 Fix diffusion training validation and honor lr_scheduler and batch size in the DiT trainer 2026-07-05 01:51:34 +00:00
Daniel Han
6a8b0b47e7 Fix review findings on image generation: failed-load VRAM, API defaults, preflights
- Free reserved VRAM in the diffusion load worker's failure path: a load-time OOM
  never commits _state and the next load's _unload_locked early-returns, so nothing
  else reclaimed the half-built pipeline's memory
- Use a monotonic clock for the denoise ETA rate
- Sync _GENERATION_DEFAULTS with the UI table: kontext, flux.2-dev, sdxl-turbo and
  SDXL base rows so /v1/images/generations stops falling back to 9 steps / CFG 0
- 400 (not sanitized 500) when /v1/images/generations hits an edit-only model
- Fail fast on pre-Ampere CUDA in the DiT trainer instead of dying in model load
- Run the trainer trust gate in the diffusion training route before freeing GPU
  residents so an untrusted base cannot tear down loaded chat/Images models
- Protect native sd.cpp companion VAE/text-encoder repos from cache deletion while
  a load is downloading them
- Exempt the task-scoped Images picker from the chat-only GGUF/MLX format gate so
  local diffusers pipelines stay selectable on no-GPU hosts
2026-07-05 01:00:47 +00:00
Daniel Han
99eb248607 Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-05 00:31:33 +00:00
Daniel Han
71c20ded19 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-05 00:31:31 +00:00
Daniel Han
79b97e9ad0 Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-05 00:31:30 +00:00
Daniel Han
ad7d5c6827 Coerce cache_latents and enable_tf32 string flags in the config dict path
The generic Studio config dict path can deliver these flags as strings, and a
non-empty string like "false" is truthy, so an opt-out silently no-ops (the
latent cache still builds, TF32 stays on). Coerce them the same way
gradient_checkpointing already is.
2026-07-05 00:12:47 +00:00
pre-commit-ci[bot]
e6d775d0fc [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-04 05:08:43 +00:00
Daniel Han
8324cdc407 Merge diffusion-train-tab-2: CI test fixes (diffusers import order, arbiter device pin, sigma-gather skip) 2026-07-04 05:08:18 +00:00
Daniel Han
3fcf218614 Merge diffusion-train-precision: CI test fixes (diffusers import order, arbiter device pin, sigma-gather skip) 2026-07-04 05:07:46 +00:00
Daniel Han
f5d5b09ae5 Merge diffusion-train-perf: CI test fixes (diffusers import order, arbiter device pin, sigma-gather skip) 2026-07-04 05:06:59 +00:00
Daniel Han
c63df7d918 Merge image-generation: CI test fixes (diffusers import order, arbiter device pin) 2026-07-04 05:05:54 +00:00
Daniel Han
1d3aa53d1f Validate the training config before importing diffusers and pin the arbiter test's device
The fp16-on-bf16-family refusal in run_dit_lora_training now fires before the heavy
imports, so a host without diffusers gets the real validation error instead of
ModuleNotFoundError. test_in_progress_returns_409_after_validation_passes pins the
resolved device to cuda because the load route only takes the GPU arbiter for non-CPU
loads, which made the ownership assert host-dependent.
2026-07-04 05:01:58 +00:00
pre-commit-ci[bot]
6ecbb2b8a3 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-04 04:41:56 +00:00
pre-commit-ci[bot]
96cecf9c47 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-04 04:41:23 +00:00
pre-commit-ci[bot]
d13ce4c74a [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-04 04:40:51 +00:00
pre-commit-ci[bot]
285c8fbd20 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-07-04 04:40:19 +00:00
Daniel Han
a82fc89d03 Merge diffusion-train-tab-2: grad norm reconciliation + review fixes 2026-07-04 04:38:05 +00:00
Daniel Han
a346a0eb20 Merge diffusion-train-precision: grad norm chart + review fixes
# Conflicts:
#	studio/backend/core/training/diffusion_dit_trainer.py
#	studio/backend/core/training/diffusion_lora_trainer.py
#	studio/backend/core/training/diffusion_training_service.py
#	studio/backend/models/training.py
#	studio/frontend/src/features/images/api.ts
#	studio/frontend/src/features/images/train/diffusion-charts.tsx
#	studio/frontend/src/features/images/train/diffusion-train-panel.tsx
2026-07-04 04:37:58 +00:00
Daniel Han
84a661b363 Merge diffusion-train-perf: grad norm chart + review fixes 2026-07-04 04:33:37 +00:00
Daniel Han
223a546cd8 Merge image-generation: grad norm chart, completion state, Windows caption keys, GGUF compute copy
# Conflicts:
#	studio/backend/core/training/diffusion_dit_trainer.py
#	studio/backend/core/training/diffusion_lora_trainer.py
#	studio/backend/core/training/diffusion_training_service.py
2026-07-04 04:33:30 +00:00
Daniel Han
2c5955bda8 Coerce num_epochs in normalized() and use utf-8 for run records
num_epochs was only int-coerced for the range check, so a string value
from a dict-built config would reach resolve_train_steps' arithmetic;
normalized() now stores the coerced int. Run record reads/writes pass
encoding utf-8 explicitly so non-ASCII prompts survive on Windows
2026-07-04 04:31:07 +00:00
Daniel Han
8ad8a58742 Add grad norm chart, clearer completion state, Windows caption keys, GGUF compute copy
- Trainers emit the pre-clip gradient norm; the service keeps a bounded
  grad_norm history and the Train tab renders a Grad Norm chart next to
  Loss and LR
- Completed runs show 'Training complete' with a celebratory marker in
  the success color instead of a plain status word
- metadata.jsonl caption keys now match on Windows (as_posix relative
  paths) in both the trainer discovery and the dataset image records
- RMSNorm eager patch skips installation on torch builds without
  F.rms_norm instead of failing at forward time
- GGUF compute description no longer says the GGUF is dequantised: the
  INT8/FP8/FP4 modes load the base model's bf16 transformer and quantise
  that directly; label no longer wraps in the Advanced panel
2026-07-04 04:31:04 +00:00
Daniel Han
2eded64b25 Merge diffusion-train-tab-2: run history robustness, epoch sentinel, torchao probe, image-generation review fixes 2026-07-04 03:25:30 +00:00
Daniel Han
7a8f363c52 Merge diffusion-train-precision: torchao functional probe + image-generation review fixes 2026-07-04 03:25:08 +00:00
Daniel Han
a4a38b5672 Merge diffusion-train-perf: image-generation review fixes 2026-07-04 03:24:44 +00:00
Daniel Han
7d6c022489 Merge image-generation: review fixes (engine unload, caption precedence, dataset counts, local diffusers tagging, family LoRA targets)
# Conflicts:
#	studio/backend/core/training/diffusion_dit_trainer.py
2026-07-04 03:24:38 +00:00
Daniel Han
dc290bdf71 Train Krea 2 LoRAs on the undistilled Raw checkpoint by default
Krea's release guidance is to train on Krea-2-Raw and run adapters on
Turbo. Raw now leads the krea-2 training bases (Turbo stays available),
both vendor repos are trust-listed, and load_krea2_pipeline fails fast
with an upgrade hint on diffusers older than 0.39 instead of a bare
AttributeError mid-load
2026-07-04 03:23:30 +00:00
Daniel Han
51de9da488 Fix review findings: run history robustness, epoch-mode sentinel, seed 0, history refresh race
- list_diffusion_runs skips wrong-shape records and the runs route tolerates
  per-record ValidationError so one bad file never breaks the panel
- max_steps: 0 epoch-mode sentinel no longer trips train_steps validation
  before epochs are resolved
- numberField keeps an explicit 0 (Seed, LR warmup) instead of falling back
- previous-runs list refetches once more shortly after a terminal status so
  the just-finished run appears even if the record write races the fetch
2026-07-04 03:23:20 +00:00
Daniel Han
89d99e31af Gate int8 and fp8 on a functional torchao import, not find_spec
The Windows ROCm torchao import stub satisfies find_spec and even lets
from torchao.quantization import quantize_ succeed, but its quantize_ is a
no-op: auto would pick int8, leave the transformer dense, and disable
compile as if it were quantized. has_functional_torchao imports the exact
symbols the int8 path uses and rejects the stub via its sentinel; both the
auto picker and the /info advertised modes now use it
2026-07-04 03:23:16 +00:00
Daniel Han
bfbb902610 Fix review findings: engine unload before training, caption precedence, dataset caption counts, local diffusers tagging, family LoRA targets
- Unload the ACTIVE image engine (sd_cpp or diffusers) before diffusion training starts, not just the diffusers singleton
- Count metadata.jsonl captions in dataset summaries so metadata-captioned datasets are not reported as uncaptioned
- Sidecar captions now override metadata rows everywhere (grid edits win); trainer and dataset API agree
- Tag local diffusers image checkpoints with text-to-image so they appear in the Images picker
- Family LoRA targets (_FLUX_TARGETS etc) apply when the config carries the generic defaults; explicit overrides still win
2026-07-04 03:23:05 +00:00
Daniel Han
ea9f7ae9f9 Merge branch 'diffusion-train-tab-2' into diffusion-krea2 2026-07-04 02:21:20 +00:00
Daniel Han
f1dbb74308 Merge branch 'diffusion-train-precision' into diffusion-train-tab-2 2026-07-04 02:21:11 +00:00
Daniel Han
1a99c81f59 Merge branch 'diffusion-train-precision' of https://github.com/unslothai/unsloth into diffusion-train-precision 2026-07-04 02:21:03 +00:00
Daniel Han
bdebf61e28 Merge branch 'diffusion-train-perf' into diffusion-train-precision 2026-07-04 02:20:54 +00:00
Daniel Han
f76d7c8d2f Merge remote-tracking branch 'origin/image-generation' into diffusion-train-perf 2026-07-04 02:20:44 +00:00