- The trainer keyed its persistent conditioning cache on family and
resolution only, while the keys themselves carry just the caption or
image content and crop variant. One cache directory reused for two
checkpoints, or for the same repo at a new revision, let a warm run
skip loading its encoders and train on the other model's embeddings
and latent statistics. Namespace on the base checkpoint and its
resolved revision as well. The revision helper now lives beside the
cache in diffusion_train_extras and the inference wrapper delegates to
it, so the two cannot disagree about what counts as the same source.
- The diffusion learning rate only checked positivity, but 1e309 floats
to inf and satisfies gt, so the route evicted the resident models and
started AdamW with an infinite rate: the first step destroys the
adapter while progress looks normal and the result is saved. Bound it
below 1.0, matching the LLM schema, which rejects inf for the same
reason.
diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only
the trainable adapter params (warmup-ramped decay, default 0.99, exported as a
second adapter under output_dir/ema), PersistentConditioningCache stores latent
posterior stats and caption embeddings as safetensors keyed by content hash +
family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect
datasets into same-area divisor-snapped shapes.
The DiT trainer wires the first two behind config flags that default to the
current behavior: ema_decay (0 disables) and cond_cache_dir (None disables).
A fully warm cache skips loading the VAE and text encoders entirely; a cache
hit is bit-identical to a fresh encode, including the per-channel qwen latent
normalization. Also fixes the stale _gather_sigmas call in the perf test that
still passed the scheduler instead of the sigma table.
