ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
unsloth/studio/backend/core/inference
History
Daniel Han b24a94d1c2 Wire the Mega-cache compile prewarm into the video backend 
The pre-warmed torch.compile cache (diffusion_compile_cache.py: fingerprinted
bundles over torch.compiler.save/load_cache_artifacts plus a persistent per-key
TORCHINDUCTOR_CACHE_DIR) was wired into the image backend only, so every video
load re-paid the full first-generation compile after every process restart (the
stock inductor dir lives in /tmp).

video.py now mirrors the image backend exactly: compile_cache.begin runs after
the attention-backend set (the fingerprint keys on the engaged kernel) and
before apply_speed_optims on a compile-eligible default/max tier, keyed on the
same fullgraph decision as the compile itself (an engaged or still-toggleable
step cache and a planned offload both drop it); the context is committed to
_VideoLoadState and compile_cache.save persists the bundle after the first
successful generation (env-gated distributor / first-run-warm mode);
_teardown_state restores the inductor dir, and a token-scoped
_rollback_precommit_compile_cache covers loads that die before the state
commit, mirroring the globals and CFG-parallel rollbacks.

Measured on HunyuanVideo-1.5-480p through the real VideoBackend (B200,
480x288/17f/30 steps): the first-generation extra drops from 107.5 s cold to
13.8 s when the 12.8 MB bundle loads into a fresh inductor dir (0.10 s load)
and to 11.7 s from the persistent per-key dir alone; through the wired
production path a restart lands at 10.5-10.8 s (bundle-only included) vs
86.5 s cold. Steady state is unchanged (2.4-2.6 s), and the loaded artifacts
are the same bits a local compile would produce, so numerics are untouched.

Tests: begin/save/restore lifecycle with the fullgraph keying, the Speed=off
and compile-ineligible skips, and the token-scoped pre-commit rollback.
2026-07-11 06:06:26 +00:00
..
__init__.py Studio: stop chat generation on the assistant-turn-end token (fixes Qwen3.5 loop) (#6804) 2026-07-06 10:07:56 -07:00
_html_to_md.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
_vulkan_probe.py Studio: add Vulkan llama.cpp support (#5819) 2026-07-09 03:39:48 -07:00
anthropic_compat.py Studio: stream reasoning tokens in the tool-loop generator (fixes DeepSeek thinking not streaming with a pill on) (#6947) 2026-07-07 19:50:40 -03:00
api_monitor.py Studio: trim serving-log noise and surface llama-server engine stats (#6377) 2026-06-17 05:37:57 -07:00
audio_codecs.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
chat_eos.py Studio: stop chat generation on the assistant-turn-end token (fixes Qwen3.5 loop) (#6804) 2026-07-06 10:07:56 -07:00
chat_template_helpers.py Studio: render thinking blocks for safetensors inference with prefilled <think> templates (#6816) 2026-07-08 08:14:03 -07:00
chat_templates.py Studio: bundle Gemma 4 chat templates (E2B/E4B + larger) and auto-apply to unsloth/gemma-4-*-GGUF (#6245) 2026-06-12 05:49:39 -07:00
defaults.py Add DeepSeek-V4-Flash-GGUF to Studio with none/high/max reasoning (#6908) 2026-07-07 06:13:43 -07:00
diffusion.py fix(review): portable bench scripts, accurate VAE auto docs, explicit TE deny 2026-07-10 06:13:48 +00:00
diffusion_arch_patches.py Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI redesign (#6769) 2026-07-03 13:45:14 -03:00
diffusion_attention.py Merge remote-tracking branch 'origin/image-generation' into r7021 2026-07-09 11:09:08 +00:00
diffusion_auto_policy.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2026-07-09 08:53:52 +00:00
diffusion_cache.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2026-07-10 17:45:30 +00:00
diffusion_cfg_parallel.py Gate auto CFG parallel on Speed=off 2026-07-10 23:02:17 +00:00
diffusion_compile_cache.py Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI redesign (#6769) 2026-07-03 13:45:14 -03:00
diffusion_controlnet.py diffusion: address review round (FBCache context guard, aiter/ROCm, video cleanup, prequant + ControlNet gating) 2026-07-09 08:52:16 +00:00
diffusion_device.py Trim the verbose comments added by the fixes 2026-07-04 13:25:08 -03:00
diffusion_eager_patches.py Add grad norm chart, clearer completion state, Windows caption keys, GGUF compute copy 2026-07-04 04:31:04 +00:00
diffusion_engine_router.py Validate a local video checkpoint's own suffix; unload the old engine before publishing the new one 2026-07-07 22:55:36 +00:00
diffusion_families.py Merge remote-tracking branch 'origin/diffusion-more-families' into fold-integration 2026-07-07 01:08:50 +00:00
diffusion_gguf_compile.py Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI redesign (#6769) 2026-07-03 13:45:14 -03:00
diffusion_ideogram4.py ideogram-4: build the FP8 text encoder at target dtype, size it as bf16-resident, optimize both DiTs 2026-07-06 10:21:10 +00:00
diffusion_inference_info.py Advertise per-family footprints and surface Auto badges for resolved controls 2026-07-04 07:47:01 +00:00
diffusion_krea2.py diffusion: add AGPL-3.0 SPDX header to Krea2 / LoRA / ControlNet files 2026-07-09 06:46:45 +00:00
diffusion_lora.py diffusion: add AGPL-3.0 SPDX header to Krea2 / LoRA / ControlNet files 2026-07-09 06:46:45 +00:00
diffusion_memory.py Merge image-generation bug fixes (#6872) 2026-07-07 16:14:06 +00:00
diffusion_patch_backend.py Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI redesign (#6769) 2026-07-03 13:45:14 -03:00
diffusion_precision.py Merge branch 'image-generation' into video-diffusion-improvements 2026-07-10 17:44:25 +00:00
diffusion_prequant.py Merge remote-tracking branch 'origin/image-generation' into r7021 2026-07-09 11:09:08 +00:00
diffusion_speed.py Merge branch 'image-generation' into video-diffusion-improvements 2026-07-10 17:44:25 +00:00
diffusion_transformer_quant.py perf(video): accuracy-first round 2 for HunyuanVideo-1.5: compile parity, cache quality presets, dual-GPU CFG 2026-07-10 14:30:00 +00:00
diffusion_vae_quant.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2026-07-10 08:02:46 +00:00
external_provider.py Studio: harden background consumer loops and streaming paths against silent UI freezes (#6653) 2026-06-26 03:31:33 -07:00
gpu_arbiter.py Video inference engine: LTX-2 family registry, VideoBackend, MP4 gallery 2026-07-04 13:08:43 +00:00
image_gallery.py Studio diffusion (Phase 1): cross-platform device policy, fp16 guard, lock split, validate-before-evict (#6670) 2026-06-30 16:33:47 -03:00
inference.py Studio: render thinking blocks for safetensors inference with prefilled <think> templates (#6816) 2026-07-08 08:14:03 -07:00
key_exchange.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
llama_cpp.py Studio: harden OpenAI-compatible GGUF streaming (#6950) 2026-07-09 12:09:08 -03:00
llama_http.py fix(studio/llama_cpp): disable trust_env on the loopback health probe (#6750) (#6752) 2026-06-30 19:09:26 +02:00
llama_keepwarm.py Run video generation as a background job so secure mode's tunnel cap cannot 524 it 2026-07-10 09:19:02 +00:00
llama_server_args.py studio: return a clean model id from the OpenAI API instead of the local .gguf path (#6518) 2026-06-26 16:07:53 -03:00
llama_stats.py Studio: trim serving-log noise and surface llama-server engine stats (#6377) 2026-06-17 05:37:57 -07:00
local_model_resolver.py Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm (#6392) 2026-07-01 06:42:23 -07:00
mcp_client.py Studio: enable stdio MCP servers on a loopback bind (#6295) 2026-06-15 03:02:32 +01:00
mcp_config_import.py studio: show MCP "Import config" on the add-server form (#6030) 2026-06-11 16:17:22 +01:00
message_content.py fix(studio): handle multimodal list content in inference text paths (#4383) (#6480) 2026-06-23 01:26:11 -07:00
mlx_inference.py Studio: render thinking blocks for safetensors inference with prefilled <think> templates (#6816) 2026-07-08 08:14:03 -07:00
model_ids.py studio: list the full local model catalog from /v1/models (#6519) 2026-06-26 20:42:06 -03:00
orchestrator.py feat(cli): support MLX distributed inference (#6845) 2026-07-08 03:25:39 -07:00
passthrough_healing.py Studio: parse Mistral [TOOL_CALLS] and rehearsal tool-call shapes (#5704) 2026-07-06 18:52:13 -07:00
presence_penalty.py Studio: apply presence_penalty on the safetensors and MLX inference paths (#6923) 2026-07-06 22:24:47 -07:00
pricing.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
providers.py Studio: Add custom provider option to Connections (#6112) 2026-06-12 13:09:35 +02:00
runtime_context.py Expose runtime context length for hub models (#6154) 2026-06-11 22:13:53 +03:00
safetensors_agentic.py Studio chat: tool-call nudging on by default (API stays opt-in) (#6883) 2026-07-06 19:41:19 -07:00
sd_cpp_args.py Reject sd-cli batch runs and clear stale output targets before a run 2026-07-05 01:49:19 +00:00
sd_cpp_backend.py diffusion: accept vae_quant in the native sd.cpp backend load interface 2026-07-09 07:58:30 +00:00
sd_cpp_engine.py Merge remote-tracking branch 'origin/image-generation' into fix/imggen-review-bugs 2026-07-06 09:16:30 -03:00
sd_cpp_server.py Studio diffusion: LoRA adapters for the Images workflow (#6771) 2026-07-03 15:42:56 -03:00
tensor_fallback.py studio: deterministic VRAM auto-fit for GGUF (MTP reserve, compute buffer, total-based budget) (#6312) 2026-06-17 03:10:22 -07:00
tool_call_parser.py Studio chat: tool-call nudging on by default (API stays opt-in) (#6883) 2026-07-06 19:41:19 -07:00
tool_loop_controller.py Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) 2026-06-09 21:17:04 -07:00
tools.py Whole-document context for RAG chat attachments (#6693) 2026-06-30 15:55:23 +02:00
video.py Wire the Mega-cache compile prewarm into the video backend 2026-07-11 06:06:26 +00:00
video_families.py Run video generation as a background job so secure mode's tunnel cap cannot 524 it 2026-07-10 09:19:02 +00:00
video_gallery.py Delete gallery videos MP4-first so a locked MP4 can't orphan the record 2026-07-07 11:57:37 +00:00
video_ltx2.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2026-07-04 14:47:26 +00:00
worker.py feat(cli): support MLX distributed inference (#6845) 2026-07-08 03:25:39 -07:00