1b4a66dcca
Text-to-video lands as a SIBLING of the image diffusion backend, not a mode of
it: video pipelines take frame/fps arguments, return frame stacks plus, for
LTX-2, synchronized audio, and persist MP4s -- none of the image module's
img2img/inpaint/ControlNet/LoRA surface applies. The image backend's hardware
and optimisation layers are imported unchanged (device/dtype resolution, memory
planning + offload tiers, attention backends, speed profiles, FBCache), and the
load-token/cancel-event concurrency skeleton is copied verbatim so lifecycle
behaviour cannot diverge.
core/inference/video_families.py: a pure VideoFamily registry (no torch) with
the ltx-2 entry -- LTX2Pipeline + LTX2VideoTransformer3DModel, base
Lightricks/LTX-2, unsloth/LTX-2.3-GGUF as the curated GGUF source, audio on,
frame lattice k*8+1, /32 resolutions with a vertical preset, and measured bf16
component sizes (the Gemma3-27B text encoder outweighs the 19B DiT itself).
MoE fields (transformer_2, guidance_scale_2) are declared now so the Wan2.2
A14B family lands later without churning the schema.
core/inference/video.py: VideoBackend with async begin_load + cache-scan
download progress, GGUF / single-file / full-pipeline loads (the GGUF DiT
assembles onto the base repo exactly like the image path), generation with
frame/size snapping BEFORE latents allocate, per-step progress + ETA and
cooperative cancel via the standard diffusers callback, and MP4 (H.264) export
through diffusers' PyAV encoder with the audio track muxed when the family
produces one. VAE tiling is always on: decoding a 100+ frame clip is the
memory peak, and the frames-aware estimate_video_runtime_mib (new, in
diffusion_memory) feeds the planner where the pixel-area image estimate would
badly undershoot. Loads are gated to unsloth/*, the official Lightricks base
repos, or local paths; PyAV availability is checked at load time so a missing
encoder cannot fail a clip after a multi-minute denoise.
core/inference/video_gallery.py: {id}.mp4 + {id}.json recipe sidecar pairs
under studio_root()/videos (an MP4 has no PNG text chunk to embed the recipe
in), with the image gallery's id/containment guards, newest-first listing that
skips orphans, delete/clear.
gpu_arbiter gains the VIDEO owner: ownership is exclusive, so the existing
evict-the-current-owner already generalises to chat/image/video all evicting
each other. The av (PyAV) dependency joins requirements/studio.txt.
Tests: video family detection/snapping/defaults, backend lifecycle on a faked
torch/diffusers runtime (GGUF assembly, shape snapping, distilled defaults,
cancel/progress, sentinel), gallery roundtrip/containment/orphans. 52 new
tests green plus the arbiter suite.
37 lines
1.3 KiB
Text
37 lines
1.3 KiB
Text
# Studio UI backend dependencies
|
|
typer
|
|
fastapi
|
|
uvicorn
|
|
pydantic
|
|
packaging
|
|
matplotlib==3.10.9
|
|
pandas
|
|
nest_asyncio
|
|
datasets==4.3.0
|
|
pyjwt
|
|
# gradio>=4.0.0 # 148 MB - Studio uses React + FastAPI, not Gradio
|
|
huggingface-hub==0.36.2
|
|
structlog>=24.1.0
|
|
diceware
|
|
ddgs
|
|
cryptography>=42.0.0
|
|
boto3>=1.34.0 # optional: S3 dataset loading
|
|
httpx>=0.27.0
|
|
fastmcp>=3.0.2
|
|
# Local image generation (Images page): diffusers' GGUFQuantizationConfig needs
|
|
# the gguf reader. diffusers itself is torch-bound and comes via the ML stack;
|
|
# gguf is torch-free so it lives here, where it's installed in every mode.
|
|
gguf
|
|
# Local video generation (Video page): MP4 encode (H.264 + AAC mux for LTX-2's
|
|
# synchronized audio) runs through PyAV, which bundles its own ffmpeg libs. The
|
|
# video load path fails fast with a clear error when this is missing, so keep it
|
|
# in the base studio set rather than an extra.
|
|
av>=12.0.0
|
|
# RAG (knowledge bases, hybrid retrieval). sentence-transformers lives in
|
|
# extras-no-deps.txt; these add the lexical+dense store and document parsing.
|
|
sqlite-vec==0.1.9
|
|
pymupdf==1.27.2.3
|
|
# 0.3.x keeps pymupdf-layout (which pulls onnxruntime) an optional extra; the
|
|
# lockstep 1.27.x line makes it a hard dep we do not need for to_markdown().
|
|
pymupdf4llm==0.3.4
|
|
python-docx==1.2.0
|