unsloth/plans/controlnet-workflow.md

Plan: ControlNet for the Studio Images workflow (stacked on #6769)

Context & research

ControlNet is the #2 most-used "beyond text-to-image" diffusion workflow after LoRA (now shipped in #6771). It conditions generation on a spatial control map so the output follows a structure. Web research (this session) on ComfyUI / Forge / A1111 usage:

• The dominant control types are Depth, Canny (edges), and OpenPose (human pose).
• The biggest 2025 shift is toward Union / all-in-one ControlNets that bundle many control modes in one model: InstantX / Shakker-Labs FLUX.1-dev-ControlNet-Union-Pro for FLUX, and xinsir/ controlnet-union-sdxl-1.0 for SDXL. SDXL has no official ControlNet (community: xinsir, TheMistoAI, BRIA). SD1.5 has the original lllyasviel set.
• Sources: comfyui-wiki.com ControlNet collections (flux-1 / sdxl), stable-diffusion-art.com ControlNet ComfyUI, education.civitai.com ControlNet guide, stablediffusiontutorials.com Qwen-Image ControlNets.

Both Studio backends can do ControlNet (verified against the live tree):

• diffusers has FluxControlNetPipeline / FluxControlNetModel, StableDiffusionXLControlNetPipeline / ControlNetModel / ControlNetUnionModel, QwenImageControlNetPipeline, FluxControlNetInpaintPipeline (diffusers 0.38 in the studio venv). Also FluxControlPipeline (the Flux.1-Canny/Depth "Control" in-model variants).
• native sd.cpp (stable-diffusion.cpp, the #6769 sd-cli base) has --control-net <path>, --control-image <path>, --control-strength <f>, --control-net-cpu, and a built-in preprocess_canny (examples/cli/main.cpp:704, examples/common/common.cpp:422+).

Studio has no ControlNet wiring today: diffusion_families.py has no controlnet fields and sd_cpp_args.py has no --control-*. This adds it, mirroring the LoRA architecture (#6771) and the existing img2img/inpaint/reference workflow patterns.

Scope (this PR = diffusers ControlNet, Union-first)

Ship the highest-value slice first; keep it shippable and consistent with the shipped LoRA design.

• In scope: diffusers ControlNet for the families with the strongest ecosystems and pipeline support: FLUX (FluxControlNetPipeline + Union Pro), SDXL (StableDiffusionXLControlNetPipeline + xinsir Union), Qwen-Image (QwenImageControlNetPipeline). Single ControlNet per generation. A control_type hint (canny / depth / pose / tile / passthrough). Canny preprocessing built in (cheap, cv2/PIL) plus passthrough for user-supplied control maps (depth/pose maps made elsewhere, matching ComfyUI where preprocessing is separate). Strength + guidance start/end. Discovery endpoint + family-gated picker.
• Out of scope (follow-ups): native sd.cpp ControlNet (--control-net, needs GGUF ControlNet assets + family support probe); heavy preprocessors (Depth-Anything, OpenPose detector) as optional server-side auto-preprocess; multi-ControlNet stacking; ControlNet + inpaint combo.

Key facts (verified)

• diffusers ControlNet pipelines are built with Pipeline.from_pipe(base_pipe, controlnet=cn_model) (or from_pretrained(base, controlnet=...)), so the resident base modules are reused with no reload -- same from_pipe machinery the img2img/inpaint/edit workflows already use (diffusion.py ~:1104-1130, _workflow_pipe). The ControlNet model (ControlNetModel / FluxControlNetModel / ControlNetUnionModel) is a small extra module loaded once and cached on _LoadState.
• ControlNet models are family-specific (SD1.5 CN != FLUX CN != SDXL CN). So discovery must be family-gated, exactly like the LoRA picker's supports_lora/family filter.
• Generate-time contract mirrors reference/inpaint: a control image (b64) + params, threaded through routes/inference.py into both backends (diffusers serves it; native rejects clearly until the follow-up wires --control-*).
• Reuse: diffusion_lora.py discovery/resolve/family-gate patterns; the reference-image upload component + the LoRA picker UI shape; _workflow_pipe from_pipe; hf_hub_download_with_xet_fallback.

Approach

Families (core/inference/diffusion_families.py)

• Add per-family ControlNet declaration: controlnet_pipeline_class (e.g. "FluxControlNetPipeline", "StableDiffusionXLControlNetPipeline", "QwenImageControlNetPipeline"), controlnet_model_class ("FluxControlNetModel" / "ControlNetModel" / a union class), and a small curated list of recommended ControlNet repos tagged by control type. Expose a controlnet: bool capability (like reference).

Discovery (core/inference/diffusion_controlnet.py, new -- mirrors diffusion_lora.py)

• list_controlnets(family) = curated family-tagged repos + a local scan; resolve_controlnet(id, family, hf_token) downloads via the xet-fallback helper; preprocess_control(image, control_type) (canny via cv2/PIL; passthrough otherwise); supports_controlnet(engine, family, model_kind, quant) gate (diffusers bf16 / bnb-4bit yes; GGUF-via-diffusers + torchao fp8/int8 dense = no, same rule as LoRA; native = follow-up).

Backend -- diffusers (core/inference/diffusion.py)

• A ControlNet manager parallel to _apply_loras: load the requested ControlNetModel once (cache on _LoadState, reset on unload/model change), build the CN pipeline via from_pipe(base, controlnet=...) in _workflow_pipe, and pass control_image + controlnet_conditioning_scale + control_guidance_start/end at generate time. Never fuse; CN model stays bf16.

Backend -- native (core/inference/sd_cpp_backend.py, sd_cpp_args.py) -- FOLLOW-UP

• Add --control-net / --control-image / --control-strength to the arg builder and a GGUF-ControlNet resolve; gate to families sd.cpp supports. Deferred out of this PR.

Routes + request models (models/inference.py, routes/inference.py)

• Add optional controlnet: ControlNetSpec to DiffusionGenerateRequest ({id, image, control_type, strength (0..2, default 1), guidance_start (0..1), guidance_end (0..1)}); thread into backend.generate; surface supports_controlnet in status; persist the chosen CN + type in gallery recipe metadata.
• New GET /api/models/diffusion-controlnets?family= (mirror the LoRA discovery route).

Frontend (features/images/images-page.tsx, api.ts)

• A "ControlNet" control in the left rail (reuse the reference-image uploader + a control-type Select + ControlNet-model Select gated by supports_controlnet/family + a strength SliderField). Show a small preview of the preprocessed control map. Thread controlnet into generateDiffusionImage; omit when no control image.

Verification

• Unit: request validation (optional/empty unchanged; bad strength rejected; unsupported family/quant rejected); discovery (family filter, resolve, canny preprocess shape); diffusers manager (loads CN once, from_pipe built, scale threaded, reset on model change) with a fake pipe; routes (no-CN path unchanged).
• Live smoke (critical): on GPU 4, drive the real diffusers backend with a real family + Union CN and a canny control image; same prompt/seed at strength 0 vs 0.8; assert (a) output DIFFERS from no-control and (b) the strong-control output structurally follows the control map (edge-overlap / SSIM vs the control).
• Playwright (unsloth_studio_workflow): upload a control image, pick type + model + strength, generate; capture screenshots/GIF against the live secure studio.
• Full pytest studio/backend/tests/ green; frontend vite build clean; ruff clean.

Delivery

• New branch diffusion-controlnet off diffusion-image-workflows (#6769 head) in an isolated worktree, sibling to #6771 (LoRA) and #6772 (fp8 fix). PR on unslothai/unsloth, base = diffusion-image-workflows, part of the single logical stack rooted on #6763 (continuation of #6658). Commit as Daniel Han; no AI/bot mentions, no emojis, no em dashes.
• Follow-ups: native sd.cpp ControlNet; server-side Depth/OpenPose auto-preprocessors; multi-ControlNet; ControlNet+inpaint.