If you need an image to follow the exact structure of a reference photo, ControlNet is the tool. It extracts edges, poses, or depth to guide the generation process.
This tutorial shows you how to set it up in ComfyUI. It is simpler than it looks.
What You’ll Build
A workflow that:
- Loads a reference image (e.g., a product photo, architectural sketch, or portrait)
- Extracts edge structure using Canny edge detection
- Generates a new image guided by those edges + your text prompt
- Runs 100% locally: no cloud, no API costs
Use cases: Product visualization, architectural rendering, brand-consistent content, style transfer.
Prerequisites
- ComfyUI installed (Desktop app or manual)
- Stable Diffusion checkpoint from Stability AI (e.g.,
sd_xl_base_1.0.safetensorsor any SDXL model) - ControlNet model (e.g.,
control-lora-canny-rank256.safetensors): see the ControlNet collection on HuggingFace - ComfyUI ControlNet Aux node pack: install via ComfyUI Manager
flowchart LR
INPUT["Input Image"] --> CN["ControlNet<br/>Preprocessor"]
CN --> COND["Conditioning"]
PROMPT["Text Prompt"] --> COND
COND --> SAMPLER["KSampler"]
SAMPLER --> OUTPUT["Generated Image"]
style INPUT fill:#1E293B,color:#FAFAFA
style CN fill:#F5A623,color:#0B1628
style OUTPUT fill:#059669,color:#FAFAFAThe Workflow (11 Nodes)
Reference Image → Canny Edge Detection → ControlNet Apply
↓
Text Prompt → CLIP Encode ──────────────→ KSampler → VAE Decode → Save Image
↑
Checkpoint → Model + VAE ─────────────────────┘Node-by-Node Setup
| # | Node | Settings |
|---|---|---|
| 1 | CheckpointLoaderSimple | Select your SDXL or SD 1.5 checkpoint |
| 2 | ControlNetLoader | Load control-lora-canny-rank256.safetensors |
| 3 | LoadImage | Upload your reference image |
| 4 | CannyEdgePreprocessor | low_threshold: 100, high_threshold: 200, resolution: 1024 |
| 5 | CLIPTextEncode (positive) | Your prompt: “professional product photo, studio lighting” |
| 6 | CLIPTextEncode (negative) | “blurry, low quality, watermark, text” |
| 7 | ControlNetApplyAdvanced | strength: 0.8, start_percent: 0, end_percent: 1.0 |
| 8 | EmptyLatentImage | width: 1024, height: 1024, batch_size: 1 |
| 9 | KSampler | steps: 25, cfg: 7.5, sampler: euler_ancestral, scheduler: normal |
| 10 | VAEDecode | Connect to KSampler output |
| 11 | SaveImage | filename_prefix: “controlnet_“ |
Key Parameters
| Parameter | What it does | Recommended |
|---|---|---|
| strength (ControlNet) | How strictly to follow the edge guide | 0.7-0.9 for structure, 0.3-0.5 for loose guidance |
| low_threshold (Canny) | Minimum edge sensitivity | 100 (default) |
| high_threshold (Canny) | Maximum edge sensitivity | 200 (default) |
| cfg (KSampler) | How closely to follow the text prompt | 7-8 for balanced, 10+ for strict |
Tips for Best Results
- Start with strong ControlNet (0.8+) and reduce if results look rigid
- Use line art or architectural photos as references: clean edges work best
- Match resolution: if your reference is 1024x1024, set EmptyLatentImage to 1024x1024
- Experiment with ControlNet types: Canny (edges), Depth (3D), Pose (human), Scribble (sketches)
- SDXL models produce higher quality than SD 1.5 but need more VRAM (6+ GB)
Download the Workflow
Ready-to-import JSON file:
Download vorlux_controlnet_canny_guided.json →
Import in ComfyUI: Load Workflow → Upload File.
Hardware Requirements
| Hardware | VRAM | Speed | Quality |
|---|---|---|---|
| Mac Mini M4 (24GB) | 24GB unified | ~30s/image | Excellent |
| RTX 4060 Ti (16GB) | 16GB | ~15s/image | Excellent |
| RTX 3090 (24GB) | 24GB | ~10s/image | Excellent |
| Mac M3 Pro (32GB) | 32GB unified | ~25s/image | Excellent |
Common Issues and Fixes
Even experienced users run into problems with ControlNet. Here are the most frequent issues and their solutions:
| Problem | Cause | Fix |
|---|---|---|
| Output ignores reference image | ControlNet strength too low or node not connected properly | Verify the ControlNet Apply node is wired between the CLIP encode and KSampler. Set strength to 0.8+ as a starting point. |
| Output looks like a blurry copy | Strength too high combined with low CFG | Reduce ControlNet strength to 0.6-0.7 and raise CFG to 8-10 to give the text prompt more influence. |
| Black or blank output image | Mismatched model types (SD 1.5 ControlNet with SDXL checkpoint) | Ensure your ControlNet model matches your base checkpoint version. SDXL checkpoints need SDXL-compatible ControlNet models. |
| ”ControlNet model not found” error | Model file in wrong directory | Place .safetensors ControlNet files in ComfyUI/models/controlnet/, not in the checkpoints folder. |
| Canny edges too noisy or too sparse | Threshold values not tuned for your image | For photos with lots of detail, raise low_threshold to 150. For sketches or simple outlines, lower high_threshold to 150. Preview the Canny output node to check before generating. |
| Very slow generation (60s+) | Resolution mismatch or model loaded on CPU | Match EmptyLatentImage resolution to your reference image. Check ComfyUI console output to confirm the model loaded on GPU, not CPU. |
Tip: Always preview the Canny edge detection output before running the full generation. If the edges do not clearly capture the structure you want to preserve, adjust thresholds first. Generating with bad edges wastes compute time and produces unusable results.
Next steps
- Check your hardware requirements to ensure your VRAM can handle the generation.
- Follow the node-by-node setup to configure your CheckpointLoaderSimple and ControlNetLoader.
- Download the JSON workflow file to import the complete setup into ComfyUI.
- Review the tips for best results to match your reference resolution with your EmptyLatentImage settings.
Related resources
- 230 Workflows library: browse all workflows including 92 ComfyUI
- AI Models catalog: find Stable Diffusion and FLUX models
- Software stack: ComfyUI in our tech stack
- Contact: need help setting up ComfyUI?
Sources:
Related reading
- ComfyUI Batch Image Generation: Create 100 Product Images in Minutes
- n8n + MCP: Connect Your AI Agents to Any Business Tool
- How to Deploy AI Locally in Your Business: Complete 2026 Guide
Work with us
We size the model and the machine by measuring, not by guessing. If you want to see your own task running on real hardware, book a 15-minute call or see how we work in consulting.