ControlNet - Walkthrough

What is ControlNet and Why is it Unique and Important in the AI World?

ControlNet is a significant advancement in AI technology, offering control over compositions in a manner not yet adopted by major industry players. Although it has been available in the open-source community for over a year, it continues to evolve and provide new capabilities.

What is ControlNet?

You might recall viral videos of dancing statues. If you’ve ever wondered how they are created, ControlNet likely played a role. While one method involves transforming every frame via image-to-image conversion, a more probable scenario is the use of ControlNet.

Example:

Consider wanting to generate an image of "a man sitting on a bench." Typically, you would describe the scene in a prompt, specifying the man's position and pose. However, the resulting images often place the man incorrectly, necessitating numerous attempts to achieve the desired outcome. ControlNet simplifies this by analyzing an image, tracing the person's pose (using ControlNet’s "Human Pose"), and using a stick figure as a reference for generation.

Video Example:

Using ControlNet in Cogniwerk's Image Generation:

There are four ways to access ControlNet:

1. Select "ControlNet" on the left sidebar.

2. Activate "ControlNet" on the right settings bar above the create button.

3. Drag and drop a reference image, click the little brush, and activate it there.

4. Use an image from storage and go via "continue workflow >> ControlNet."

In all cases, you should see a reference image next to the create button. Preprocessing typically starts automatically with ControlNet’s "Soft Edge," which detects the image's outline. For our example, we will select "Human Pose."

Next, insert a prompt as usual. For example: "An old farmer in Pixar animation style standing in front of his farm shop."

If the farmer needs to be on the right side of the image, adjust the pose and hit create again.

 >

Advanced Settings:

• Control Weight: This setting determines how much influence the reference image has on the generation. A value of zero means no influence, while a value of one ensures precision. Values above one can create artifacts, so it is best to stay at or below one.

• ControlNet Influence: This setting defines when the influence of the reference image starts and finishes during the generation process. Typically, the default settings are sufficient, but specific use cases might require adjustments, especially when using the "Soft Edge" ControlNet.

   

   

Other ControlNet Types

Our current ecosystem includes three types of control elements: human pose, soft edge, and text in image.

• HumanPose requires an image of a person to recognize and use their pose as a guide for image generation. It doesn't work with images of animals, buildings, or other non-human subjects.

• SoftEdge: For cases where the image doesn’t contain humans, such as with animals or buildings, the soft edge element might be more suitable. It uses an image to trace its structure and generates an inverted line drawing representation which can then be used as the guide for image generation. 

• Text In Image is a special variant of Soft Edge and comes with its own small interface, which lets you create a text first which gets preprocessed as a guiding image. It works also exceptionally well if you describe the text in the prompt as well, like "A photo of burning letters 'FIRE' on black ground".

With this knowledge, you can now explore the capabilities of ControlNet in image generation. Enjoy experimenting and check out our tutorials on fine-tuning and image-to-image techniques. All tutorials can be found in our blog-section.