The popularity of ControlNet should also be discussed from this back photo.
A girl posted a back photo with three friends on the beach in a circle of friends. Very common daily sharing, photos are beautiful, but not much!
Then, a man named @viggo saw this photo, made some processing with AI painting and sent it to social software, which caused an uproar about AI painting.
Viggo used AI to process photos twice before and after painting. For the first time, he generated two comic photos with great aesthetic feeling, which changed the gloom of the original picture and made the whole picture look brand-new. The most surprising thing is that,AI did not obliterate the posture of the original picture, but greatly preserved the posture of the four girls.For example, the second girl on the left is carrying her hand behind her back, which is also preserved in the generated works.
Perhaps seeing the enthusiasm of netizens, viggo followed by a second round of creation. This time, the effect is even more amazing, no matter from the layering of the picture, the sense of lines or the use of special effects in the scene, it makes people shine.
With the popularity of his paintings, viggo shared the production method very openly. He exposed himself with the help ofStableDiffusionandControlNetThe power of.
As we all know, Stable Diffusion is a high-performance model, which generates higher quality images, runs faster, consumes less resources and occupies less memory, and is a milestone in the field of AI image generation.
01
So what is ControlNet?
A plug-in, which can accurately control and adjust the image.
ControlNet is an image morphing algorithm based on control points, which is mainly used in digital image processing, computer vision and computer graphics. In addition, it can also carry out nonlinear deformation of the image according to the given control points, so as to realize accurate control and adjustment of the image.
Let’s look at the basic structure of ControlNet first. ControlNet manipulates the input conditions of neural network blocks, thus further controlling the overall behavior of the whole neural network. here"network block"Refers to a group of neural layers, which are put together as a common unit for constructing neural networks, such as resnet block, multi-head attention block and Transformer block.
The concept is abstract, let’s take a look at its drawing effect!
1. Canny edge detection:By extracting line drawings from the original image, images with the same composition are generated.
2. Depth detection:By extracting the depth information from the original image, a map with the same depth structure can be generated.
3. ControlNet with semantic segmentation:
4. Diagram of HED edge detection
02
Where is the power of ControlNet?
-highly refined
Its advantage is that it can adjust the image with high precision without distortion.Compared with other image morphing algorithms based on grid or shape, the morphing effect of ControlNet is more natural and smooth, and it can better adapt to the characteristics of the image.
Simply put, this technology can add an additional input to the AI diffusion model and limit the output direction of AI. Just like building roads and signs in the endless desert, it provides a direction for lost travelers.
Before the advent of ControlNet, AI painters had to work hard on "magic spells" if they wanted to produce images with specific characteristics-users often needed to add a series of actions and position modifiers to describe the posture and physical characteristics of the characters in the pictures. Even so, drawing still needed a lot of luck. With ControlNet, AI can produce image files that meet specific requirements through sketches, human key point features, depth maps, human bones and other features.
At the beginning, it is mentioned that the back of four girls is generated. According to viggo, he first uses StableDiffusion pictures to convert words, then uses Text2Prompt plug-in to expand the search for keywords, and finally uses ControlNet plug-in to bind bones to start trying to change keywords.
Many people think that the release of Stable Diffusion is a milestone in the generation and development of AI painting. It provides the public with an available high-performance model, which not only produces very high-quality images, but also has low requirements for resources and memory. It doesn’t need any complicated operation, just select keywords, and it will create an image with great visual effect.
The emergence of ControlNet solved the pain point of AI painting. Using keywords to generate pictures will inevitably be flawed, especially in details.And ControlNet can improve the effect of drawing, and go deep into very subtle places.
More than that, it can also realize the conversion from line drawing to full-color drawing, and input a line drawing to get a filled drawing.
In a word, a stunning AI painting can be done by using Stable Diffusion to generate high-quality big pictures, supplemented by ControlNet icing on the cake.
Of course, its power is far more than that, besides generating static images, it can also generate dynamic videos!
03
The idea of ControlNet video
Recently, #AI One-Click magic change Video # has become one of the hot topics recently. In fact, this is the video version of ControlNet.It can generate pictures with the same composition through line drawing extraction, posture detection, or model recognition.
For example, IKUN and Titanic, which we are familiar with, can generate magic change videos with one click!
There are many ideas of videoization in ControlNet:
Method 1:Export the original video frame by frame, and then use ControlNet to transform the style of each picture. It is recognized that there is no one of the most cumbersome ways!
Method 2:Combined with EbSynth, key frames are generated by using ControlNet. Compared with the former one, it is much more convenient.
Method 3:Combining ControlNet technology with Pix2PixVideo technology, an online demonstration version of video generation supporting ControlNet is developed, which is the simplest generation method at present.
The user interface is very simple. When the original video is imported, the video will first be disassembled into a frame sequence, and the human pose map will be detected by using the Openpose model in ControlNet, and then a new portrait will be generated according to the text. Thanks to the control of the pose map, the human posture can be kept unchanged.
It can be seen that with the iteration of technology update, more and more technologies similar to ControlNet will flood into the market, and many people think that the painter profession is facing a great threat.
04
ControlNet should not let painters stop imagining.
But to help it better control the operation of the brush.
With the introduction of ControlNet in AI painting, it will bring a series of influences.This technology can help machines accomplish more tasks, improve productivity, improve painting quality and efficiency, enhance human-computer interaction, and provide new visual experience.At the same time, it also means that traditional artists may face competitive pressure.
As more and more enterprises began to use AI painting to draw images, ControlNet technology began to enter people’s field of vision. At present, many enterprises are trying to use ControlNet technology to improve their image rendering technology, and hope to gain more competitive advantages by using this technology. Applicable industries include construction industry, machinery manufacturing, energy industry, medical industry, government agencies and other industries.
It is generally believed that this can better control the "motion" of the brush and draw the desired image more accurately.
In a word, ControlNet has an important influence on the development of artificial intelligence and AI painting, but it never makes human painters lose their imagination! In the future, the development trend of ControlNet is more prospective.
Attached is the installation tutorial of ControlNet:
Premise: You should update the WebUI of AI painting to the latest version.
Pull:
The contents in https://github.com/Mikubill/sd-webui-controlnet warehouse can be pulled into the extensions folder under the WebUI file directory, or the zip can be downloaded directly.
Download:
Https://huggingface.co/lllyasviel/ControlNet/tree/main/annotator/ckpts, and put it in the ckpts directory under annotator in the plug-in directory.
Download:
The model under https://huggingface.co/webui/controlnet-modules-safetensors/tree/main is transferred to the models directory under the plug-in directory.
Now your ControlNet has been installed!
Specific use of the tutorial recommended bilibili video:
ChatGPT question and answer
ask
What are the defects of AI painting at present?
answer
Current artificial intelligence
Lack of creativity: AI painting can only generate new images based on existing images or data, lacking human creativity and imagination. Although AI can extract the features and styles of images through deep learning and generate corresponding images, AI lacks intuition, emotion and creativity, and it is difficult to create works of art with unique personality.
Unreal picture: AI painting can generate realistic images, but there are still some shortcomings in details and textures, and sometimes the generated pictures are not real enough to be easily identified.
Dependence on data: AI painting needs to rely on a large amount of data for training, and the lack of data will affect the quality of generated images. At the same time, the generated image is also easily limited by data, and it is difficult to create a new image beyond the data range.
Insufficient imitation: AI painting sometimes simply copies and repeats existing images or styles, lacking innovation and breakthrough, and it is difficult to achieve real imitation.
Insufficient ability of dialogue and interaction: At present, AI painting still lacks the ability of dialogue and interaction, so it can’t really interact and create with users, and it is difficult to meet the individual needs of users.
