What do you think about “converting live-action video into animation”? This is not simply motion capture, but recording a sequence in real life and then artificial intelligence (AI) will take care of translating everything into anime style animation. But how can an image-generating AI convert video? Of course, this is not possible in theory, but in practice there is a way to do it.
Specifically, the AI will convert each frame of the original video into anime style, creating a collection of images, and when played in sequence one by one will recreate the original video but with a new style. However, this approach still has certain problems, because the AI does not know that it must maintain “same character in all iterations”.
Photo: u/neilwong2012
The main problem is that the AI creates “anime characters” based on the model’s position in a given frame, but the generated character images are not always exactly the same. When these images are projected in sequence, it can be noticed that the AI-generated character will change at certain times. While that creates a visually appealing effect, it’s not technically correct.
However, it seems that this is gradually being rectified. Reddit user u/neilwong2012 shared the transition from real-life video to anime style, using the Stable Diffusion and MultiControlNet tools, and the results were astounding.
Obviously, the results that the AI produces in the video above are impressive and quite stable in style. This attracted a lot of comments from netizens:
– The animation is very consistent, there is no constant blurring of certain parts, and the distortion is very subtle compared to other videos.
– You’ve taken rotoscoping to a whole new level and it’s really impressive to watch.
– Just a few more steps to perfect this technique.
– Let’s wait about five more years and this will become the way anime and cartoon production will be in the future.
– This approach also has great potential for video games.
– Will this become the anime quality standard for motion animation for years to come?
– In a few months, one could make any cartoon. Think of the sequels to the anime series that were never made, like Highschool of the Dead Season 2, for example.
