Many of us, myself included, have never artistically graduated from stick-men drawings. Therefore, a Van Gogh painting, for example, tends to spark feelings of awe, jealousy, and the question of “how the hell did they do that?” Well thanks to artificial intelligence (AI), their secrets may have now been revealed.
Researchers at MIT have produced a machine learning system, called “Timecraft,” that can generate a time-lapse video showing the most likely painting process of a given piece of artwork. The team trained their system with a dataset of 200 existing time-lapse videos of artists creating digital and watercolor paintings. This enabled the system to learn the usual patterns of painters, which it could then use to generate “step by step” videos.
“Within each time step, artists tend to work within a single section of the scene,” project leader Amy Zhao said in a YouTube video. “They tend to use only one to two colors at a time. Artists also typically work in a coarse-to-fine manner (i.e. start with the “bigger picture” then work on the details).”
Existing systems seem to neglect these techniques and create time-lapse videos showing blurry transitions where work is carried out all over the image. However, Timecraft much more realistically mimics the expected brushstrokes of artists. In fact, when the researchers carried out a survey asking participants to compare the videos produced by their system, existing systems, and the real deal for the same image, more than 90 percent of the time Timecraft videos were seen to outperform the existing benchmarks. The newly produced time-lapses even gave the real videos a run for their money, as nearly half the time participants were confused as to which one was which.
As the team demonstrated in their paper, which was presented this week at the Conference for Computer Vision and Pattern Recognition (CVPR), their system can also be used to show how some of the greats may have painted their masterpieces, including Vincent Van Gogh’s “Self Portrait.” In this particular example, creating a realistic time-lapse for a face was no mean feat either.
“Our method was only trained on a few faces, however, in these examples, it captures painting-like dynamics quite well,” Zhao explained in the YouTube video.
Maybe one day, with a bit of help from AI, my stick-person will flourish into a recognizable human after all.
1
