AI rips objects from video and reimagines them in 3D AR

There’s an AI capable of teleporting John Travolta and Uma Thurman into your living room and forcing them to dance for you. The machine takes a 2D image, like the dance scene from Pulp Fiction, and reimagines it in augmented reality as a 3D object.

The tool is called Volume and it’s being developed by artists Or Fleisher and Shirin Anlen. It’s currently in the experimental stage, but the concept is simply incredible. Fleisher told TNW:

Our experiment with Pulp Fiction allows users to step inside one the film’s scenes in Augmented Reality, using Apple’s ARKit framework on an iPad. This experiment, is one of a few we are conducting at the moment, which illustrate the power of being able to reconstruct 3D scenes from 2D images. The possibilities of being able to reconstruct archival and static footage into 3D environments are one of the main motivations behind the development of the tool used to create these experiments called Volume.

The developers use a deep learning system called a convolutional neural network to train the AI. In essence, they give it a flat image to break down and analyze. Once the network processes the pixels and information, it tries to reproduce the 2D images as realistically as possible in AR.

This isn’t the first machine learning application that takes a 2D image and converts it into 3D, but this project doesn’t aim to provide another suite of back-end development magic for industry insiders to create content with – at least not specifically. It’s being designed as an end-to-end web solution, which means anyone using almost any smartphone, laptop, or tablet, will be able to input images or video for automated conversion.

As you can see, it’s not perfect. But there were will be innumerable applications for the technology once it matures. If Volume’s AI can get past the nausea-inducing blurs and wobbles, VR and AR enthusiasts are going to love this. And there’s potential beyond just watching 90s pop icons dance in front of your couch.

With this tool, for example, Dr. King’s “I Have A Dream” speech from 1963, or The Beatles 1969 rooftop performance could come alive in any classroom. It’s easy to imagine a high-school sports coach using clips from pro games to demonstrate fundamentals using AR, or a custom VR experience you create by feeding it your favorite horror or fantasy clips.

Like many cutting-edge AI applications there just isn’t enough research yet to determine the full implications of this technology. But one thing’s for certain: this is definitely going to change the way I watch UFC fights.