A love letter from augmented reality to Pokémon Go

The first half of 2016 has been all about virtual reality: The buzz around Oculus Rift, Sony’s Playstation VR, and HTC’s Vive have been lighting up the internet. VR is awesome and it’s ready for prime time.

But now VR’s friendly brother AR stole the show – thanks to Pokemon Go, the so-called “AR killer app” the industry has been waiting for.

But before I go on about AR, let me sincerely (I really mean it) thank the guys at Niantic for creating Pokémon Go.

For the first time in my almost seven years as CMO of mobile AR pioneer Wikitude, I do not have to explain what AR is. But what I do want to do is tell you all the other things AR can and will do – because Pokémon Go actually touches on only one main category of mobile augmented reality, and there is so much more.  

You may be surprised the world’s hottest mobile game is running on eight year old AR technology – but that’s the truth. 

Three main categories of AR technology

GEO Augmented Reality

As the term suggests, GEO AR is used for location based augmentations.

Pokémon Go is by far not the first app to utilize this simple form of AR. Blackberry (remember your Crackberry?) integrated its BBM, a proprietary messenger app similar to WhatsApp and WeChat today, with an AR feature to show users where other BBM users are located around them.

Lonely Planet, Trip Advisor and Yelp! are among many other big brands who have taken advantage of GEO AR to help their users understand their surroundings better and to find what they are looking for.

The way GEO AR works is actually rather simple: Take the GPS coordinates of the user to get his position in the world, read his digital compass to see in what direction he is holding the phone, and capture the signals from the accelerometer or gyroscope to understand the movements of the device.

At the same time draw content relevant for this location from the server and render it on the camera display between the user’s position and the longitude and latitude of the object in question, whether that’s a little info bubble on a museum or a highly-desirable Pokémon.

2D image recognition and tracking

The second kind of AR tech already involves highly sophisticated computer vision. 2D refers to the fact that the algorithms at play are optimized to recognize and track planar objects, or to put it into simple terms, flat surfaces.

The print industry is probably the vertical that takes advantage of this the most, by augmenting books, brochures and billboards. You might have tried 2D image recognition and tracking already with an IKEA catalogue – they’ve been using it since back in 2012, when they were one of the first major companies that saw the opportunity in AR and introduced its first “print-to-web” app, to build a bridge from the paper catalogue to the online IKEA store. Need more history? Check out this four year old article.

Roughly, it works like this: Take a picture of a magazine cover, run it through a computer vision algorithm, which looks for something called “feature points” in the image. The combination of all the feature points found make it unique and identifiable amongst millions of other images. That’s the “recognition”.

The bit about ‘tracking’ refers to the ability to stick an augmentation on top of the magazine cover when scanning your AR enabled app over it. The tracking ensures that the augmentation, whether that’s a buy button or a product video, is glued onto the surface of the image. 

3D recognition and tracking

Now things get a bit more complicated. Tracking the three dimensional space around us is what’s keeping computer vision engineers around the world very very busy, and this will not change for a while.

Let me be honest with you: The nut has not been fully cracked, and nobody has the holy grail yet – but that’s exactly why companies like Magic Leap have been able to raise $1.39 billion.

Everyone is ramping up their computer vision R&D units. Recognizing and tracking a magazine cover is one thing – recognizing and tracking objects, rooms and spaces is another story altogether.

What’s the point? Well, here’s an attempt to explain this category of AR by giving you some examples of what it will be capable of doing:

Objects: Imagine the ability to recognize and understand all of the objects that surround us.

Imagine holding your smartphone’s camera at car you see on the street passing by and immediately identifying the make and model, and being delivered all specs, price and dealer locations with a blink of an eye.

Imagine looking at a flower, a tree or an animal with your AR glasses on to find out its name, characteristics or any other fact you could possibly want about it.

Rooms and spaces: You’re in a busy train station trying to figure out which path will take you to your destination. With 3D tracking, you’ll simply scan your surroundings with your smartphone to get directions.

Or you can stroll through a mall and what’s on sale in your style – without even going through a door.

Imagine the ability to design the interior of your apartment by hanging augmented paintings on your walls. Maybe see how a brand new sofa looks in the living room.

To get any of this to work, we have to crack the hardest nut of all in all of computer vision and AR: Real-time 3D tracking.

This is only the beginning

Where will AR take us? We’re not really sure as things are only starting to take shape, especially for 3D recognition and tracking.

Looking at the current developments and the amount of money that’s being poured into the industry, there is a very high probability of the real and virtual worlds becoming increasingly indistinguishable.

No matter the outcome, you can be sure some of the brightest minds in the world are working on it – and now with Pokémon Go, we’ve got a shining beacon of Augmented Reality success, and even more people will be heading down the path.

It’s certainly going to be interesting to see where it leads us. Thanks again, Niantic – AR is here to stay.