2016 has been a remarkable year that’s brought continued growth and awareness to the worlds of Augmented, Virtual and Mixed Reality. Set to become a $165 Billion dollar industry by 2020, there’s still a common question that lingers among many newcomers trying to understand this fast moving digital phenomena we are just beginning to watch evolve: What’s the difference between them and how will it impact the digital world as I currently know it?
Before we jump into the mind-blowing future Mixed Reality is set to usher in over the course of 2017, let’s first discuss the distinctions between Virtual and Augmented Reality. Their technologies are very similar but have some fundamental differences.
Virtual Reality
Virtual Reality (VR) is a digital environment that shuts out the real world.
VR is able to transpose the user. In other words, it can bring us someplace else. Through closed visors or goggles, VR blocks out the room and puts our presence elsewhere.
Below is a hilarious example of Conan O’Brien demoing VR on the HTC Vive in a ”Job Simulator” environment.
As you can see, VR is pretty amazing because its allows users to do so many things like watch a 3D experience of Cirque du Soleil, attend an exclusive concert or sporting event, ride in a roller coaster, play games, create art projects – and in some cases – hang out with friends in virtual worlds. (Conan did a poor job making friends in his virtual world in the example above by the way.) Recently, Virtual Reality has even been used to help sick children escape their hospital environments and bring them to happier surroundings. So the question you might be asking is, what does this have to do with Mixed Reality? The answer: VR is a gateway for a lot of the more advanced technology that will be used in Mixed Reality.
Augmented Reality
Augmented Reality (AR) places digital content on top of the physical world you see around you.
AR works by adding 2D or 3D layered content on top of real world objects or locations, allowing the user to unlock additional information that may be relevant, therefore turning the physical world around them into digital media.
Some things you can currently do within an AR experience:
- Play games
- Watch Videos
- Connect to Websites
- Listen to Music
- Enter a Sweepstakes
Right now the majority of AR is used on mobile devices, tablets and integrated into mobile apps but that is quickly changing. Mixed Reality, also a form of Augmented Reality, has a much higher degree of complexity and is much more realistic as you will read below.
Below is an example of a Mercedes Benz AR app activated off a sales brochure.
Mixed Reality — The next big paradigm shift
A hybrid of both AR and VR, Mixed Reality (MR) is far more advanced than Virtual Reality because it combines the use of several types of technologies including sensors, advanced optics and next gen computing power. All of this technology bundled into a single device will provide the user with the capability to overlay augmented holographic digital content into your real-time space, creating scenarios that are unbelievably realistic and mind-blowing.
How does it work?
Mixed Reality works by scanning your physical environment and creating a 3D map of your surroundings so the device will know exactly where and how to place digital content into that space – realistically – while allowing you to interact with it using gestures. Much different than Virtual Reality where the user is immersed in a totally different world, Mixed Reality experiences invite digital content into your real-time surroundings, allowing you to interact with them.
The use of transparent lenses, spatial sound and an understanding of your physical environment will allow holograms to look, sound and behave like real objects that are able to interact with the environment around them and also with each other. Take for example the video below of Robots crashing through the walls and firing weapons at the Microsoft Hololens user. Not only mind-blowing but truly a magical experience!
“Believability is a core design principle in Mixed Reality”
Microsoft Hololens Engineer – Cris Derr
What can Mixed Reality do?
Mixed Reality will not just be another advanced gaming console to play the latest version of Halo or Madden NFL. Instead, it will add a whole new world of interactions, apps, games and experiences we have yet to imagine. The world around you will become an entirely new canvas for you to play, learn, communicate and interact with.
Several Use Case Examples:
Communication: Holoportation will allow device users in different cities or countries to sit and interact with each other in the same space virtually, while remaining miles and miles apart.
Education: Below is an example created by Project Esper displaying how medical students will benefit and gain more comprehensive views and interactions with the human anatomy
Entertainment: Microsoft Hololens is already working with the NFL to completely change the way fans can watch and interact with players, other fans, real-time game experiences and its advertisers and sponsors.
Use-cases for this technology are endless: sports, music, television, art, fashion, business, education, medicine, interior design, retail, construction, real estate and virtually everything we do will be affected. During the first evolution, Mixed Reality will replace your mobile device and then slowly start to replace all remaining devices including TV’s, laptops, tablets, etc. All of the content you consume on those other devices will soon become readily available within your new world of Mixed Reality and through one single pair of lenses.
When will we see mass-consumer adoption?
The key to driving mass consumption of these platforms lies in making Mixed Reality devices that are smaller and much more affordable to everyone. Right now, Microsoft Hololens is the only premium device in the market with a price tag of $3,000. It’s really only designed for enterprise and engineers and does not yet have a mainstream consumer base. Meta 2 is supposed to start shipping soon at $949, and Magic Leap is still unannounced.
The key will be to start developing smaller affordable devices able to ensure that massive amounts of data can be transferred without being slowed down or limiting the experience. For this, 5G Networks will play a vital role – and it’s right around the corner. Additionally, off-loading compute functions from the device to the cloud will free up significant bulk in the lens or headset. Ultimately, we need to get to the point where these capabilities become seamless and affordable. We’re getting very close, and are likely to see big improvements in 2017.
What’s Affordable Right Now?
ZapBox – Cardboard Mixed Reality for $30
AR company Zappar recently announced the creation of world’s first cardboard Mixed Reality headset. It comes complete with cardboard controllers and a special wide-angle camera lens for only $30 on Kickstarter.
Zappar’s mission is to democratize and make MR affordable to all in a similar manner that Google did with cardboard. ZapBox takes an outside-in approach to the MR experience and delivers some pretty cool engagements where the digital content is brought into your physical space to interact with you.
The Bridge MR / VR headset by Occipital for $399 (iOS only)
Start-up Occipital has also created a very cool hybrid MR / VR HMD solution called The Bridge that is designed specifically for iOS with a $399 price point. The headset comes with a 120 degree wide angle lens and a depth sensor called the “Structure Sensor” that Occipital has been working on for several years. This allows it to perform inside-out positional tracking (one that doesn’t rely on external sensors) that could be fast and accurate enough to allow augmented objects to appear anchored in the real world so they actually seem real.
Microsoft has plans for Mixed Reality affordability in 2017
Microsofts Hololens has already set the stage by being the absolute best MR device in the market place to date. That said, the current price of $3,000 is way out of reach for the average consumer.
Microsoft recently made some announcements during the opening day of the company’s WinHEC event in China on December 7th. The company has provided official PC specs that will be required to support the first VR headsets able to handle mixed reality capabilities.
Additionally, Microsoft confirmed in October that companies like Lenovo, HP, Acer and ASUS have all made commitments to developing PC tethered headsets with prices starting at $299 beginning in early 2017.
Unlike Hololens, which is a stand-alone device, to make these new headsets affordable to the mass market, the new devices will need to be tethered to a PC which is required to process much of the compute power.
What about Apple?
What about Apple?
Apple’s entry into Mixed Reality will be an obvious and complete game changer that could literally seed the market with MR capabilities at serious scale overnight when they are ready to pull the trigger.
Tim Cook boldly made his opinion known to the world that “AR will be much bigger than VR.”
There are also reports swirling that Apple is prepping to launch the iPhone 8 in September of 2017 and that it will be a Mixed Reality device.
Robert Scoble was first on record to report Apple is readying a fully transparent mobile device complete with a next gen OLED Display, Wireless Charging and a 3D Sensor by Primesense. Apple reportedly has more than 600 engineers in Israel busily working on the sensor alone. The compute will be neatly packaged and hidden within a strip at the bottom. The device will also connect to a headset. Eye sensors by Eyefluence will give the new iPhone 8 a higher frame rate and polygon count than a PC-tethered MR device with a Nvidia graphics card. (See below patent.)
To some, a transparent phone might seem crazy, but Taiwanese company Polytron already developed a clear, multi-touch device back in 2013 as a prototype that still hasn’t hit the market yet. (Image below.)
Recently the CEO of Foxconn, Apples manufacturing partner, has confirmed the wireless charging and next-gen OLED display. Patents filed by Apple detailing the transparent mobile device have also surfaced.
What this all means is, effectively, on the first weekend Apple launches the iPhone 8 in September 2017 (coincidentally, iPhone’s 10-year anniversary) we could potentially have over 60 million Mixed Reality capable devices in the world, overnight! Are brands, advertisers and content providers ready for this?
Note: *Robert Scoble has recently announced that brands like Samsung, Huawei and others are also readying new MR devices but details are still forthcoming.
Patent Filed for Apple’s Wearable MR Headset
Advertising in Mixed Reality
Mixed Reality will offer brands and advertisers the ability to connect, interact and sell to consumers in ways that never existed before. This will have a profound impact on their bottom lines, as brand loyalties become harder to stabilize.
Product packaging is becoming a preferred digital platform for brands wishing to tell stories and communicate with their target consumers.
For example, a Coke can now deliver a realistic 3D gaming, music or sporting event experience. Food products like Hellman’s Mayonnaise or Extra Virgin Olive Oil will be able to deliver exclusive 3D recipe tutorials given by a Rachel Ray hologram. Many of the items that make it into your grocery cart will have digital discounts and product benefits displayed straight from the package as you walk down the aisle and shop. Since these devices function primarily by acquiring a 3D understanding of our environment, brands are going to have incredible targeting and re-targeting opportunities with their amazing new content.
Much of this engagement will be sought after and wanted by consumers, so it will be pull-based through interactions with objects and brands making it extremely valuable to both the advertiser and consumer. But, as with all quality advertising and messaging, there will also be fierce competition by brands looking to push as much unwanted messaging directly within your line of sight as possible.
Because everything in the world will now become prime real estate for brand marketing, new filters and restrictions will need to be put in place and adhered to in an effort to protect consumers and their privacy. In a recent article, Beehance founder and VC investor Scott Belsky, offered five predictions for the future of tech (all of which I agree with). He suggests that Adblockers may be a suitable solution for these concerns. He also cites the need for industry governance through a professional association that can establish a clear set of rules and guidelines for advertising within the new world of Mixed Reality.
Artificial Intelligence and Machine Learning:
Because we will rely on these new MR devices for practically everything we do, and because we could potentially wear them throughout most of our day, there will be a significant increase in the amount of time and content we consume on a daily basis. For brands, this could mean more advertising opportunities that allow them to capture – and learn from – behavioral data.
Ultimately, however, this will also mean that more physical objects in our world must be indexed in order to have some type of relevant content associated with them. So how will all this messaging and content be segmented so that it’s delivered in all the right places and to all the right audiences?
Using advanced computer vision, AR start-up Blippar has been working on a real-time deep learning engine that will recognize average everyday objects like fruits and vegetables, flowers, plants, animals, etc. and will then scrape the web to apply a thin layer of relevant digital content on top of these objects – otherwise referred to as the Internet on Things.
Imagine the ability to pull nutritional content, recipes, store locators, etc. simply by looking at a piece of fruit. (See images below)
This type of engine could become an extremely valuable integration to a Mixed Reality HMD manufacturer, especially since much of what we are looking at will soon be detected by Eye Sensors.
Think of it like a next gen Siri assistant or Google search engine, except you will use your eyes to unlock search queries from physical objects you look at, rather than voice or typed search commands. This can also help to solve everyday problems related to unknown objects, language barriers and much more. In the end, the result could become even bigger and more expansive than the internet we know today.
Let the fun begin!
2017 will be the start of an era that will bring some of the biggest digital advancements we have ever seen on the face of the earth for all of mankind. Through Mixed Reality we will all be able to share, experience and understand the world with a whole new set of eyes: a smarter more connected and magical set of eyes!
Get the TNW newsletter
Get the most important tech news in your inbox each week.