Google showed us a dream of wearing AR-powered spectacles everywhere with Google Glass, but that didn’t last. Now with many companies — including Apple — working on consumer-facing AR glasses, we will soon get to experience augmented applications.
While these glasses are a few months to a few years away, developers will need to start thinking about what applications they might build for the new hardware. Currently, most AR developers rely on Google’s ARCore or Apple’s ARKit to build applications that live on phones and tablets.
In this story, we’ll take a look at what applications we’ll first see on these glasses, and what the challenges in building them might be.
What’s the vision for the apps on consumer-facing AR headsets?
In the near future, we’ll likely wear our AR glasses outdoors, and that’ll allow us to leave our phones at home. However, that means packing a lot of computing power in a small frame, and having constant 5G connectivity for uninterrupted data flow.
We might not get world-bending software with the first generation of consumer glasses. But the initial set of apps will give us contextual information that will reduce the need of picking up our phones — just like smartwatches with their notification mirroring systems.
Leo Gebbie, senior analyst, XR & VR at CCS Insights, says that it will take time for developers to make advanced applications for AR such as search:
The initial set of applications will feature what we call contextual computing. The hardware will act as a heads-up display to show information such as notifications, and reminders based on the time of the day. Once we have apps that are effective in this, we’ll move on to advanced applications such as search and social interactions.
Timoni West, vice president of AR & VR tech, Unity has similar views. She adds that the apps would also interact with the Internet of Things (IoT) to present contextual information from connected devices around your home and office. I would like it if my AR glasses could talk to my AirTags, and point me to my keys with visual directions.
If you want to take a look at how the future application on AR glasses look like, you don’t have to wait. At Microsoft’s Build developer conference earlier this year, AR giant Niantic showed a demo of how you might be able to play Pokémon through Microsoft HoloLens sometime in the future. Notably, the company’s CEO, John Hanke, teased the firm’s own AR glasses.
In Jaunary, Lenovo revealed its new set of ThinkReality AR glasses, which supported setting up virtual monitors that don’t take up physical space. This could be useful for a wide range of people including programmers and freelancers. Alas, this device is only available to enterprise customers for now.
What is the challenge in putting an app together for AR glasses?
When you translate an AR app from a phone to glasses, one of the biggest challenges is to build a way for users to interact with it.
Cyril Diagne, the maker of the ClipDrop AR app, said that we’ve been using the same interaction models since the 70s — such as moving your hand on a plane (mousepad) to move the cursor. With AR glasses, these movements might become more three-dimensional and natural.
Aaand here it is..!!! ?
After months of hard work with @jblanchefr, @ClipDropApp beta (AR Copy Paste) is now publicly available on #Android, #iOS, #macOS, and #Windows
Here's a thread of what you can already do with it ↓ 1/n#ML #AR #AI pic.twitter.com/0fQJQ8KRBv
— Cyril Diagne (@cyrildiagne) October 22, 2020
So developers will need to find new ways to include gestures such as voice commands (Siri talking to Apple Glass?), or pointing controllers or your hand to highlight an area. The challenge is that on a phone, buttons and other interactive surfaces are well defined on the screen. When we talk about AR glasses the button could be anywhere in the room you’re in.
The rise of VR has helped the cause a bit, as different kinds of controllers allow app makers to set a up platform that lets users interact with spatial objects.
However, West notes that gestures are like sign languages for computers, and they defer from person to person. So platforms for AR apps will need to be good at understanding what gesture a user is making. For example, if you’re throwing a can in an AR environment, the platform will need to capture data to understand how hard you have to move your hand to make it disappear out of bounds. Plus, it’s hard to track hands because of how they move.
Some companies such as Facebook are already experimenting with neural response-based controllers to have refined interactions in the AR space whenever consumer-grade glasses come along.
Diagne, who was also a former professor for media and interaction design at ECAL university in Switzerland, thinks that app developers also need to make software for AR glasses like plugins or Chrome extensions. What it means here is that those apps should be easily integrated into the environment around you, rather than working in just specific conditions.
Another challenge for developers is accurately mapping the world around you. West said that at Unity, the company has developed a MARS companion app that captures different information including depth through the mobile camera. These details could be crucial in setting up AR apps for headset-like devices.
“Making a simple AR game or an app is 100 times harder than making a mobile app because it’s hard to know where your user is. They could be in a big room or they could be in a space as big as a closet,” she said.
For instance, you could be at a party where a Lego-based app needs 1-meter x 1-meter floor space, but since a lot of people are walking past it doesn’t work well. West said that to solve this, Unity developed a model that can be developed once, and designed in such a way that it has graceful fallbacks in case of not finding enough space. In this instance, creating a fake floor space.
Where does machine learning fit in?
Machine learning and AI play an important part in almost every software-based implementation today, and AR is no exception. Google and Apple are already using some AI components in their mobile AR platforms to figure out different surfaces or to detect objects and movements.
On AR glasses, it’ll play a much more significant role by also trying to recognize different environments and gestures made by users. Plus, it has to generate meaningful responses to these actions.
An article on Towards Data Science points out that machine learning can fill gaps in visual data. For example, it can construct or reconstruct parts where the visual captured by the camera has flaws.
Privacy implications
In order to accurately map your world, AR apps have to constantly gather and update information about your surroundings. This means they’re collecting a large amount of data all the time, and it could lead to some serious privacy-related problems.
A report on data mapping published by legal firm Ice Miller published in 2018 says that AR mapping “may not only capture personally identifiable details, but may also lead to data collectors retaining the information indefinitely or transferring the data to third-party companies that will benefit from AR mapping data.”
Electronic Frontier Foundation, a digital rights organization, highlights in its 2020 report that AR maps can lead to devices that are always on and recording. That could mean even if you’re a bystander, your privacy is violated.
West said that these maps are data-rich, and they’re hard to anonymize. But apps could potentially throw away all the textures that can reveal someone’s identity and just keep the depth information for processing.
What’s the future?
We can expect that when the first set of truly consumer-facing AR headsets are out in the market, we will see contextual information take precedence of complex games and apps initially.
Diagne believes that the “Citizen Kane” application for an AR headset would be a simple button that makes some kind of task easy — just like Uber made ride hailing easy on phones.
West says that this will be third generation for AR with contextual information and lightweight devices that will be cheaper than your average mixed reality headsets used in the industry. Plus, there might be platforms such as Qualcomm’s XR1 and Apple’s rumored rOS that can easily cooperate with existing operating systems.
All of this sounds dreamy. But most companies including Apple, Snap, Niantic, and Oppo are either working on a product, or have distributed test versions to only select people. So you’ll have to wait for at least a couple of years to see the weather forecast floating in front of your eyes.
Get the TNW newsletter
Get the most important tech news in your inbox each week.