Chatbots are coming to HoloLens – and they’ll help workers make better decisions

Soon conversational chatbots won’t just live in phones and computers, but assist industrial workers from within HoloLens.

Microsoft is big on bots. Last year, during Microsoft’s Build Developer Conference, CEO Satya Nadella declared that “bots are the new apps” and the company followed suit. Microsoft launched its Microsoft Bot Framework, a platform where developers can find the tools to create conversational bots of their own. These bots can communicate with Cortana, Microsoft’s virtual assistant, and be integrated into platforms like Skype and Slack. Need Cortana to order you a pizza? Have her take it up with Domino’s chatbot.

Pizza-ordering chatbots are a fun improvement to everyday life, but the possibilities extend far beyond the consumer market. For instance, by using chatbots in the industry, workers are already able to look up necessary details, visualize information, and follow procedures without extensive training. These chatbots don’t just live on laptop screens and iPads: they will soon address industrial issues using augmented reality, thanks to AR technology such as HoloLens.

For those who aren’t familiar with HoloLens: It’s a device that looks similar to other VR headsets, but is fundamentally different. For one, HoloLens is a poster child for augmented reality – or mixed reality – adding virtual objects to your actual surroundings. VR goggles like Oculus Rift, on the other hand, are meant for virtual reality applications generating a completely virtual environment. HoloLens can project overlays onto our surroundings because it’s equipped with a camera that sees depth. This enables the device to show virtual objects in their actual size. HoloLens can be controlled using hand gestures, such as waving and pinching, as well as certain voice commands.

Now imagine a construction worker wearing the HoloLens headset. By being able to see a mock-up of the building he’s working on, he is able to detect possible flaws in the design before they occur.

This article by MIT Technology Review, for example, mentions an American construction worker using an application by software company Trimble. When reviewing the 3D model for one of his projects, he discovered some steel frames the company planned to order were too long to fit the design.

“Having spotted the issue ahead of time, the company could ask the supplier to cut the frames shorter in his shop rather than make workers adjust dozens of tracks that would hold the frames in place,” he explained.  As a result, the construction company saved thousands of dollars in labor costs.

HoloLens is already assisting technicians and maintenance engineers as well. ThyssenKrupp, one of the world’s largest elevator companies, developed a HoloLens application for their service technicians. This allows them to review 3D models of the elevators that need fixing remotely, watch specific tutorials on how to proceed, and check the elevator’s service history together with notes other technicians have left in the system.

Dutch company Montapacking, specializing in packaging goods ordered in web shops, uses HoloLens in its 10,000 m2 warehouse. The application projects arrows in augmented reality, which direct employees to the products or shelves they’re looking for to fulfill an order.

Time is money in the ecommerce industry where customers expect their packages delivered in one working day, or even sooner. So by saving time on order picking, Montapacking gains a competitive advantage.  

Even in hospitals, HoloLens has proven its worth. In a working environment that requires impeccable hygiene standards where gloves need to be worn at all times, surgeons can use voice commands to visualize data or look up medical records. In some cases, augmented reality can even help them operate. Thanks to a navigation system developed by surgical tech company Scopis, for instance, spinal surgeons can review an overlay that tells them exactly where they should cut.

The applications mentioned above are voice-controlled, which means the system understands certain words and commands. Soon, however, HoloLens applications will also integrate more intelligent chatbots, also called “conversational chatbots,” into their systems.

According to Sam Denton-Giles, Microsoft’s advanced analytics lead who spoke with Aviation Week, these bots will be able to manage far more complex tasks. They can respond to questions, learn user preferences and will use and understand ordinary language. Logging into systems and clicking through displays will no longer be needed, as these bots will walk you through the process. And because they are self-learning, they become smarter with every query.

To make all of this happen, Microsoft is currently building machines that can perform Machine Reading Comprehension; a cognitive skill that allows bots to read large chunks of texts – technical manuals, for example – make sense of them, and answer questions about them in a spoken conversation. This means the system doesn’t just recognize words and phrases, but also understands things like context and nuance, enabling it to answer questions without pre-existing knowledge.

Bots who can engage in natural conversations will be game-changers for those working in knowledge-intensive industries, says AI-expert Martijn Breet. Last year, together with cloud business software company Exact, he developed a proof of concept for HoloLens: a life-size, holographic virtual assistant created to help entrepreneurs by answering questions about their financial situation. “Research shows many entrepreneurs, specifically those working alone, often feel the need to talk to someone or share ideas.”

The virtual assistant was demonstrated at German tech conference CeBit, where she received raging user reviews. Nonetheless, it’s still too early to tell if the bot will be assisting Exact’s clients in the future. “Right now, the adoption of VR headsets in businesses is still very low, so most companies are simply not ready for such an application. But I expect that will change in the coming years.”

How will HoloLens’ marriage to conversational chatbots affect industrial jobs? One of the most important benefits is that it will drive greater efficiency. For those working with their hands, whether surgeons or engineers, using HoloLens onsite increases safety, saves time and helps people make better and faster decisions. The arrival of smarter bots will push the envelope even further – turning simple bots into highly valuable assistants.

And we don’t even have to wait that long: Breet predicts that HoloLens business chatbots will go mainstream in 2019. “Rumor has it Microsoft launches the next generation of HoloLens in 2019, which will be a lot less expensive for consumers. Once more people start using HoloLens, this will open up many new opportunities for using conversational chatbots.”

Artificial intelligence is coming to your industry, so make sure your business isn’t missing out on its endless possibilities. Cloud business software company Exact already provides 375,000 entrepreneurs worldwide with a full stack services, offering them an overview of today and insights into the opportunities of tomorrow.