Leap Motion’s CEO wants its gesture control in cars, as a software upgrade to track hands nears

Leap Motion was borne out of a “deep frustration” that the process of creating a 3D object on a high-tech platform, the computer, took up more time than molding something with clay or Play-Doh.

“Computers are supposed to make us better and faster, but they make us slower,” company CEO Michael Buckwald told TNW at the sidelines of GMIC Beijing. “If you think about our keyboards and even touchscreens, they’re all binary… it’s such an incredibly elementary way of communicating to your computers,” he said.

This is why Buckwald is keen to bring Leap Motion’s 3D gesture controller to other form factors. He told TNW that he wants Leap Motion’s gesture control to be integrated in every car, TV set, phone, tablet and even VR headsets.

One of the big steps it is taking towards fulfilling this vision is preparing for Version 2 of the Leap Motion software, which should be launched in the near future. Currently, the Leap Motion software can only track all five fingers, but Version 2 will be able to track fingers that the sensors can’t see, allowing you to do things like fold your hands. Buckwald said:

Today the Leap is the best device in the world for tracking fingers, but it can only track fingers that it can see. If I have my hands extended like this in front of me, it can track all the fingers very accurately and very quickly, but if I try to interlock my hands or if my hands go over each other, it loses tracking.

And with Version 2, we’re actually able to track fingers that the sensors can’t see. So your both hands and all five fingers on each hand are tracked regardless of how you move your hands. The reason we have so much more robustness for Version 2 is because we’re collecting so much more data. So instead of the tracking finger directions, we’re actually tracking the full skeletal model of each hand, and of the joints.

Other than benefiting users, Version 2 will also help developers have a more consistent approach to Leap Motion. Now each developer adapts a different way of abstracting the hand on screen, such as implementing cursors or fireballs, but being able to track the entire hand will make it easier for developers to “bring the hand into the app.”

This progress to hand tracking instead of finger tracking is something that has got Buckwald excited in particular about cars. So far, Leap Motion gesture control has only been applied on PCs, with the first tablets and smartphones to use its technology still scheduled to arrive by the end of this year. Buckwald, however, has a vision of how Leap Motion’s gesture control can be incorporated in cars.

I think there are three use cases we’d like in cars. One is people in the rear seat interacting with the rear seat entertainment system, with their hands. The other is the driver using movements in space to do things like control the volume.

But the one that we’re the most excited about is the driver interacting with the heads-up display. You’re sitting there and there’s a translucent heads-up display on the windshield in front of you. Without actually taking your hands off the wheel, just using small finger movements, you can control it. Today there are actually about 30 car models that have heads-up displays, but the way you interact with them is still by fiddling with this knob, which defeats the point.

Buckwald revealed that Leap Motion is currently in talks with many automotive companies and people in the space, as cars move towards differentiating their products via technology. For example, earlier this year Apple launched CarPlay, integrating your iPhone with your car. This followed after Google revealed it was working to bring an Android-based system into cars.

“It’s a process but we really think we can change how people interact with cars,” Buckwald said.

Leap Motion isn’t only pushing to make its technology available on more platforms, but also in more countries. Early this year, Leap Motion inked a Softbank partnership to bring its 3D gesture controller to Japan. Buckwald revealed that it is in talks with companies in China, which will be the next country it will probably launch a big push to enter.

It was Buckwald’s first time in Beijing, and he told TNW that Asia has always been a key market for the company. He said:

Even back during the pre-order period a year ago, about 50 percent of the orders were from outside the US and countries like Japan, South Korea and China — and on a per capita basis, Singapore — were consistently at the top of that list. I think it’s because the culture here values technology and innovation so much, and you have such a large and growing young urban population, which overlaps really nicely with the types of people that use Leap Motion.

On the flipside, you have so many developers here, that’s also really important for us. Our product is really whatever developers build on top of it… So developers are really key in that sense. As we think about our retail strategy in these markets, they’re actually very related, because we want to make sure that there’s enough content for people to have a good experience before we enter a market.

As Buckwald laid out his plans for Leap Motion, it is clear that there is still a long way for the technology to go — in terms of availability, platform, software and devices. In March, Leap Motion had to lay off 10 percent of its workforce amid disappointing results, and Buckwald recognizes that it’s still the beginning of the journey.

“These are very early days and I think we’re at the tip of the iceberg both in terms of the platform and in terms of our software,” Buckwald said.