The possibilities of the iPhone X’s camera are way bigger than Animoji

More than two months after the iPhone X launch, it remains a conundrum for some to put a finger on exactly which features might justify the $1000+ price Apple has given its flagship phone.

Some might wager on the virtually edge-to-edge OLED display, which for the first time on an iPhone, does away with the tactile home button and prominent top and bottom bezels of every previous model. It is admittedly gorgeous.

As it turns out, however, the iPhone X’s most groundbreaking feature is not even discernible at first glance, even though it literally stares you in the face.

Meet the TrueDepth IR Camera

Apple has been dogged by perceptions that the company’s product roadmap has grown conservative and predictable in the six years following Steve Jobs’ passing. The new TrueDepth IR camera, as Apple calls the iPhone X’s secondary front-facing camera, could well help Apple dispel those rumors of complacency.

New to the iPhone X and housed in the black notch at the top of the display, the technology includes a number of high tech components such as a “flood illuminator,” infrared (IR) camera, and an infrared emitter. The latter emits 30,000 infrared dots in a known pattern when a face is detected, enabling the iPhone X to generate a 3D map of a user’s face. Notably, the TrueDepth IR camera can do this fast enough to support the creation of 3D motion data as well.

 

At present, there are currently two supported use cases for the new TrueDepth IR camera on the iPhone X:

• Animoji, or animated emoji — the camera animates a 3D emoji with your captured facial expressions, which you can then send via Messenger or SMS.
• Face ID — the camera uses a locally stored 3D image of your face to authenticate and unlock your iPhone X. Because it uses 3D image data, it’s much harder to trick than previous mobile face recognition systems that used a 2D image and therefore could be fooled with a photo

“What — that’s it?”

Admittedly, the current use cases for this technology are limited.

Animoji are just emoji with a fun twist, and unlocking phones hasn’t exactly been a pain point for modern smartphone users, with the prevalence of fingerprint sensors on many phones and pattern drawing as an adequate secondary option.

What’s incredibly intriguing is that the TrueDepth IR camera technology can and will have much stronger applications in the very near future. As with the iPhone’s previous Touch ID (fingerprint sensor), an iOS SDK makes it easy for app developers to use Face ID and the TrueDepth IR camera for a myriad of other more compelling use cases. In fact, Face ID uses the exact same API as Touch ID, making implementation into mobile apps incredibly simple.

Here are some of the areas where the TrueDepth IR camera could spawn useful or interesting mobile applications:

Financial Services and Internet Banking

For more than a decade, the gold standard in Internet banking authentication has been two-factor authentication. One of the most common forms of two-factor authentication has involved sending a verification code sent to your phone or email address, which must then be manually entered along with your password to access your account.

In the past few years, mobile banking apps have been quick to add support for fingerprint-based authentication. This has made it easier than ever to access your bank accounts online using your mobile phone. For the banks, this means customers have easier access to their services, resulting in increased revenues.

Why, given the already enhanced convenience of fingerprint-based authentication, would banks care about implementing face recognition into their mobile apps? To understand why let’s consider false positive rates — the rate at which any given authentication protocol provides a false authentication. Apple claims that Touch ID, its fingerprint-based authentication, has a false positive rate of 1 in 50,000. Compare that to the chance of guessing a four-digit PIN, 1 in 10,000.

Now, let’s look at the iPhone X’s new Face ID feature. Apple claims Face ID has a false positive rate of approximately 1 in 1,000,000. If true, this represents a 20-fold improvement in accuracy compared to fingerprint-based authentication and 100-fold improvement over a four-digit PIN.

That’s sure to get the attention of the banks offering Internet banking, as well as of the consumers who also benefit from enhanced fraud protection. In fact, many banks have already implemented Face ID support into their iOS apps. Chase, for example, added Face ID support within two weeks of the iPhone X’s launch.

Entertainment and Fun

iOS 11 introduced ARKit, a new framework that allows developers to easily create unparalleled augmented reality experiences for iPhone and iPad. Combined with the iPhone X’s TrueDepth IR camera, ARKit opens up a revolutionary capability for robust face tracking across multiple app categories.

Imagine:

• Entering the character creation screen for Assassin’s Creed: Identity and being able to not only choose your assassin’s class but also their visual identity, using your own face in place of the default character model’s
• Having a more accurate view of a prospective match’s face before you choose to swipe left or swipe right on Tinder, Bumble, or POF
• Being able to skip a trip to Sephora or ULTA and experiment with new makeup without shelling out for everything up front

A few years ago, virtual reality (VR) technology was in the same place that the technology behind TrueDepth IR is now: a promising technology needing stronger applications in video games and entertainment.

Oculus, a virtual reality company, was founded in 2012 and ran a successful Kickstarter campaign that year, raising $2.5 million. Just two years later in 2014, Oculus was acquired by Facebook for $2 billion, or nearly one thousand times the initial financing round. Could companies that invest in entertainment applications using TrueDepth IR technology find a similar trajectory?

There are already precedents in the space, too. Two years ago — when face tracking was still based on two-dimensional images — Facebook purchased (for an undisclosed amount) Masquerade, a startup whose MSQRD app lets users create self-altered images and video using face tracking technology.

Medical and Fitness

Imagine being able to have a medical examination at any hour of the day, on-demand, from the convenience of your bed, sofa, or office chair. With the help of the iPhone X’s TrueDepth IR camera, this could become a reality.

TrueDepth IR technology has the potential to be used for medical imaging, because it can create detailed 3D maps of your face, including the surface of your skin. Notably, it can also capture 3D movement, including subtle changes in facial expression.

Consider the following potential medical use cases:

• Have a dermatologist examine an abnormal skin growth, with the ability to rotate the image for a better look at the structure of a potential tumor, for example.
• Have your phone automatically alert you when it notices facial cues or subtle movements that could be associated with the onset of a stroke, seizure, or heart attack.
• For recovering stroke patients, you might have an app that tracks any muscular degeneration over time and provides your doctors with this detailed 3D image data on demand.

While these use cases would certainly not replace the need to have in-person examinations, regular screening via phone camera could play a supporting role in enabling early detection or regular monitoring of known medical conditions.

Consider also that Apple has published a software development kit (SDK) called Apple Healthkit, which allows app developers API access to rich health and fitness information across a variety of sources: fitness wearables, health devices, and software applications. New or updated mobile apps might combine health vitals from Apple Healthkit with 3D facial image data to offer a more complete assessment of users’ health or fitness.

The Future is Here

The iPhone X is the first phone to offer depth-sensing IR cameras, but it certainly won’t be the last.

“With the wide spectrum of new use cases that the technology makes possible, the impetus is now on software developers and bold businesses to bring these applications to life,” said Harry Lee, CEO of CitrusBits. Based in Los Angeles and San Francisco, CitrusBits is a mobile app development agency that designs and develops award-winning mobile apps for some of the world’s best-known brands, including Burger King, Symantec, and Quiksilver.

“Those that do embrace the technology now can enjoy first-mover advantages such as the ability to build a sizable market share and enjoy strong network effects that discourage future market entrants — not to mention the opportunity to get bought up by Facebook, Microsoft, or another titan of industry,” Lee explains.

To see more stories like this one, subscribe to CitrusBits’ newsletter, Bits & Bytes.