Why do the iPhone X’s Animoji work after covering Face ID’s sensors?

Ever since the iPhone X was unveiled, Apple’s Animoji have been lauded as the greatest innovation to grace smartphones since multi-touch. So advanced, in fact, that Apple had to create a fancy new sensor just to map your mug to a poop emoji.

Sarcasm aside, it turns out Animoji doesn’t always need Face ID to work.

Marques Brownlee, tech YouTuber extraordinaire, made the point in his iPhone X review. Animoji work fine even if you cover up all of Face ID’s TrueDepth sensors, and only stop working if you cover the regular RGB camera. See for yourself (start around 11:40):

In other words, it doesn’t seem there’s a clear reason Animoji can’t work on the iPhone 8 and 8 Plus. Was Apple lying to us this whole time?

Well, no, not quite. Animoji use ARKit and – as seen above – the single front camera can at least temporarily suffice for face-tracking shenanigans. However, Apple currently doesn’t allow ARKit to work on the front camera for any iPhone other than the iPhone X.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

I couldn’t find a real answer as to why; after all, ARKit works just fine on the single rear camera of the iPhone 8, and all of Apple’s 2017 phones use the same processor. But for Animoji specifically, my guess is that Apple primarily uses the RGB camera and call on TrueDepth for additional detail like fine muscle movements, even though at first glance I can’t see any animation degradation when the TrueDepth camera is covered up. Another possibility (I’m totally theorizing by this point) is that Face ID’s sensors are used for an initial depth map, and then the RGB camera takes it from there. [See update below]

It’s also worth considering that Apple likely plans further refinements and features with TrueDepth that will only be possible on the iPhone X. In any case, the fact that Animoji work well after covering the infrared sensor covered shouldn’t be a huge surprise; plenty of Snapchat and Facebook masks track faces just fine without any fancy camera hardware. Of course, they’re probably less accurate because of it.

We’ve contacted Apple for more information, and will update this post if we hear back. In the meantime, it looks like Animoji Karaoke will be exclusive to the iPhone X club.

Update: We tested it on a couple of iPhone X’s, and it seems my earlier theory was actually correct. While there’s little or no difference when you cover up Face ID’s sensors after Animoji have already begun, if you cover the TrueDepth sensors before you initiate the process, the resulting animoji are more laggy and stuttery than usual. That suggests Apple is using Face ID’s sensors to create an initial depth map – possibly refreshed periodically – which is then mapped onto the RGB camera.

Again, I’m still not convinced Apple absolutely needs TrueDepth for Animoji to be effective. After all, the Pixel 2 and Mate 10 can create realistic depth maps without special hardware for portrait mode on their selfie cameras, and masks works well on other apps not using fancy depth sensors. I’m sure plenty of iPhone 8 users would have been just fine with Animoji made using the RGB camera.

But maybe the Apple didn’t want to settle for ‘just fine.’ In any case, it’s clear the company is using TrueDepth on the iPhone X for at least some additional accuracy, and chances are it will only use it more as features and refinements are added in the future.