Here’s what you missed at Facebook’s #F8 keynote (Day 1)

F8, Facebook’s annual developer conference, is underway.

The keynote kicked off the event on a weird note, as a giggly Mark Zuckerberg took the stage to make dad jokes about the “other” F8 — the eighth movie in the Fast and the Furious franchise: The Fate of the Furious. Facebook, it seems, is sharing the hashtag with Vin Diesel and Co. for this year’s event, an unfortunate circumstance that could have been remedied by getting The Rock on stage to discuss breakthroughs in artificial intelligence and machine learning.

But Facebook dropped the ball there.

Zuck then threatened to read his entire six thousand word manifesto, published last year, to kick off the event. Mercifully, he just summed it up instead.

AR Studio: An AR platform in your camera

Calling all Scaleup founders! Join the Soonicorn Summit on November 28 in Amsterdam.

Meet with the leaders of Picnic, Miro, Carbon Equity and more during this exclusive event dedicated to Scaleup Founders!

Apply Now

Zuckerberg quickly moved on from dad mode to Facebook co-founder and CEO and started talking about AR. He expressed a desire to “extend the physical world online” and asked each of us to “think of how many things in our life don’t need to be physical,” before using the monitor behind him to detail a world of virtual chess, or watching TV — sounds riveting.

At Facebook, Zuckerberg pointed out the team is working on three specific use cases for AR:

The ability to display information
The ability to add digital objects to real-world images and video
The ability to enhance existing objects

Using a technology known as SLAM — simultaneous localization and mapping — Zuckerberg gave a demo of the hyper-sensitive location mapping technology using a bowl of cereal as the example. Using complex real-time visual information, the most important meal of the day quickly turned into a fun AR experience with sharks circling the bowl, bobbing up and down through the table as if it were water. It was impressive, and shows just how powerful SLAM is in detecting the most minute of location details and mapping it to add digital flourishes to real-world experiences.

To have realistic 3D objects, Zuck said, you need precise location tracking. SLAM seems to offer just that for Facebook’s new AR platform. From there, you can take a simple 2D still image, adjust lighting, allow for a rotating/panning camera, and fun effects like filling the room with water, adding bouncy balls, or making it rain Skittles — “the future is delicious,” he said.

Facebook is working on other use cases as well.

There’s an AR tool for Nike for visualizing and sharing your runs in new ways. The effects are mostly silly, but adding personal touches like sweat, crazy running gear, or new backgrounds to run data could lead users to share more personal data on Facebook. The company, remember, is struggling with getting people to share personal status updates. Now, most of us use Facebook as a platform to talk on Messenger, manage a business page, or share memes, news, and video. Adding personalized touches to images and video could correct the problem. Or, at least that’s what Facebook hopes.

Perhaps the coolest feature of the AR platform demo was the ability to add notes to every day objects — circling your favorite menu item at a restaurant, pointing out the best table at a dive bar, or leaving a note for your wife on the refrigerator. All of these offer additional use cases for AR that are both practical, and useful.

VP of Platform and Marketplace Deb Liu also announced some cool features developers will see in the months to come, like hand and motion tracking. Many of the more advanced features of the platform are slated to arrive later this year.

Facebook’s AR Studio will make augmented reality mainstream

Image recognition

Facebook CTO Mike Schroepfer took the stage next to talk about the company’s advancements in image recognition. It started in 2012 with ‘alexanet,’ a basic neural network capable of making elementary calculations about image contents. It wasn’t great, but if there was a person in the photo, or a tree, perhaps, alexanet could alert you to its presence.

The technology improved from there. By 2015, Facebook as able to recognize multiple objects in an image and (mostly) mask them accordingly.

And now, using a technology known as Mask 4-CNN, Facebook can draw a sharp mask around each object in an image, identify it, and even use 17 point key deduction to tell the computer (or human) exactly how its posed and oriented in relation to the scene. This, while exciting, is the tip of the iceberg. Just released last month, Mask 4-CNN gives Facebook users the option to one day remove objects entirely from an image in near real-time. Schroepfer used a video of the ocean in which a windsurfer entered the scene. As he was shooting the video, Schroepfer was able to remove the windsurfer and display only the ocean behind him.

It’s unforgiving, as Schroepfer detailed, but when you get it right the results are amazing. And this is exactly the type of technology that will aid Oculus — or Facebook’s new prototype headset, a standalone model with SLAM, and four tracking cameras — in taking real-world input and translating it to create virtual environments.

Facebook Spaces

Head of Social VR, Rachel Franklin, took the stage next. “VR is a naturally social platform, and we’re building it with people at the center,” she said before unveiling Facebook’s first social VR app, Spaces.

The app brings in all the people you interact with on Facebook into the virtual world. After creating an avatar you can bring in up to three people to join you in a VR experience. These social experiences allow users to watch 360 video together, hold a virtual game night, or show off your new apartment to friends and family — all inside Facebook’s virtual space.

Better, you can call friends that don’t have a VR headset and open a virtual session with them once they answer using video calling on Messenger.

Spaces launches in beta later today at the Oculus store.

Messenger Platform 2.0

David ‘The Rock’ Marcus, Facebook’s VP of Messaging Products, closed out the keynote. He noted the progress made on the Messenger platform — four updates since release that brought quick replies, persistent menus, and native payments — and the success retailers were having with the platform — a dating bot that converted at 30-percent higher rates than any other channel, or Rogers’ customer satisfaction scores rising by 60-percent since providing customer service on Messenger.

But the highlight was obviously what’s to come in Messenger 2.0.

First, and perhaps most exciting, both Spotify and Apple Music are coming to the platform. No more switching apps to grab links for pasting within Messenger, both will now offer native experiences, even on group video chat.

Next, Marcus detailed a desire to “invest in discovery” by offering the best bots and interactions in your local area. What started with 33,000 bots last September is now over 100,000, and discovery is sorely needed. Facebook’s ‘M’ will help with some of this, as well as progress logically on its own — ordering food in a group chat looks pretty cool — and once additional businesses jump headfirst into Facebook’s payment program, Messenger could be an all-encompassing part of our daily lives, a force as disruptive as Facebook itself in its early days.

And last, but not least, Marcus detailed some updates coming to Facebook’s Games platform. The update will feature new tabs that keep track of the games you’re playing, as well as incoming challenges for friends. All told, Marcus said more than 1.5 billion games have been played on the platform to date.

Follow all our coverage from Facebook’s F8 Developer Conference here.