Facebook is working on AI that understands what’s in your photos

To make your Newsfeed better, Facebook is working on artificial intelligence that can understand what’s going on in your photos.

Speaking at Web Summit, Facebook’s CTO, Mike Schroepfer, detailed a new system the company has been developing to understand what’s happening in photos and translate that into natural language elements.

One demonstration showed a photo in Newsfeed of a man skateboarding, which the computer deconstructed and described in words to understand what was happening.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

It broke it down into what it saw “a skateboard, a man, a trick, his skateboard” and what may have been happening “doing, riding, is doing” and constructed a description of what was occurring.

Schroepfer said that “when the machine understands attributes of an image” it is “easy to understand the applications to Facebook.”

The technology would allow the company to “do the best job of showing you what you want in your feed by understanding photos based on the pixels.”

It doesn’t appear to be in use on the service right now, but rather as part of a system that’s being researched for future use at the company.

Another demonstration showed a prototype application using the same technology but targeted at visually impaired users. A photo is given to the app, and you can ask questions using speech in which the AI will describe in words.

Facebook said today that it’s planning to further its AI team’s research in a new paper at NIPS, an artificial intelligence conference next month — it’s reportedly 30 percent faster than industry standards right now.

One day your Newsfeed might be able to figure out what’s in photos shared by friends and show them to you based on what you actually like to see.

It’s a slightly creepy way to imagine a future iteration of the service, but Facebook is pursuing a world where it only displays to you things you’re interested in. That future means Facebook knows even more about you based on what’s in your images, but it felt kind of inevitable.