Facial recognition company Face.com’s CEO talks APIs, mobile and the London riots

What do you do when you set out to leverage the power of existing facial recognition technology to launch a new business, only to discover that the existing technology isn’t really up to scratch? That was one quandary the good people at Israeli startup Face.com encountered. And here’s its story.

Gil Hirsch & Face.com

I spoke with Gil Hirsch in the build up to his talk at the Wired 2011 event in London today, he’s the CEO and one of four co-founders of Face.com. Their story can be traced back to a few years before work started on Face.com. “We set up this community of geeks here in Israel called the Garage Geeks“, says Gil. “Since 2004, many industry leaders came in, such as Sergey Brin, Jimmy Wales and Tim Armstrong. Whenever they visited Israel, we’d normally have them over for an unofficial talk with the local tech community”.

It was within this community that the idea for Face.com started to take shape. “The one thing we noticed, around the time we were going to launch Face.com, was that people were taking a lot of photos and uploading them in bulk to Flickr, Picasa, Facebook and tagging them manually, or not at all”, says Gil. “So we thought – wouldn’t it be cool to have this tagging process optimized by throwing some technology into the mix?” So that was the basic premise behind the initial Face.com concept.

“During one of the events we set up at Garage Geeks, a few of us met up and decided to do this”, continued Gil. “We are techies, but we thought that face recognition was already a solved problem. So when we first set out to do this, we assumed we’d take something off the shelf, and try to use that in combination with our knowledge of social networks and how to combine all these things together”.

It transpires that they were very wrong though. “We didn’t really know at the time what the level of facial recognition technology was”, says Gil. “But it was clear that it needed to be taken to the next level. This meant the quality of the results had to be greatly improved. The existing technology was okay-ish for small circles of friends and family, such as the Picasa tool on the desktop, or iPhoto. But these are very basic technologies, they don’t scale to a social graph”.

Face recognition in the wild

“We also noticed that people were taking photos that perhaps weren’t of the highest quality”, says Gil. “Particularly on Facebook. Only parts of the face were visible, people were wearing sunglasses, hats, there was bad focus, poor lighting and low resolution photos too”.

So this was the environment in which they set out to try to solve the problem of face recognition. “It’s actually known as ‘face recognition in the wild'”, says Gil. “Where people aren’t facing the camera, or being told to take off their hat or to smile. So that was the first problem we had to face. The second problem we faced to try and scale the technology to cater for billions of photos”.

So during 2008, with a new CTO on board, the company set out to build this technology from the ground up and in 2009 the first Face.com apps started to surface: Photo Finder and PhotoTagger.

In a nutshell, Photo Finder allows you to scan all your friends’ uploaded Facebook photos to find untagged images of yourself and your friends. But what about friends-of-friends, and what about the rest of cyberspace? “We’ve engineered privacy into everything we do”, says Gil. “Privacy is in the design. It’s been built to scale on a social context, but not to allow you to scan the whole of the Internet for someone. That isn’t even a use-case for us. It’s all about the social context”.

Photo Tagger is a simple app for grouping and identifying the faces in an album of photos, which then suggests tags. Users can confirm (or correct) the suggestions, and tag entire albums in a fraction of the time it takes using manual tagging methods. So if you have a plethora of wedding or party snaps that you want to tag the users in, this has the potential to save you a lot of man-hours.

Benchmarks

So the Face.com technology has been designed and developed from the ground up to be the best technology of its kind. But is it really the best there is? Surely governmental or security agencies must have superior technology? “It is the best”, says Gil. “And there are a couple of different benchmarks. The one we’re using is called Labeled Faces in the Wild which is run by the University of Massachusetts. We publish our results there on a regular basis. We recently announced that we had a 30% jump in accuracy. We’re pretty confident that we’re up there”.

The Labeled Faces in the Wild benchmark that Gil talks of is the standard testbed for unconstrained (“in the wild”) face recognition. It recently achieved a mean accuracy (average score) on the test set of 91.3% +/- 0.3, which by all accounts is the best results reported to date.

Whilst the goal here isn’t to nit-pick, I was genuinely curious to know how Face.com’s technology would fair against the sort of surveillance technology used by security agencies. “I do want to stress that it’s very different technology to that being used, say, in airports”, says Gil. “I don’t know how we would compete with that sort of technology, as we haven’t tested it on that data set or on those use cases”.

Hacking the Face.com API

Face recognition technology for consumers to use in photos. But would it work in videos? “Sure, we’ve seen videos being used with our technology”, says Gil. “After Photo Finder and PhotoTagger, we opened the Face.com API in 2010, which enabled external developers to build stuff on our technology. Although we’re about photos, videos consist of individual frames – which are essentially photos – and people have used it in that way. We’ve had 25,000 developers use our system and we’ve seen a wide range of uses of it”.

So what are some of the cooler use-cases of the Face.com platform? And how have developers been innovating upon Gil & Co.’s own innovations? “One that springs to mind is some the one where someone wrote a plug-in for the Boxee“, says Gil “It allows you to pause a movie and it will tell you who the actors are. Someone else used this idea to tag YouTube videos containing celebrities”.

I asked Gil for a few more of his favorite ones, and there are some quite cool, funny and bizarre use-cases. For example, there’s The Emotional Breakdown which is an experimental visualization tool that examines the mood of people in photographs from The Guardian’s website, summarizing whether people were largely happy, sad, angry, surprised or neutral on a given day. And then there’s Find Your FaceMate which is a ridiculously simple but clever dating site which works on the basic premise that humans are attracted to people that share similar features, and matches people up accordingly.

London Calling: Face.com as a crime-fighting tool?

During the riots that permeated London and other parts of the UK back in August, a Facebook Group called London Riots Facial Recognition popped up, which sought to tap into Face.com’s technology and use it as a sort of crowdsourced crime-fighting tool to identify looters.

Without delving too far into the ramifications of using the technology for vigilante justice, I asked Gil what his thoughts on this were, and whether it could actually be used for such a deed. “I know they tried to use it, but it would be pretty difficult”, he says. “For it to work, they’d have to get a lot of people to opt-in. The whole system is based on opting-in, you can’t just hijack people’s photos”.

Talking numbers

Face.com’s apps and API services scan billions of photos each month.”We’re getting thousands of photos per second coming through our servers, a lot via the API”, says Gil. “We’ve seen a 400% increase from our latest figures compared to the same period 12 months before”.

But how is it funded and how does how does it make money – does it charge for access to its API? “For most developers, the answer would be no”, says Gil. “We have a 100%, completely free service that’s just limited by how many photos you process in an hour. So you can process over 100,000 photos each day and it’s free, in fact you could probably process a lot more than that and it would still be free. When it comes to payment, we’re only after those very big sites that are hugely successful and making a lot of money of it. We’ve geared our entire offering so you can build a service for free and not really have to worry about money”.

Face.com has secured $5.3m in funding in total so far, $1m of which was brought in before launch, with the following money arriving a year and a half later. So in the past two or so years, how pleased is Gil with their progress? “We’re thrilled. We never knew it would take off the way it has”, he says. “In a sense it has been a case of ‘if you build it, they will come’.

We weren’t’ sure how many people out there were waiting for this type of technology to become available, and how much business there was to be made around face recognition technology. I really enjoy the fact that over the past few years that we’re seeing a wider adoption of face recognition technology not being used for security or surveillance, but being used to hack our lives together”.

“We’re working hard to develop mobile”, says Gil. “How to speed up, for example, photo-tagging on mobile. We’ve had a couple of demos running on mobile, with other people developing the applications, we’re happy with the results, but we think we could do it a lot better. So that’s a big focus for us – how can we improve the mobile experience around face recognition?”