Inside Google’s plan to stalk your social media accounts

Google, once again, is excited about social media. But not in the ways you might think; this isn’t about another in a failed string of chat apps, or the knockout success that never was in Google Plus. Instead, it’s an entirely new way of recognizing human faces, and one made possible by — you guessed it — creeping on your social media profiles.

In a recently approved patent, Google detailed a system of “Facial Recognition with Social Network Aiding.” It’s exactly what it sounds like, an attempt to parse social connections as a way to better identify your mug.

Reverse Image Search works by attempting to match visual cues in photos with other, similar photos. It works mostly as intended, with some results being dead-on, and others complete misses. But as long as the photo is well-lit, and is of a high enough resolution, Google usually nails it.

Its a system that works, but could use improvement. Sometimes the results are only semi-relevant, and for those who don’t have as much data to pull from, it often overlooks matches entirely. Grab your friend Katie from middle school’s Facebook photo, for example. Chances are it only returns the Facebook account using the photo.

But the new system envisions a world where artificial intelligence would identify faces using not only visual cues, but various forms of data from personal communications, social networks, collaborative apps, and even your calendar.

It’s like this

Users enter a “visual query” in the form of a photo, screenshot or scanned image. The system would then analyze the image and look — using advanced image recognition — for others that are both visually similar, and a potential match using the data sources mentioned above.

Rather than just matching based on visual cues — Google’s current system — the software would take additional steps to build further confidence in the person’s identity. Pulling from a number of data sources, including your hometown, age, occupation, and various others, Google would strive to be absolutely sure it returns images relevant to your query, based partially on the data it pulled from your social accounts and other collaborative apps.

Let’s attempt to wrap our head around this with a real example. Inputting an image of Amanda, from accounting, for example, should lead to relevant results with the new software due to a number of connections between you, the searcher, and her, the subject of your search. You’re connected by your employer, you’re friends on Facebook, and she routinely retweets you on Twitter. To Google, these all build confidence that Amanda’s image is indeed the one you’re looking for.

Or maybe you’d search for a childhood friend, Jeff. The same reasoning follows: you’re connected by your hometown, your high school, and an approximate age range.

Where the patent falls short is in detailing exactly how Google would use the technology. We can be fairly sure that Reverse Image Search and Google Photos would benefit. The latter, due to some wording that describes automatically sharing a group photo and tagging everyone in it.

But it’s other, less obvious use cases that could be worrisome. If Google were to bring back Glass, for example. Or if it were to reverse course and continue to provide artificial intelligence to the military. If you’re looking for a tinfoil hat example, that exists too; Google could begin feeding this data to law enforcement, like Amazon.

Rekognition, Amazon’s feature-rich AI, provides real-time facial recognition to a handful of police departments in Orlando, Florida, and Washington County, Oregon. It’s not difficult to envision a competing product from Google.

The patent does note that in certain scenarios — scenarios Google fails to define — a person would have to opt-in to have their identify appear in these results. For now, it appears that we’re in the clear. But since this is only a patent, and not working software — to the best of our knowledge — things could change quickly.

It’s something to keep an eye on, but we certainly wouldn’t lose any sleep over it. At least not yet.