Google and ProPublica launch machine learning tool to track hate crime in America

Documenting hate crimes in America has long presented a challenge to researchers, journalists, and social scientists. Record keeping in this area has been patchy at best, with data either woefully incomplete or non-existent.

Yet, it’s never been more important that we understand them, especially after the events of last weekend, and the violent, racially driven crimes that followed Trump’s shock electoral victory — committed not by ordinary Trump supporters, but rather an emboldened and ascendant far right, who are a small yet increasingly visible and vocal presence.

To try and make better sense of the issue, Google and ProPublica have teamed up to create an artificial-intelligence powered tool that parses through news articles, in order to build a bigger picture of where hate crimes are occurring across the country. Here’s how it works:

The Documenting Hate News Index — built by the Google News Lab, data visualization studio Pitch Interactive and ProPublica — takes a raw feed of Google News articles from the past six months and uses the Google Cloud Natural Language API to create a visual tool to help reporters find the news happening all over the country, from Oklahoma to Florida, California to Kentucky. It’s a constantly-updating snapshot of data from this year, one which is valuable as a starting point to reporting on this area of news.

The tool aggregates articles that suggest “hate crime, bias, or abuse.” This could include things like anti-semitic graffiti, or information on relevant court cases. Google plans to monitor the dataset to ensure that no erroneous events creep in — like articles that simply mention the word ‘hate.’

Users of the Documenting Hate News Index can search by date, or with keywords. It also algorithmically suggests relevant keywords, like names of offenders, places, or categories of hate crime.

And to ensure that the information is always current, the database updates on a daily basis, and dates back to February of this year. Third-party researchers can access it through Google Trends’ Github page.

Anything based exclusively on news articles will have its limitations. The majority of hate crimes go unreported. It stands to reason that even fewer make their way to the pages of the media.

But it’s also true that the current system isn’t fit for purpose. While the FBI is legally obligated to collect information on hate crimes, they’re dependent on local authorities reporting them. As there’s no requirement to do so, not all do.

Hopefully, Google’s tool will be a helpful tool to journalists and researchers alike. And if that translates to better policy, bringing down the rates of hate crimes, even better.