Did you know Neural is taking the stage this fall? Together with an amazing line-up of experts, we will explore the future of AI during TNW Conference 2021. Secure your online ticket now!
A 2002 National Geographic survey demonstrated that only about 17% of adults in the US could point to Afghanistan on a map. And this was less than a year removed from the 9/11 terrorist attacks, so you can be certain the Middle East was splayed across 24-hours news networks ad infinitum.
You’d be forgiven, then, if you didn’t think highly of the so-called “wisdom of crowds.”
But there’s a new AI-human hybrid forecasting paradigm that’s showing incredible improvements over the old “majority rules” method of crowd-sourcing predictions.
Up front: Remember when the New York Times and everyone else predicted about an 85% chance that Hillary Clinton would win the 2016 US presidential election?
Well, that’s because everybody and their mother said they were going to vote for Hillary Clinton. Nobody took the scattered millions here and there who said they weren’t very seriously.
When people were asked who they thought was going to win the election, almost all of them answered: Hillary Clinton.
We should have been asking them to give their opinion on how other voters were going to answer that question.
Background: Researchers at Princeton University came up with a paradigm called the “surprisingly popular answer” that apparently outperforms other crowd wisdom models.
It works by asking an objective question such as, is Philadelphia the capital of Pennsylvania? And then you also ask, will most other people answer yes or no?
People familiar with US geography will know that Harrisburg is the capital, so the correct answer is “no.”
And people from Pennsylvania (as well as students of human nature) will know that a lot of people who aren’t from the state are going to get that one wrong.
The outcome of the survey will, according to the paradigm, clearly show that the correct answer is a surprisingly popular answer.
The number of people who know the correct answer but also assume others won’t, coupled with the number of people who answer incorrectly while assuming others will answer as they did, makes the actual number of correct “no” answers surprising – they don’t follow what the crowd expected.
This might not sound ground-breaking, but when researchers applied math models to algorithms using the surprisingly popular answer paradigm, they found a marked increase in accuracy.
Per a 2017 press release:
Across all topics, the researchers found that the “surprisingly popular” algorithm reduced errors by 21.3 percent compared to simple majority votes, and by 24.2 percent compared to basic confidence-weighted votes (where people express how confident they are in their answers).
It also reduced errors by 22.2 percent compared to answers with the highest average confidence levels. On the 50 test questions related to state capitals—such as the Harrisburg-Philadelphia question—the SP method reduced incorrect decisions by 48 percent compared to the majority vote.
What’s new: A lot’s changed since 2017 in the AI world and the surprisingly popular answer paradigm has gotten a massive upgrade.
According to a new press release, this one from Pennsylvania State University, researchers have figured out how to use SP on ranked lists.
This means it’s as efficacious at finding ground-truth rankings, in order, as it is at predicting the answer to a yes/no question.
Per team lead Hadi Hassani:
One does not require to fully elicit complete rankings and predictions of others. We are able to recover the ground truth by combining both the vote and predictions without eliciting full distributions over all possible n! rankings. And this is true for recovering either only the top choice or the full ranking.
Quick take: Crowds can be stupid, but trying to figure out which individuals are and aren’t subject-matter experts or trustworthy representatives of their population can be as political an exercise as running for office.
We need a method by which people in a crowd who know what they’re talking about can be heard louder than those who don’t – but one that doesn’t rely on identity to discern who deserves consideration.
Behold the power of math! By sorting out who’s right based on predictions about who’s really right about who’s right, we end up in a weird place where the numbers make more sense than just trying to pick a winner.
You can read the Pennsylvania team’s whole research paper here.
Get the TNW newsletter
Get the most important tech news in your inbox each week.