AI developed by Microsoft and Alibaba, independently, last week became the first to beat humans at reading comprehension. The reported implications of this technological advancement seem to indicate machines with the ability to read better than humans have arrived or will very soon. This probably isn’t the case, however.
What this definitely means is chatbots are getting a little better at their jobs, update-by-update, thanks to cutting-edge research in deep learning networks. And two giant companies have set the bar for accuracy higher yet again.
Of course there’s a little more to it than that, but some experts think “AI can read better than humans” as Newsweek put it, might be a tad hyperbolic.
The deluge of misleading reports about machine reading has indeed begun. Here’s another that is spinning the same result (basically, that machines can highlight text relevant to a query) into Armageddon @erikbryn don’t believe this for a minute. https://t.co/aH3qaRIYIR
— Gary Marcus (@GaryMarcus) January 15, 2018
Prepare yourself for a batch of grossly misleading reports on machine reading today. The SQUAD test shows that machines can highlight relevant passages in text, not that they understand those passages. https://t.co/Jdxt5U0j6w
— Gary Marcus (@GaryMarcus) January 15, 2018
Machines from both companies achieved higher scores than a group of humans answering queries from the Stanford Question and Answering Dataset (SQuAD).
The test has (human) participants read a Wikipedia entry and then answer questions about it. AI has access to the entire Wikipedia database, thus it’s able to answer a plain language query like “What year did Ghengis Khan die?” and search for the correct answer — especially when the database is limited to a specific article. This represents a feat closer to ‘memory and recall’ than ‘reading and comprehension.’
The AI made by Microsoft and Alibaba to pass the test isn’t unique. The companies participated in a competition where numerous other companies also developed AI to take the test. And the machines only outscored humans by a margin of 82.4 (Alibaba) and 82.6 (Microsoft) to 82.3 – which is hardly indicative of millions of jobs suddenly being at stake.
Why the hyperbole?
Because it’s just so delicious, to be honest. It’s China versus America on the biggest stage we can imagine: the future.
We want to believe in a narrative where Alibaba and Microsoft represent a race to see which company’s home of origin will win the final sprint towards creating the artificial intelligence that will rule us all.
It would be more prudent to care about the truth: these machines are only a few iterations away from becoming useful.
Current AI isn’t very smart. It’s magnificent at sorting data, like a toddler who can point to all the red cars, but it isn’t very wise.
I asked Alexa, Amazon’s popular AI virtual assistant, to recommend a good book on deep learning. The exact command I used was “Alexa, do you recommend any books on deep learning?” and it responded with “Sorry, I don’t know that one.”
Advances in word recognition and comprehension can only make AI better at handling those kinds of requests. Machines that can better understand our questions will give us better answers. That could mean a doctor getting more accurate and actionable information with which to base a diagnosis, or a teacher more clearly representing a concept to a student.
When these things can do more than perform illusions and parlor tricks (for more information see: The Chinese Room Argument) they’ll finally become useful.
But passing the SQuAD with a higher score than people doesn’t indicate machines have learned how to read, and certainly not better than humans.
Get the TNW newsletter
Get the most important tech news in your inbox each week.