[Best of 2019] How AI will save us from the mess of big data

We are drowning in data that’s supposed to help us. The big data revolution has come from inexpensive data storage and automated data collection, giving organizations exponentially more data than they had in the past. From business logic data like computer log files to detailed weather patterns, purchasing data to TV viewing, we have access to so much data on how the world works and about the lives and habits of individuals.

A lot of this data is just sitting in warehouses, either real or virtual, being ignored or occasionally examined for trends. It’s just too much for people to handle.

When it comes to healthcare, advanced new diagnostic techniques mean that more information is collected about patients than ever before.

Doctors can’t and don’t ignore new information about their patients. Once a diagnostic option is possible and accessible, they then use it to save lives. In the case of CTs and MRIs, for example, these once-rare diagnostic tools used only in the most complicated of cases, are now routine. As CTs and MRIs themselves become more accurate, they now take more images of each person per exam, too. A recent study found that the average CT exam took 82 images of a patient in 1999; by 2010, this number had risen to 679 per patient, a 730 percent increase.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

That’s good news for patients, who can get more accurate diagnoses. But it puts a huge strain on radiologists. Thanks to the digitization of medical devices, radiology professionals are having to sift through far more data. The same study revealed that the average radiologist has to interpret one image every three to four seconds in order to keep up with their demanding workload.

One response to this new workload is, of course, to hire more radiologists. But radiologists are doctors, and it takes time to train them with their highly-specialized skills. Even then, there’s a scaling issue. Say we’re capturing 50 times more images than 30 years ago. Clearly, training and hiring 50 times more radiologists isn’t a viable option!

That’s where AI comes in.

What’s AI doing for radiologists? Older AI Computer Aided Diagnosis solutions detected one condition and often required the radiologist to manually send the images for inspection. This made using them a lengthy hassle. Newer solutions are ‘always-on’ and check every image automatically, sometimes for a range of different abnormalities.

It’s not just spewing out diagnoses. Some AI solutions prioritize cases, working out which cases should be seen by a doctor first because they have an urgent condition like an embolism. A patient with cancer probably requires major surgery within weeks, but a patient with an intracranial hemorrhage could have only hours to live unless immediately treated.

AI is not trying to replace radiologists — or other profession that can benefit from big data — with algorithms. Instead, it’s a productivity tool for radiologists, that helps them keep on top of patient care. Radiologists analyze more than 100 studies (each study with hundreds of images) in a long 10-12 hour workday. AI can improve accuracy despite this heavy work burden. A report by Accenture found that AI is redefining healthcare delivery and augmenting human activity.

There is still plenty of work that needs to be done to assess the value of AI in healthcare and to ensure that it reaches its full potential. It isn’t perfect yet and there are still limitations and lacunae. But AI has moved past the “shows promise” stage into “delivering value.” The US Food and Drug Administration has approved a range of new AI-powered solutions for the medical market, and many have EU CE marks too.

The data is going to keep coming. CT and MRI machines will become more accurate and faster, and able to take even more images per person. AI is one of the only ways to help doctors keep up with this deluge of data, allowing them to spend more time working with patients.

AI isn’t going to make business data analysts obsolete; instead, it’s making it possible for them to handle the hundreds of millions of transactions that a company like Amazon deals with. AI isn’t going to put IT operations out of a job; without it, they’ll never really be able to handle the sheer quantity of log data.

A similar trend happened in automobile manufacturing. In 1928, almost half a million people were employed in the US auto industry; today it employs nearly a million people. Robotic automation didn’t cost auto jobs; instead, it made it possible to build more advanced cars.

There’s an irony here. In order to train the deep learning models that make AI accurate and powerful, you need data. Lots of data. It’s big data that makes advanced AI possible in the first place. But equally, in order to take full advantage of the big data revolution, we need the AI revolution.

AI is saving us from big data. Instead of drowning in an endless ocean of data, AI is the lifeboat that lets us navigate it. In some cases, it can help businesses operate better or software applications perform faster. In medicine, it can help doctors use all the data they have to save lives.

TNW Conference 2019 is coming! Check out our glorious new location, inspiring line-up of speakers and activities, and how to be a part of this annual tech extravaganza by clicking here.