We can’t fix the data scientist talent shortage with $40 online courses

The demand for data scientists is at an all time high. In fact, a recent LinkedIn study found that US-based business need more than 150,000 data scientist jobs filled. And they need them right now.

The reason for this demand lies in the massive amounts of data that businesses in every industry are now able to collect through digitization and technology. New data is available all the time, volumes are increasing and businesses need to use that data to optimize functions and, quite frankly, to run at all. Using this data properly can mean the difference between business success and failure, and a data scientist is the key to unlocking the story behind the data.

This demand has spurred a glut of “get rich quick” schemes to get people started in the field, but simply reading a book or taking a $40 online does not a data scientist make. Individuals in this role need years of training and experience to do the job effectively and efficiently.

In fact, a career in data science should be likened to any skilled profession that requires advanced training, education, and experience — such as a doctor, lawyer, or architect. Each of these requires not only a basic aptitude for the job, but also a massive amount of training, education, and knowledge gained through on-the-ground experience.

Data science should be approached with the same amount of rigor, as this role can be responsible for providing the data-driven basis for very real, very expensive business decisions.

Putting data science in perspective

Think about the genesis of this type of work, hearkening back to early scientists. A good example of this was Tycho Brahe, who kept painstakingly detailed astronomical diaries, observing the locations of the planets night after night.

This detailed data, collected over many years, are what enabled Johannes Kepler to discover his three laws of planetary motion, which include the now well-known fact that the Earth orbits in an ellipse around the sun. He crunched reams of data, tested hypotheses, and came up with the revolutionary evidence (pun intended) that Copernicus and Galileo were right.

Data scientists today are doing many of the same workflows, on problems ranging from understanding consumer behavior to predicting disease progression, but with the benefit of significant computing resources. Massive amounts of data are coming in from digital sources, and ambitious analyses can be performed through smart use of mathematics, statistics, software engineering and technologies like automation, machine learning, and artificial intelligence.

Marketing and product development can show a more modern example of data science.

Let’s imagine that a brand wants to see how well a product is doing. The company needs to understand who is buying it, when, where, why and how often. This information is all stored in the data.

A data scientist applies computational and statistical techniques on all the customer data that is flowing in order to find patterns and groups within that data. This analysis can advise the brand about to whom they should be marketing or even if they should change up their product offering.

What’s the job anyway?

A data scientist finds the stories that the data are telling, and separates true patterns and trends from randomness, just as Kepler did when he studied planetary motion. The job essentially lies at the intersection of probability/statistics and software engineering.

Because data points often flow from multiple fragmented and noisy sources, the data scientist must understand the context of the data, set up pipelines to integrate and clean the data, and apply rigorous statistical methods to coax out what the data are trying to tell us.

It is vital that they do their job correctly and accurately because, as mentioned above, the stories that data scientists weave from the data feed directly into business decisions connected to real money and risk. Done correctly, data science can be a transformative role within any organization, converting abstract data sitting in databases into real-world insights and effective actions.

When seen in the light of basic supply and demand, it is easy to see that there is a talent gap that shows no signs of lessening. Those that are entering the job market to fill the gap are inexperienced workers who can ultimately cause businesses a lot of pain in the short term. In the long term, experts will emerge and hiring managers will become more savvy as to which skills to seek out.

In the meantime, a consultancy model that marries statisticians/mathematicians, software engineers, and project management all under one roof may be the solution. By engaging a team of qualified experts to advance an in-house data science program, businesses incur lower project costs and execution risk, efficiently leverage the latest advances in the field, and more quickly get to the heart of what their data are trying to tell them.

TNW Conference 2019 is coming! Check out our glorious new location, inspiring line-up of speakers and activities, and how to be a part of this annual tech bonanza by clicking here.