Big Data — useful tool or fetish?

In 2012, the Obama administration unveiled a two hundred million dollar initiative to facilitate the growth of the Big Data industry and demonstrate its potential in large-scale markets. However, the initiative was an apparent failure. Despite government endorsement, Big Data has been unable to develop into an industry in its own right; the analysis and evaluation of billions of data points remain a trend rather than an emerging technology.

Tom Kalil, Deputy Director for Policy at OSTP, released a statement on behalf of the Obama administration in which he declared that “to launch the initiative, six Federal departments and agencies will announce more than $200 million in new commitments that, together, promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data”.

The statement went on to say, “we also want to challenge industry, research universities, and nonprofits to join with the Administration to make the most of the opportunities created by Big Data”. Unfortunately, high hopes and large investments did not translate to practical and effective solutions, and the US Government failed to demonstrate the potential of Big Data in the industries they had targeted.

Why the US government’s initiative failed

The downfall of the Obama administration’s initiative was undoubtedly the result of the increasingly negative perception of Big Data. For many Big Data represents the impractical and uncritical “fetishization of data”, or the unnecessary aggregation of large sums of information. The US government effectively placed a very large bet on the expectation that Big Data would evolve into an industry in its own right, however, it has distinctly failed to do so. Conversely, other forms or branches of Big Data, such as data visualization used by Netflix and leading technology service providers, have succeeded.

Since the launch of the Hype Cycle, technology consulting firm Gartner has failed to include Big Data on its list of emerging technologies. The Gartner report in 2015 extensively covered Virtual Reality (VR) and Artificial Intelligence (AI), based on their data evaluation tools for technologies and hype cycles. However, due to its lack of adoption and maturity, Big Data did not meet the requirements to be considered an emerging technology. In comparison to these other industries, it has shown very limited signs of growth. The incredibly modest demand and enthusiasm for Big Data in the market has meant that many consulting firms like Gartner have continued to dismiss it — even with large-scale government initiatives.

“Gartner Hype Cycles provide a graphic representation of the maturity and adoption of technologies and applications, and how they are potentially relevant to solving real business problems and exploiting new opportunities”, said the Gartner team. “Gartner Hype Cycle methodology gives you a view of how a technology or application will evolve over time, providing a sound source of insight to manage its deployment within the context of your specific business goals”.

Despite this, it is important to note that Big Data has been embedded into almost all the operations and services of social media, technology and streaming platforms. However, firms like Netflix which actively use Big Data, prefer the term “data visualization” rather than Big Data due to its negative connotation.

How Netflix has demonstrated the potential of Big Data

Popular social media, technology, and streaming platforms like Netflix collect billions of data points on a daily basis in order to offer video recommendations and customized product marketing to their users. Earlier this year, the Netflix development team noted that data visualization is of “paramount importance” to the platform; it allows them to provide users with recommendations for movies, TV shows, and concerts based on their preferences and viewing history.

At the Hadoop Summit in 2013, Jeff Magnusson, the manager of data platform architecture at Netflix, offered a rare insight into the company’s utilization of sophisticated data visualization and Big Data technologies. During the presentation entitled “Watching Pigs Fly with the Netflix Hadoop Toolkit”, Magnusson stated that the data should be accessible, easy to discover, and easy to process for everyone. “Whether your dataset is large or small, being able to visualize it makes it easier to explain”, he emphasized. “The longer you take to find the data, the less valuable it becomes.”

Since early 2013, Netflix has continued to develop advanced data visualization tools, which have been a leading factor in the company’s success. As shown in the screenshot of Netflix’s data visualization tool, the Netflix platform has been relying on Big Data to investigate and analyze the consumer base for each Netflix show, the preference of genres based on age, and other important data that is crucial for both the company and the consumer base of Netflix.

The problem faced by large conglomerates in major industries, with regards to utilizing Big Data, is not necessarily technological. Instead, it is a problem with the misrepresentation of Big Data and negative public opinion, which has gotten worse over time. This is in part due to the fact that the average consumer has become significantly more privy to issues surrounding online privacy.

In the long-term, Big Data could eventually solve public policy challenges including privacy. As Economist and Data Scientist, Matthew C. Harding pointed out, we are dealing with really complex problems in Big Data which require a complex interdisciplinary line of approach and behavior change. Harding has been using Big Data to answer crucial policy questions in Energy/Environment and Health/Nutrition. He believes that it is necessary to find ways to change human behavior. “Big Data is a tool that we have which allows us to make better decisions”, he stated in the talk.” I’m not afraid of Big Data.”

This post was written by Joseph Young for Binary District, an international сollaborative technology community which creates unique competency-based workshops and events on new technologies. Follow them down here: