The answer is: Yes. Here’s why.
While you may not need data scientists to harness big data, you do need them to understand it, and more importantly, to use it along with statistical and modeling techniques to better answer business questions.
Data scientists have been characterized as computer scientists that can tell the story of data by using business understanding, statistics, analytics, and math. Big data gave very high visibility to the previously obscure role of a data scientist. We could easily conclude that big data is too complex to handle and that we need data scientists to manage it. If we imagine a neatly segmented utopia with all the unstructured and semi-structured data analyzed, and things like buzz index and sentiment algorithms run and ready to use, would we still need data scientists then?
The answer is still: Yes. Just like you need a chef to come-up with something on the fly based on the specific need of the customer, the ever-changing incoming ingredients and the capabilities of the kitchen, a data scientist is skilled at managing incoming big data, and can achieve the best results possible from the available ingredients, techniques, and equipment for that specific customer.
It is the Hadoop map and reduce master chef who actually prepares the data. It is the data scientist who actually uses the data to answer the business questions. Data scientists deal with big data on a day-to-day basis so they know how to get around and access the data. With big data there are just more variables to look at before we can answer a relevant business question. Most likely we will need the:
- Applied statistics to make sense of the numbers.
- Understanding of the business and the business question to apply the right statistical techniques.
- Knowledge of data management (including big data) and how to access and manage such data.
It is the data scientist who draws on these disciplines to get to the bottom of what data communicates, and selecting the right data points from multiple sources to resolve key business challenges.
Although big data brought the data scientist role to prominence, it is not the complexity of handling big data for which we need data scientists but rather the increased dependency on applied statistics and business knowledge to interpret the outcome.