Are There Enough Data Scientists to Go Around?
Big data and the challenges of analyzing it have never been more important to enterprises. However, if you believe the findings of a recent EMC survey, data analysis is frequently the Achilles heel of enterprises. Why? Because there are not enough people with the right kinds of qualifications to analyze the increasingly complex information being collected by companies.
EMC’s survey of 500 data science and business intelligence professionals revealed that the explosion of digital data and new tools for analyzing it all have created a corresponding explosion in enterprises’ desire to generate value and insights from the data.
But, the survey found, only a third of the pros are very confident about their companies’ ability to make business decisions based on new data. The survey revealed a shortage of talent — a problem that is likely to worsen in the future.
Sixty-five percent of the respondents said they believe demand for data science talent will outpace the supply over the next five years, with most estimating the supply will be sourced from new college graduates. Eighty-three percent of respondents said they believe new tools and emerging technology will increase the need for data scientists.
Other key findings include:
Barriers to Data Science Adoption. Most commonly cited barriers to data science adoption include: lack of skills or training, mentioned by 32 percent of respondents; lack of budget/resources (32 percent); the wrong organizational structure (14 percent); and lack of tools/technology (10 percent).
Advanced Degrees. Data scientists are three times as likely as business intelligence professionals to have a master’s or doctoral degree.
Augmenting Business Intelligence. Although respondents found an increasing need for data scientists in their organizations, only 12 percent saw today’s business intelligence professionals as the most likely source to meet that demand.
Higher-Level Skills. Data scientists require significantly greater business and technical skills than today’s business intelligence professionals. The survey found that data scientists are twice as likely as BI pros to apply advanced algorithms to data and 37 percent more likely to make business decisions based on that data.
Involvement Across the Data Lifecycle. Data scientists are more likely than business intelligence professionals to be involved across the data lifecycle — from acquiring new data sets to making business decisions based on the data. This includes filtering and organizing data as well as representing data visually and telling a story with data.
In a blog post, Chuck Hollis, EMC’s CTO, describes data scientists as the “rock stars” of the future.
“‘Data science’ (however you choose to define it) is a fundamentally different profession with a different profile than the BI analysts that came before it,” he wrote. “Data scientists are more likely to have advanced degrees, frequently have a background in the sciences (vs. business) and they interact with data in more ways — and using different tools. I feel a bit justified with my honest assessment: data science is not your father’s BI.”
Hollis said data scientists are essentially “data experimenters” rather than people who do rote analysis. They’re also likely to interact with IT organizations in far more positive ways than the norm, he opined.
“Neither tools nor people alone can solve the challenges of big data,” said Michael Driscoll, co-founder and CTO at MetaMarkets. “They (tools and people) must work together and that is the promise of data science. Despite advances in software tools, the number of people with experience using these tools, and with real-life exposure to large-scale data sets, is small.”
Driscoll added that data science is a young field, and its growth will be fueled as much by technology as through the mentorship of new acolytes by leading practitioners.
In conjunction with the release of the survey, EMC introduced data science and Big Data analytics, a vendor-neutral learning course. The course aims to help statisticians, mathematicians and others transition to the role of data scientist. Topics covered include an introduction to the data scientist role, big data analytics in specific vertical industries, data exploration and visualization techniques, predictive modeling, analytics on unstructured data and Hadoop.
The goal of the course is to help students lay a foundation for a career as a data scientist, said Alok Shrivastava, senior director of education services at EMC.
“It would be an insult to tell somebody you could come to a one-week long course and become a data scientist,” Shrivastava said. “But what we can do is prepare them to define what data scientist really means — that’s a confusing field by itself — and then start to become a productive member of a data science team.”