Data workers struggle as complexity grows

Eighty percent of organisations now leverage data across multiple organisational processes, but despite increases in innovation, data workers still waste 44 percent of their time each week because they are unsuccessful in their activities. Data workers spend more than 40 percent of their time searching for and preparing data instead of gleaning insights and, on average, use four to seven different tools to perform data activities, adding to the complexity of the data and analytics process. Among other key findings:

• On average, data workers leverage more than six data sources, 40 million rows of data and seven different outputs along their analytic journey.
• The top frustrations cited by data workers in the survey are indicative of root causes that are responsible for inefficiencies and ineffectiveness. For example, more than 30 percent of data workers say they spend too much time in data preparation, a task that can often be automated.
• Eighty-eight percent of data workers, approximately 47 million people worldwide, use spreadsheets in their data activities. Spreadsheet functions are often used as a proxy for data preparation, analytics and data application development tools but are error-prone and expose the organisation to compliance and trust issues.

“Data is at the core of digital transformation, but until organisation leaders address these inefficiencies to improve effectiveness, their digital transformation initiatives can only get so far,” said Stewart Bond, director of data integration and integrity software research at IDC. “Consolidating platforms and looking for tools that address the needs of any data worker, whether a trained data scientist or an analyst in the line of business, can help reduce the friction that many organisations experience on their path to becoming data-driven.”

The survey found that data workers are unsuccessful for a variety of reasons, including lack of collaboration, knowledge gaps and resistance to change. Participants reported the lack of creative and analytic thinking, analytic and statistical skills, and data preparation skills as the highest-ranked skills gaps responsible for productivity issues, indicative of the pervasive talent gap that exists between data scientists and data workers in the line of business. To overcome these issues and more, many organisations are hiring Chief Data and/or Analytics Officers to streamline analytic processes, build a culture of analytics and encourage data literacy across the enterprise as part of their broader digital transformation strategy.

“Collecting data alone won’t digitally transform a business and the answer is not as easy as hiring a leader, a few data scientists or over-investing in disparate technologies. The key is to empower all users, many of whom are currently stuck in spreadsheets, to analyse data effectively to drive real, business-changing results,” said Alan Jacobson, chief data and analytics officer (CDAO) of Alteryx. “As the data landscape becomes more complex, this survey exposes the tip of the iceberg when it comes to the sheer volume of workers needing to conduct analysis on a daily basis and the untapped potential for them to drive meaningful business impact.”