Data & Analytics Market Evolution Over the Last Decade

There has been an increase in the usage, volume and importance of data and analytics in the last
decade. This dynamic brings new opportunities and risk to organizations.

The writer of the article has presented on many data and analytics events during the last 15 years.
On three of them, in 2014, 2018 and 2023, a series of questions were presented to the audience.
The intent was to summarize members’ views on data and analytics, risk and benefits realized, and
progress made during the last decade. Those events were attended by more than 150 attendees,
providing a diverse range of industry knowledge, backgrounds, and professional experience.

During those events, 10 questions were put to the audience about how organizations are using data
and analytics and concerns about the possible changes that may arise. Those questions were the
same on the mentioned events. While it was acknowledged that some of those questions may be
slightly outdated, organizers decided to ask the same questions to accurately assess progress.

It is worthwhile to summarize the answers the audience provided, comparing any similarities or
differences between the results of the survey in 2014, 2018 and 2023 noting any key comments and
thoughts raised by the audience, and summarizing predictions and practical steps for organizations
in the next years.

Question 1
Is there anything new about big data (figure 1)? The response options were:

• No, there is nothing new.
• It is a fad and it will all disappear when people realize nothing has changed.
• It is new, but it is not going to change the world.
• It is going to change the world.

Result

In 2014, more than 61 percent of the audience believed that the changes brought about by analytics
would be significant, drastic, and short-term change would occur in organizations’ operations. In
2018, this view has not changed, as most of the audience still believes that is going to change the
world. Comments raised during the event indicated that changes expected in organizations are taking longer than they anticipated four years ago, and some members of the audience did not feel
that their organizations had implemented specific examples or programs.

Comments from the audience in 2023 are consistent with the previous survey. Progress made in
organisations has been significant, however, hype generated previously made expectations higher.
The audience expected a larger impact and change and believe that bigger changes are coming in
the next few years.

Question 2

Is data and analytics changing your business (figure 2)? The response options were:

• No changes so far; no changes soon.
• No changes so far; there will be changes soon.
• Some changes so far, but they are minor.
• Big changes so far; more in the near future.

Result

Interestingly, the responses to this question when asked in 2018 were quite similar to those
provided in 2014, but quite different to the ones in 2023. Most of the people in the audience
believed that there have been changes and they will continue during the coming years. Some of the
comments indicated that the speed of the changes varies depending on the sector and size of the
organization. The audience noted that the industry with the most changes has been financial
services, in which there have been larger investments in team and infrastructure. Some members of
the audience pointed out that several organizations or teams claim to make data-driven decisions,
but nothing has really changed. Changes in 2023 answers were mainly driven by recent development
in Generative AI.

Question 3

What will stop you from embracing Artificial Intelligence (AI) (figure 3)? The response options were:

• Skills
• Technology
• Management’s disinclination to embrace it.
• Do not know where to start.

Result

Skills continue to be one of the key challenges to embracing big data; however, during the last few
years, organizations have partially addressed this through effective training and hiring strategies.
Universities have incorporated Data Science as one of their mort important and acclaimed degrees
and master, which helped to solve the problem. This is evident as enterprises now have roles such as
chief data officer (CDO) or chief data strategist (CDS), both roles that did not exist a few years ago.
The responses also identified that management disinclination is an increasingly big challenge when
implementing data analytics programs. The percentage of respondents indicating so has increased
drastically during the last years.

Question 4

What do you believe is the main risk of not embracing AI (figure 4)? The response options were:

• Regulatory noncompliance
• Lose advantages against competitors.
• Do not react to potential issues on time.
• Miss potential opportunities

Result

It was broadly recognized that the loss of competitive advantage and missed opportunities
constituted the main risk scenarios of not implementing data and analytics programs. However, the
fact that they can miss potential opportunities has been consistently the second concern in the list.

Question 5

Does your organization use data analytics tools (figure 5)? The response options were:

• No.
• Does Excel count?
• Well, we bought them, but we have not used them.
• Yes, and we use them to analyze our data.

Result

As expected, there has been an increase in the usage of data analytics tools. More than half of the
audience affirmed using data analytics tools to analyse data. Some comments from the audience
recognized that this is mainly because tools and processing have become cheaper and more
accessible, and open-source tools are available in the marketplace. However, in several cases, there
were comments that these tools are managed by IT and not embedded in business processes.

Question 6

Does your organization collect social media data (e.g., Twitter, LinkedIn, Facebook) (figure 6)? The
response options were:

• No.
• Minor social media analysis. Our marketing people look at Twitter and Facebook.
• We have done some analysis of social media data.
• We regularly capture, analyse and act on social media data.

Result

The majority of the audience recognized that unstructured data (e.g., social media) are not collected
or used properly for further analysis. Despite this, there has been an increase in the usage of
external data sources recently. Some comments from the audience reflected that their organizations
are making and have made some progress but are in the early stages of the journey. The key take
away from this question is the consistent increase in collection and analysis of this type of datasets.

Question 7

How do your customers feel about data issues (figure 7)? The response options were:

• They have no interest in data issues.
• They care only about privacy and security.
• They are interested in how we use and profile our data.
• I have no idea.

Result

Customer data privacy and security were key issues for the audience. The topic of the EU General
Data Protection Regulation (GDPR) was prominent, particularly as the enforcement of the regulation
began only two months after the 2018 survey was conducted. The main difference between 2014
and 2018 responses is the increase in customer interest around data profiling. Some examples were
raised such as Netflix or Amazon, organizations that are using machine learning techniques to profile accounts to proactively suggest movies or products, minimizing customer effort and improving the
user experience. The answers in 2023 suggest that the security aspect was not as key as it was
previously, and the audience was more interested about data usage and profiling.

Question 8

Do you have the data you need (figure 8)? The response options were:

• No.
• Yes, but the quality is poor.
• Yes, we have a good set of data.
• Yes, we have a good set of data and we enrich it with external sources.

Result

The consensus was that organizations are collecting more data from more sources. However, there
were some concerns among the audience about the quality of the data and the impact on the
results. Alongside this, respondents stated that data collected from customers differs depending on
the industry. Telecommunications and financial services were identified as being the sectors that
collected the most data.

Question 9

What do you think is the biggest advantage of AI (figure 9)? The response options were:

• Greater confidence in decisions
• Faster decision-making
• Richer engagement with customers
• Better regulatory compliance

Result

There has been a change in the perception within the group regarding the biggest advantage of AI.
Richer engagement with customers was again identified as the most important aspect; however,
respondents assigned more significant to confidence in decisions in 2018 when compared to 2014.
Some examples of richer customer engagement were specifically highlighted. These included the
Netflix and Amazon examples previously mentioned and some use cases from financial services.

Question 10

Where in your business do you believe that AI is going to have the biggest impact (figure 10)? The
response options were:

• Finance
• Customer management/relations
• Operations
• IT management

Result

The survey respondents indicated no changes in this area during the last years. Customer
management is the area identified by the audience where analytics can have the biggest impact.
Some comments from the audience indicated that depending on the nature of the division, the
usage may be different (e.g., usage in technology or operations departments would be different than in marketing), but it was recognized that benefits in customer relationships would dominate and
have been increasing during the last years.

Conclusion

The audience recognized that most organizations started their journey toward building a data
analytics capability a few years ago. Organizations have built stand-alone data analytics teams, and
the role of CDO has been created in most large organizations. However, the speed of change is
slower than expected a few ago.

The survey showed that the commercial side of the organizations is the area where these
technologies have had a bigger impact. Customer management and the identification of new
business opportunities have been the areas where these technologies have been implemented. This
is due to the presence of a clear business need, which makes it easier to convince stakeholders.

On the other hand, the main concern identified a few years ago was that the lack of experience and
skills implementing these technologies could cause data quality issues, which may lead to inaccurate
analysis. To help mitigate these issues, organizations should implement controls such as
reconciliations or validation tests to ensure the validity and security of information. These concerns
were prevalent in both surveys.

It was also recognized that the impact of data and analytics is dependent on the nature of the
organization. Larger organizations with retail customers tend to have larger databases and the value
of these techniques is higher.

During the event, it was broadly recognized that the speed of building data analytics and science
capabilities will be faster during the coming years, compared to previous years. Organizations have
access to tools, data and processing (e.g., cloud solutions) that they did not have previously, which
can provide advantages over competitors.

Some of basic practical steps mentioned during the event to successfully implement data analytics
and capabilities were:

• Data quality and governance—Get the foundations right and ensure that the data quality on
  source systems is adequate and fulfils business expectations.
• External sources to complement your information —Value from data is maximized when
  enriched with other data.
• Get external help if needed — Organizations may not have the expertise needed in-house.
  Bringing in external help may be beneficial to set up the infrastructure and upskill
  employees.
• Walk before running—Prove the value of the solution on a small scale before implementing
  the final solution.
• Business cases for everything—Identify areas where these solutions can provide value to
  the business and prioritize them.

The audience recognized that even if key changes have been made, most of the changes will take
place in the next few years.

About the Author

Angel Serrano is currently running the data science team at Santander UK. Previously, he built one of the
data and analytics teams at PwC UK, focusing on analytics, data science and business intelligence in the financial services sector. Angel holds an MBA and has been an active public speaker on Data & Analytics events and articles writer. Angel has been selected during last 3 years in the top100 DataIQ as one of the most influential data and analytics practitioners and is a member of the BoE/FCA Reporting and Data Standards Transformation Committee.

Sign up for the free insideBIGDATA newsletter.

Join us on Twitter: https://twitter.com/InsideBigData1

Join us on LinkedIn: https://www.linkedin.com/company/insidebigdata/

Join us on Facebook: https://www.facebook.com/insideBIGDATANOW