Finance experts instinctively understand that data have value. It’s the lifeblood of their industry after all. But some in the financial domain may not grasp all that data have to offer. Data science and big data have led to an expansion of the number of types of data sets, often referred to as Variety (one of the 3 V’s of big data along with Volume and Velocity) and the associated influx of information could very well shape investment strategies and create new businesses.
Big data applies to the finance industry in two flavors. One is analyzing things like investments, econometrics, trading activity, and longer-term data analysis. That’s clearly part and parcel of the finance business, and people in the space already have great familiarity with this side of data. The second flavor is the integrated approach to data in all facets of how organizations do business. This involves understanding customers, understanding competitors, understanding behavior, taking advantage of the world of sensors, and using a computational and quantitative mindset to make sense of a very complex world.
Everyone is struggling with the semantics, so finance isn’t worse off than others. They’re actually making an effort to understand the nuances. Adding to the semantic confusion, the terms “data science” and “big data” are sometimes co-opted by organizations trying to show how they embody these attributes. That’s fine, but the finance ecosystem has a responsibility to learn as much as it can about these areas. The best way to do that is directly from the data science practitioners: see the tools data scientists use and how they approach their work. That firsthand experience will help finance experts inform their investment strategies and see where the data space is heading.
Many professionals who’ve been in the finance field for many years may rightly want to know what relationship is present between data science and “business intelligence?” Traditionally, the front-end access to the data warehouse was known as “business intelligence” in the ’90s. These early data warehouses were mostly constructed out of quantitative data from operational systems — things like order entry and customer service systems. “Business intelligence” tools were used to access the mostly well-understood operational data in the data warehouses.
What’s changed is the explosion of data types. For example, no one was doing analysis on social media back in the ’90s because the technology didn’t exist yet. Now, we need new tools to help accommodate what analysts need to do these days: data analysis, natural language processing, machine learning, visualization, etc. Moreover, the old business intelligence tools were based on operational things, like how many orders a customer placed. They weren’t built to tackle these new tasks.
Will data science and big data incrementally improve existing techniques in the financial industry with new tools? Many believe this will happen and may even create new companies and industries. The analogy might be to when open source software first became widely used. While there were open source business models and companies, the real growth of open source came from companies like Google, Yahoo and Amazon that based their core technologies on the open source stacks. There was this two-headed approach that came out of the adoption of open source. The same could happen in finance with the adoption of big data.