In this special guest feature, Wayne Applebaum, Vice President of Analytics & Data Science at Avalon Consulting, LLC, observes that as an ever increasing variety of data gets captured and analyzed, we face challenging issues of data governance in general and metadata governance in particular. He is responsible for delivery of Avalon’s Analytics Services and developed the company’s Analytics Framework™. Wayne has more than 30 years experience in data analytics and enterprise consulting. Prior to joining Avalon, he was a Principal Business Architect at SAP responsible for designing and leading the implementation of analytic solutions — a role he also performed at Oracle before joining SAP. Prior to Oracle, Wayne was Director of Business Development for Business Objects where he managed the global growth of the company’s Extranet product line. He spent 17 years at EDS (now HP Enterprise Services) in a variety of roles including Director of Quality Measurement and Practice Director for Employee Disability Services. He holds an MA and Ph.D. in Statistics from the University of Pittsburgh; he earned a Bachelor of Science in Psychology from The City College of New York.
As an ever increasing variety of data gets captured and analyzed, we face challenging issues of data governance in general and metadata governance in particular. Data governance has always been a challenge. Organizations have struggled for decades to get well-defined customer or product master data. But now, in the world of data increasing in velocity, variety and volume, they are also faced with maintaining increasingly complex metadata and data lineage caused by not only the variety of ways a piece of information can be used but the increasing importance of relationships among various data elements. This is particularly true in the area of networked devices, the Internet of Things (IoT).
Historically, data governance has been centered on two types of data: Transactional and Master. Implementers of ERP systems have rightly focused on these two types of data. But, the game changes when instead of just wanting to process transactions (such as orders, payments, payroll) we want to consolidate or analyze transactional information to make better business decisions.
Also, transactions aren’t only coming from ERP systems; in the Big Data world it is also a tweet, a post, a click or a sensor reading. The key is consolidating this disparate information so we are able to make a decision. This “Decision Data” results from the combination of information. It could be as simple as finding the mean and variance of orders of a given product or as complex as creating a scoring algorithm for the litigation potential of an email sent to customer service. In short, any time we combine data to create new information we are creating Decision Data and metadata. This metadata needs to be documented and governed.
It is also important to note that in many cases we are dealing with third party data. While the metadata around third party data can be known and trusted, the consumer can’t govern it.
Decision Data has been largely ignored in the field of data governance until fairly recently. The calculations and algorithms often go undocumented. As the number of big data and analytics projects increases, so does the amount of decision data, thus accelerating the problem.
The key point is, if the definition is not well documented or governed and really just captured in the algorithm, then problems will arise. Multiple organizations may duplicate efforts and produce their own definitions, making cross organizational comparisons difficult if not impossible.
Let’s turn our attention to the Internet of Things (IoT), where we have a network of devices of different types producing streams of data. Not only must this data be decoded and transformed, but its interpretation may be dependent to other devices in the same network. Devices of different types, characteristics, and configurations need to be able to be added, which requires a flexible and extensible model.
We need to consider data governance in terms of some new conditions that have resulted from the Big Data movement:
- Data now comes in a wider variety of different forms
- The form of the data and what we intend to do with it determines how we process it
Because of the complex and changing relationships between data elements, the solution for these problems are likely to lie outside the relational world in the area of semantic or graph data technologies.
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind