What you need to think about to successfully deploy your Big Data solution
Many companies see big data as a big opportunity. Around 76% of companies plan to maintain or increase their big data investments, according to recent DNV GL research. But it’s not enough to be sold on the idea, you have to successfully implement a big data solution before you can enjoy the benefits and that’s easier said than done.
If you approach the task with a strategy for tackling the major challenges you’re going to face, then you can boost your chance of success significantly. Here are four major challenges that anyone building a big data solution is going to have to overcome.
1. Lack of test data
When you replace your existing system with a new big data solution you want to be sure that it’s going to be able to handle the volume of data. In order to test properly, you need a lot of test data that is virtually indistinguishable from the real thing. You’re going to have to consider how to extract or create this data, keeping privacy and other regulations in mind, and how to feed it into your new system in a way that emulates the real pattern of data flow.
Plan ahead to ensure you have the expertise you need and the structural support to generate high quality test data in large volumes. This data might be very complex, but the closer you can come to emulating real data, the more useful your testing will be in terms of identifying bottlenecks and other issues.
2. Finding the right people or training
There are many different technologies in the marketplace designed to service big data and they all work differently. Because the field is so new, there’s a dearth of talent with the skills you really need. For a smooth implementation you’re going to want people with deep knowledge of your chosen technology. Consider the current talent pool. Do you have the resources and contacts you need to recruit the right people? How long will it take to train people up?
When you search for the right technology don’t just focus what it does or how it works, think about the people you’ll need in order to extract maximum value from it. If you can’t get the exact experience you need, find people with similar skills and train them up. Spread the expertise around and provide the opportunity for people to learn from their colleagues.
3. Technology is not mature
Any new technology with minimal road-testing is going to lack certain features. The big data solution you’re employing hasn’t been as thoroughly tested as something that has been in use for decades. Some of your integration challenges or tool requirements may require some internal development. You’re going to need to figure out some workarounds and you can bet there will be delays. Try to avoid making assumptions ahead of time and build in buffers to help you come to grips with the new technology.
Understand that there will be a learning curve and make sure that you schedule for the possibility of troublesome issues. It’s not always possible to hit the ground running with immature technology, so temper your expectations.
4. Infrastructure and configuration
You need a solid infrastructure in place for your implementation to succeed. Make sure the environment is set up properly for development, testing, and production. Cloud computing is the easiest way to ensure that you can scale quickly and cost effectively. You’ll need DevOps people with the time and the expertise to configure everything and tweak it for optimum performance. Even distribution of data is vital if you want to avoid bottlenecks.
Consider the complexity of your data sources and the structure you need to be able to get the data you require immediately. The system will need to be able to solve complex data at scale and at speed to deliver real-time insights. Poor configuration will have a potentially devastating impact on efficiency.
Failing to plan ahead for these challenges can lead to a frustrating, stuttering start for your new big data solution. Working out a solid strategy makes for a smooth implementation and enables you to realize the potential benefits much more quickly.
Contributed by: Kaushal Amin, Chief Technology Officer for KMS Technology, a software development and testing services firm based in Atlanta, GA and Ho Chi Minh City, Vietnam. Prior to joining KMS he provided leadership to LexisNexis’s background screening technology organization and help to create new products within drug testing, fingerprinting, court records, and international screening markets. Prior to that, he was VP of Engineering at Revenue Technology (acquired by Oracle). Kaushal led development of Revenue Tech’s price optimization and deal management platform for the manufacturing and retail industries. From 2001 to 2003, he was VP of Development at ViryaNet where he led development of a mobile field service platform. From 1989 to 2001 he led engineering organizations at Intel, McKesson, and IBM. Kaushal graduated with a degree in Electrical and Computer Engineering from the University of Michigan.
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind