Sign up for our newsletter and get the latest big data news and analysis.

Headed for the Cloud? Watch Out: Legacy Data Integration Can Bring You Down

Most IT organizations today are under pressure to increase their “cloudification” and speed up every data-driven aspect of their business, but the challenges of integrating disparate data, applications, and APIs are causing a slowdown that threatens to ground some companies even while others seize the opportunities of the cloud.

If your organization is facing a data integration challenge, you don’t have time to craft point-to-point integrations. They’re hard to build, and they’ll only slow you down when—not if—they break. Sooner or later you’ll have to make changes—big or small—to one or more of the “points,” and chances are your infrastructure won’t be able to handle that. For example, you may need to add a new column to a table or a field to an API, which will bring your integration to a halt if it doesn’t recognize the new field. Or your old integration software—the one you haven’t upgraded —will break, and since it’s now unsupported, you’ll be out of luck. Maybe you’ll need to transition your CRM to the cloud and are unable to figure out the least disruptive way of doing it. Or perhaps you’ll see an opportunity that you can only seize if you can stream your existing data into a new cloud-based or on-premises data lake environment to support your new enterprise analytics strategy.

For any and all of these reasons, IT organizations are looking to modern integration platform as a service (iPaaS) to connect data, applications, and APIs more rapidly and effectively. Rather than having to create and maintain hundreds of individual connections, organizations want each integration to be part of a service layer, so it can be reused for dozens of similar applications. Additionally, they want a partner with integration expertise to deal with the headache of keeping all integrations, old and new, working well in a constantly changing environment.

Simplicity, agility, and cost-effectiveness are the top three reasons why organizations choose to go with iPaaS. Your checklist almost certainly includes other features too, such as an easier-to-use user experience, metadata-driven integrations, pre-built connectivity without coding, data transformation and other ETL operations, as well as support for hybrid deployments. With iPaaS now gaining traction in the enterprise, here are a few ways to begin your modern cloud and big data integration journey.

Strategize before you subscribe

Making the right choice means more than checking off features. Find a forward-thinking consulting partner with a great track record and satisfied clients to help you assess where you are, where you want to go, and how quickly you can get there. Not every system or data asset is going to benefit equally from being put in the cloud; some might need to stay in your data center. A good consulting firm should be able to create an iPaaS roadmap without being tied to legacy vendors, prioritizing projects and setting timelines that reflect your needs and your budgets. Then, when you’ve mapped out your plan, choose an iPaaS provider that offers more than the basics.

Demand change management

Your organization probably wants its systems to adapt to new business requirements right away. That’s the ideal of the “agile enterprise,” but traditional integration technologies will struggle to support this as last-generation ETL and EAI tools are strongly typed and brittle. Make sure whatever iPaaS solution you select is resilient enough to handle updates and variations. To ensure an adaptable integration infrastructure, look for “loose coupling” and a JSON-centric approach that doesn’t require rigid dependency on pre-defined schema.

Think “platform” not “point”

To support cloudification, your integration layer must seamlessly transition from connecting on-premises to cloud systems (and vice versa), while still ensuring business continuity. Make sure you’re not getting a “cloud washed” solution that’s just hosted or provided as a multi-tenant cloud monitoring service. Additionally, get a little suspicious if a vendor says they’d require on-premises ETL or ESB technologies or eclipse-based developer tools: they may be trying to sell you a legacy “agent” running behind the firewall. Look for elastic scale and the ability to handle big (and small) data volume, variety, and velocity. Also, ensure that your iPaaS solution runs as close to the data as necessary, regardless of where data reside.

Get out of the integration upgrade business

If you’re still running old versions of integration software because of the fear of upgrades, that’s understandable. Anyone who’s had to upgrade integration software knows why you’d want to avoid it. Your iPaaS vendor should shield you from any upgrade issues, and should also support self-service instead of offering developer-centric desktop IDEs. Look for vendors who offer browser-based designers for building integrations and automatic access to the latest functionality.

Get ready for the future

iPaaS is important now—and essential in any plan for the future. For one thing, legacy integration technologies will soon have to be replaced. For another, what with the requirements of social, mobile, analytics, cloud, and Internet of Things (SMACT) data, you’ll need to expand and contract compute capacity to process variable workloads. The ability to handle big data cost-effectively will be key in unlocking its benefits. A hybrid cloud integration platform helps by moving data in a lightweight format and adding minimal overhead, using JSON instead of XML. Finally, REST-based streaming APIs are essential as they continuously feed into an analytics infrastructure, whether it’s Hadoop or a cloud-based or traditional data warehouse environment.

iPaaS: the future is now

Don’t let legacy integrations be a barrier to “cloudification” and leveraging big data technologies designed to accelerate your modern analytics initiatives. The best cloud apps can fuel your growth, so get on that cloud—and stay on it—with infrastructure as a service (IaaS) and iPaaS empowering you to build up and change systems as quickly as you want. The right approach to iPaaS re-designs, re-imagines, and dramatically disrupts traditional data and application integration technologies, so that your business can integrate cutting-edge capabilities with its existing assets to transform its operations. With the right iPaaS solution, you focus on your area of expertise, and let your vendor’s integration platform bring you to the benefits of the cloud and big data leverage faster.

10 Modern Integration Requirements

  1. App Integration via REST and SOAP
  2. Large volumes move to data lake or cloud data warehouse
  3. From batch to continuous streams
  4. Event-based, not clock driven
  5. Document centric (JSON)
  6. Hybrid (cloud to cloud and ground)
  7. Accessible via APIs
  8. Broad connectivity (beyond relational)
  9. Elastic scale
  10. Delivered as a service (iPaaS)

Darren CunninghamContributed by: Darren Cunningham is the Vice President of Marketing at SnapLogic, the industry’s first unified data and application integration platform as a service (iPaaS). He has over 15 years of enterprise integration and analytics industry experience, having run product marketing and product management teams at Business Objects, Salesforce, LucidEra and Informatica. 


Sign up for the free insideBIGDATA newsletter.



  1. Eugene Breger says:

    Legacy data and associated data models will remain the Achilles heel in all our “cloud in the sky” imaginations.
    Darren Cunningham is right to start off his essay acknowledging this challenge.
    iPaaS is not the complete solution to achieving “cloudification” if data remains mired in legacy tar pits.

  2. Thanks for the comment Eugene. It’s important to note that the right modern integration solution should “respect data gravity” and run the execution in the cloud or on premises. It should therefore be able to connect to legacy on-prem packaged and custom apps as well as databases, files and disparate sources, whether they are behind your firewall or in the cloud. Here’s a post on the modern requirements that you might find interesting:

Leave a Comment


Resource Links: