Sign up for our newsletter and get the latest big data news and analysis.

Pepperdata Enables Opower to Rely on Hadoop for Real-time Big Data Analytics

Pepperdata-logoPepperdata, a leader in Apache Hadoop performance optimization, has announced that Opower, a leader in cloud-based software for the utility industry, has deployed Pepperdata Supervisor, a real-time cluster optimizer, to ensure its production Hadoop environment can scale to hundreds of billions of data points reliably, efficiently and on time.

Opower relies on its big data platform powered by Hadoop to help utilities better inform customers of their energy usage and transform utilities into energy advisors,” said Eric Chang, Infrastructure Technology Lead at Opower. “Pepperdata ensures that our production Hadoop environment remains reliable enough to handle all our diverse workloads so we can meet our SLAs with confidence.”

Opower’s multi-tenant Hadoop environment currently supports more than 95 utility partners and their customers, as well as its internal data scientists and other users. With the daily ingest of hundreds of millions of meter reads, as well as third-party data feeds and event data, the company found it increasingly challenging to rely on Hadoop to deliver consistent service for its growing user base with differing jobs, priorities and data sets. Chang and his team needed to ensure the reliability of their system without implementing policies like gating users or breaking up clusters.

Pepperdata has allowed Opower to address these and other key challenges that organizations face when they scale Hadoop in multi-tenant production environments. Interactive dashboards give Opower unprecedented visibility into their clusters, thanks to Pepperdata’s unique software that monitors and controls detailed hardware usage in real time, allowing administrators to quickly isolate the source of performance bottlenecks and take remedial action.

Opower is also using Pepperdata Supervisor to dynamically adjust job resources to reflect their service level priorities, ensuring that their clusters devote sufficient resources to appropriate jobs. As a result, jobs now run faster, more reliably, and more efficiently on Opower’s existing servers, allowing the infrastructure team to scale services with fewer hardware resources.

We have a mix of carefully curated MapReduce jobs that have gone through a development and testing cycle that are running side-by-side with Hive queries and HBase on the same production cluster. We needed to be confident that we could reliably run multiple workloads in a single Hadoop cluster, and that’s why we find Pepperdata appealing,” Chang notes.

Opower is in the planning stages of deploying Hadoop 2, known as YARN, which will allow the company to run a much broader range of applications on top of their Hadoop clusters.

Hadoop is already the standard for large-scale data processing, and YARN will enable companies like Opower to run even more diverse workloads,” said Sean Suchter, founder and CEO of Pepperdata. “At the same time, these new users and new kinds of data processing will cause increased contention on multi-tenant clusters. The peace of mind that Pepperdata provides will be especially critical in this new world of YARN.”

 

Sign up for the free insideBIGDATA newsletter.

Leave a Comment

*

Resource Links: