Sign up for our newsletter and get the latest big data news and analysis.

Lustre Scalability, Affordability, Manageability

This fifth article is an editorial series that explores Lustre solutions in the cloud for an exploding commercial data universe. This week’s looks at Lustre scalability, affordability and manageability.  You can download the entire Lustre in the Cloud series in PDF from the insideBIGDATA White Paper Library courtesy of Intel.

Lustre, the predominant high-performing file system installed in most of the supercomputer installations around the world, could be deployed to business customers in a hardened, tested, easy to manage and fully supported distribution in the cloud. Proven to scale up to extreme levels of storage performance and capacity as measured in tens or even hundreds of petabytes, shared and accessible to tens of thousands of clients, Lustre offers high throughput with high availability using vendor-neutral for server, storage and interconnect hardware coupled with various distributions of Linux. Here are some feature of Lustre that go beyond performance.

Big Data Business Solution CloudScalability

The unique architecture of the Lustre file system has been proven to scale easily to petabytes of data that are immediately accessible to thousands of clients. The inherent scalability of Lustre aggregates storage capacity across many servers. I/O bandwidth also scales as more storage servers are added, and can be dynamically adjusted as needs change and demands for more storage capacity and bandwidth grow.

On the other hand, Lustre can also scale down efficiently to still provide high-performance parallel storage for smaller enterprise configurations.

Affordability

Because this software is a distribution of open source Lustre on AWS, it enables system administrators to customize their storage infrastructure and costs as they see fit and according to immediate demands.

Offered through the AWS Marketplace (aws.amazon.com/marketplace/pp/B00GBFUO9A), it provides both the open source Lustre software as well as powerful administrative tools, and full Intel support.  An array of Amazon Machine Images (AMI) are offered for use with instances of elastic block store (EBS) that can be adjusted and configured as needed.

The AWS pay-per-use model allows business developers the capability of testing and tuning their applications in a Lustre environment without investing in new infrastructure.

Manageability

Software developed in the labs and academia often have a reputation of being difficult to install and manage. And early versions of Lustre, while considered potentially useful for business applications, were considered too difficult for an enterprise to manage.

The big advantage in this release of Lustre from Intel is that users can install, provision, scale-up, and get applications running with just an account on AWS and a few mouse clicks.

Intel also provides open source tools for monitoring file system health and key performance indicators  in real time for tuning system performance over the entire virtual cluster on AWS.

Next week we’ll explore ‘High Performance Data Analytics in the Cloud.’ If you prefer you can download the complete series in a PDF from the insideBIGDATA White Paper Library courtesy of Intel.

Leave a Comment

*

Resource Links: