This third article is an editorial series that explores high performance storage solutions in the cloud for an exploding commercial data universe. This week’s lustre 101 article looks at the history of lustre and the typical configuration of this high-performance scalable storage solution for big data applications. You can download the entire series in PDF from the insideBIGDATA White Paper Library courtesy of Intel.
Fifteen years ago, the HPC community began developing a super fast and highly scalable parallel file system to meet the demanding needs imposed by high performance computing. Called Lustre, it is the dominant file system technology now in use at half of the top 100 supercomputer installations worldwide (top500.org).
No longer confined to Big Science, Lustre has become increasingly more attractive for use commercially where sustained high I/O bandwidth is needed to process large streams of data. Lustre has proven it can support more than a terabyte per second I/O throughput, tens of thousands of client systems, and petabytes of storage.
Unlike NFS, where all I/O goes through a single node, Lustre concurrently stripes data across multiple storage servers, with meta-data (file names, owner, permissions) operations dedicated to separate servers. This allows the system to optimize for different workloads and demands, and permit access to files from multiple clients independently. By configuring redundant servers and enabling failover storage, Lustre offers a file system that is highly reliable with no single point of failure.
Lustre itself is open source. The source code is freely available, and is maintained and developed within the HPC community. Two consortia of leading industry, research, and academic organizations, Open Scalable File Systems (OpenSFS.org) and the European Open File System (EOFS.eu), are charged with the ongoing stewardship and coordinated development of Lustre, independent of any single platform or vendor hardware.
Next week we’ll explore commercial versions of lustre. If you prefer you can download the complete guide in a PDF from the insideBIGDATA White Paper Library courtesy of Intel.
Speak Your Mind