Sign up for our newsletter and get the latest big data news and analysis.

Intel Steps up with Enterprise Edition for Lustre Software

In this video from the 2014 Lustre Administrators and Developers Conference, Brent Gorda from Intel describes how the company is adding enterprise features to the Lustre File System.

Previously used only by the world’s most advanced research centers, academic institutions and laboratories, enterprises of all kinds are investing in high performance computing (HPC) to solve today’s most important problems and create powerful competitive advantages. Very fast, highly scalable storage software is a critical component of HPC. Data- and compute-intensive applications require storage solutions that combine extreme performance at large scale with simple but powerful management tools to create HPC solutions that are perfectly balanced and maximize resource utilization. The latest Lustre*-based solutions from Intel bring significant performance and capability enhancements to HPC clusters and enterprise organizations—from cloud to Big Data.

Full Transcript:

insideBIGDATA:  I wanted to ask you about what’s going on with Intel as far as Lustre and bringing it to the Enterprise? What are you guys up to there?

Brent Gorda: So, a year ago we announced the Intel Enterprise edition for Lustre, which is Intel’s release with a whole bunch of value add. So for example, we’ve got a Hadoop coupler, we’ve got the Hadoop integrated with [slurm?] and we’re working on other schedulers there. The real piece that’s adding significant value is the Intel Manager for Lustre. And get this: Lustre is going from a research project where you need a propeller on your hat to be able to install it, to a one-button install on your favorite hardware.

So the Intel Manager for Lustre is the only cross-platform management tool deeply integrated with Lustre. It’s built by the guys that bring you Lustre and deeply integrated below with the hardware so that you can do the one-button install. So you can do the debugging, so you can see what’s going on in your file system. So that really started about a year ago it’s maturing now, looking really, really good. At SCU, we’ll have a demo you’ll be able to see the new interface, really sexy. We’ve got some great UI guys that are making that stuff look crisp, it’s really great.

insideBIGDATA: Okay, so if I was a business guy– I mean, Lustre’s created for, you know, super computers, national labs. Why should I care about what you guys are up to with this esoteric, open-source thing?

Brent Gorda: Well, you should care because you have a big data problem and you’re trying to move data to compute. HPC’s always been about balanced architecture and more and more we’re finding people are realizing that the bottle neck is to the actual disk storage. So, let me give you an example, in the past month we wired up a customer that was an NFS customer, and they turned around and gave us huge glowing reviews that they’re now running 100 times faster, just by swapping out the old NFS file system.

Now many businesses out there, they hit the top of a NFS file system and they go buy second one. Now you’ve got half your files on one system, half on the other. You need to remember where they are, and you can’t combine the performance of the two. You peek out at maybe a gigabyte a second, where Lustre can go all the way up to a terabyte a second in the most extreme cases.

You said earlier, “Lustre’s the world’s fastest file system.” Much like the world’s fastest car, you only get there if you’re efficient across the board – which means you scale down as well as scaling up. So simplify it, if you’re a business using a network filer, and you’re thinking about adding a second one, you should think about looking at Lustre, because we can give the aggregate storage capacity and the aggregate performance of that hardware.

insideBIGDATA: Okay. So the story there is about performance, but what are you guys doing to make it more amenable to like business requirements?

Brent Gorda: Well, we’re adding a whole bunch of features there, as well – enterprise class features. The first one that came out – almost a year ago now – is the HSM feature that the CEA originally authored. We put in a huge amount of effort with them to get that up and running and coming out in the open-tree first, and then in Intel Lustre with what we’re releasing now. That will be deeply integrated with the Intel manager for Lustre, so you set all of your policies, graphically and all that sort of thing. It’s just the first of many features that we’re working on and bringing out with our release.

insideBIGDATA: Well, you’ve been in this space for a long time working with Lustre and high-performance computing. Is it exciting for you to see these worlds coming together?

Brent Gorda: Yeah. It’s incredibly exciting. You go into a big data type environment and you tell people, “Well the technology we work on can move data at a terabyte a second,” And it blows their hat off their head. Right? They’re thinking a gigabyte a second or 10 gig a second. The HPC of technologies are so incredibly powerful and so efficient that it’s really turning heads in the business world.

See more talks from the LAD’14 Conference * Sign up for our insideBIGDATA Newsletter

Leave a Comment

*

Resource Links: