“This talk will provide an overview of challenges in accelerating Hadoop, Spark and Memcached on modern HPC clusters. An overview of RDMA-based designs for multiple components of Hadoop (HDFS, MapReduce, RPC and HBase), Spark, and Memcached will be presented. Enhanced designs for these components to exploit in-memory technology and parallel file systems (such as Lustre) will be presented. Benefits of these designs on various cluster configurations using the publicly available RDMA-enabled packages from the OSU HiBD project (http://hibd.cse.ohio-state.edu) will be shown.”
“The Hadoop framework has become the most popular open-source solution for Big Data processing. Traditionally, Hadoop communication calls are implemented over sockets and do not deliver best performance on modern clusters with high-performance interconnects. This talk will examine opportunities and challenges in optimizing performance of Hadoop with Remote DMA (RDMA) support, as available with InfiniBand, RoCE (RDMA over Converged Enhanced Ethernet) and other modern interconnects.”
Today Mellanox Technologies announced that Yahoo! Japan has deployed Mellanox’s end-to-end FDR 56Gb/s InfiniBand solutions for database acceleration at several data center sites in Japan.
“With a novel data analytics approach, Mellanox is providing double performance to data Extract, Transform, Load (ETL) solutions based on Hadoop Map Reduce. The code to enable such performance gains is part of the Hadoop community code. With acceleration of the networking stack, Mellanox is quadrupling the number of clients served on a single Memcached server, a dominating key-value caching service for large scale web applications.”
Today Mellanox announced a collaboration with IBM to deliver a high-performance infrastructure for NoSQL Databases and In-Memory Data Grids.
“This is the first truly disruptive advancement in high-end server technology in decades, with radical technology changes and the full support of an open server ecosystem that will seamlessly lead our clients into this world of massive data volumes and complexity,” said Tom Rosamilia, Senior Vice President, IBM Systems and Technology Group. “There no longer is a one-size-fits-all approach to scale out a data center. With our membership in the OpenPOWER Foundation, IBM’s POWER8 processor will become a catalyst for emerging applications and an open innovation platform.”
In this video from the HPC Advisory Council European Conference 2013, Eyal Gutkind from Mellanox presents: RDMA Accelerated Big Data Applications. Download the slides (PDF) and check out more videos at the HPC Advisory Council European Conference Video Gallery.
In this video from the 2013 Open Fabrics Developer Workshop, Sreev Doddabalapur from Mellanox presents: Accelerating Big Data over RDMA. You can check out more OFA videos at our Open Fabrics Workshop Video Gallery.
In this video, Eyal Gutkind from Mellanox presents: Hadoop Acceleration with RDMA. The presentation was recorded at the HPC Advisory Council Stanford Conference 2013. Download the slides (PDF).
Today Mellanox announced that Teradata has chosen its InfiniBand interconnect solution to accelerate the Teradata Aster Big Analytics Appliance. Designed for demanding analytics which require high computational power and the fastest data movement, the Teradata Aster Big Analytics Appliance offers up to 19 times better data throughput and performs analytics up to 35 times faster […]