Singapore Startup Develops Ultra-Fast Big Data Unification Technology by Avoiding MapReduce

Print Friendly, PDF & Email

percipient_logoSingapore-based startup, Percipient, has developed a way to sidestep a common Hadoop big data function, thereby shortening query processing time by more than 15 times.

Hadoop is an open source software framework that has enabled the world to embrace big data. Percipient’s flagship product, UniConnect, unifies large volumes and varieties of data at high speed by leveraging on specific Hadoop technology.

However, UniConnect replaces one of Hadoop’s core components, the MapReduce model, with a revolutionary function called SkipMR. By doing so, UniConnect is able to significantly boost its data processing speed. Tests in Percipient’s lab on a 16 GB Apple Macintoch desktop showed that using MapReduce, a single query of 1 million rows required 16 seconds, compared to under a second using UniConnect.

UniConnect has now been trialed at several large banks and the results have consistently outperformed existing processes. With big data technology and analytics now a firm fixture in many organizations, Percipient’s CEO, Navin Suri, points to the impact that this increased speed can bring.

Shaving off a few hours, minutes or seconds to an existing process can make all the difference when preventing potential defaults, enhancing customer experiences, or conducting mobile marketing campaigns,” said Percipient’s CEO, Navin Suri. “Traditional batch processes are far too slow, and MapReduce only comes half way in being able to solve this problem.”

The Hadoop framework is used by enterprises to store and process data using parallel processing and clustered commodity hardware. Compared to traditional servers, this framework enables enterprises to handle much larger amounts of data, including unstructured data.

Within the Hadoop framework, a MapReduce job is comprised of two functions. The Map function processes independent chunks of data in a parallel manner, which are then collected together again by the Reduce function. This allows the MapReduce model to accurately process high volumes of data. However, the speed at which the data is processed is, in some cases, the same as traditional, non-MapReduce models.

As such, the Hadoop MapReduce framework may not satisfy enterprises seeking to accomplish both high volume and high speed processing. With UniConnect, Percipient is able to help enterprises comfortably bridge this gap.

 

Sign up for the free insideBIGDATA newsletter.

Speak Your Mind

*