The basics of High-Performance Computing (HPC) are fairly well known by those acquainted with even a modest amount of computer science. Instead of relying on a single machine for computations, HPC approaches involve multiple machines, which produce two notable effects that directly benefit analytics.
First, the performance is much more prodigious, swifter, and equitable to the time frames for solving business problems with data today, even on fleeting analysis of real-time datasets. Moreover, because multiple machines are involved, the limitations around memory, compute, and storage that can slow just preparing data for analytics (to say nothing of running analytics) with traditional computing methods are all but eradicated.
Still, the particulars of how HPC clusters are able to function in unison for a single job, and just what the longstanding repercussions for Artificial Intelligence, analytics, queries, and data mining are, are typically enigmatic to business users.
According to Katana Graph Principal Engineer Bo Wu, the architectural foundation for HPC methods involving a scalable knowledge graph engine includes “a partitioner, a runtime [scheduler], and a communication engine.” These three elements work together to not only make HPC a reality, but also to master computations so that AI or any other computational load is facile.
The Partitioner
A partitioner’s role in a scalable graph engine employing HPC architecture is to shard the workloads—in this case the actual graph—onto different machines. Sharding is a technique for distributing different pieces (or shards) of a computational process onto various computers to reap the performance gains HPC supports. Sharding is crucial for dividing up datasets that will otherwise slow the activities on a single machine. “If you want to compute a large graph on a number of machines, you need to partition your graph into different machines so each machine gets a small portion of the overall graph, because the overall graph is too big to fit into the memory of a single machine,” noted Keshav Pingali, Katana Graph CEO.
For example, if organizations have assembled a large graph to detect instances of fraud in real time, partitioning that graph onto what can include potentially hundreds of machines drastically increases the performance of this application. Without HPC methods for this and other use cases, firms are stuck with enormous upfront costs for purchasing machines that have tremendous storage capacities. “They’re restricted to a single machine, so…users have to buy these enormously expensive machines with two and three terabytes, and even that’s not enough,” Pingali said.
The Runtime Scheduler
The runtime capabilities Wu described are responsible for orchestrating the endeavors of the different machines so they uniformly work on the same job. For a scalable knowledge graph query engine, that work might include any aspect of graph AI from graph neural networks to clustering, as well as several algorithms for mining, analytics, and queries. Once a dataset (like a graph) has been partitioned, “in order to compute on each machine on the graph that has been loaded into it, there is a runtime system and a set of data structures that have been optimized for the use of graph computing,” Pingali explained.
The runtime system is responsible for orchestrating various facets of the computations, which might pertain to embedding techniques, for example, on a medical knowledge graph to determine which specific compounds can spur the development of a new drug. “The runtime schedules operations on the local machines and can [optimize] all the computation resources to maximize performance,” Wu indicated.
The Communication Engine
The communication engine plays a critical role in HPC methods by enabling different machines to express to one another what parts of a single job they are doing, have done, or require another machine to finish. To query large graphs to determine supply chain optimization and routing, for example, the individual machines in a HPC cluster must relay to one another “this is how much of the sub-graph I’ve matched so far, or the rest of this sub-graph is on another host, or I need to communicate and match what I have so far to the other host and let that other host pick up where the previous one left off to match the pattern graph,” Katana Graph CTO Chris Rossbach remarked.
The most credible HPC undertakings rely on Message Passing Interface (MPI) standards for communicating. “These support idiomatic communication patterns that most HPC applications need to use but, because they’re implemented by experts at the library level and optimized in different library implementations, they can be super fast,” Rossbach mentioned.
Unbeatable Analytics
The aforesaid partitioner, runtime capabilities, and communication engine are integral elements for using HPC architecture for rapid computations on a scalable graph engine. In turn, HPC is the blueprint for the fastest analytics available to the enterprise, whether that involves AI or not. This architecture is necessary for the most performant computations—at enterprise scale—that organizations can get to accelerate their data-driven undertakings, as well as increase their respective bottom lines.
About the Author
Jelani Harper is an editorial consultant servicing the information technology market. He specializes in data-driven applications focused on semantic technologies, data governance and analytics.
Sign up for the free insideBIGDATA newsletter.
Join us on Twitter: @InsideBigData1 – https://twitter.com/InsideBigData1
Speak Your Mind