Cognitive Insights™ Uses Machine Learning to Proactively Surface Issues in Your Data Before it’s Too Late

Print Friendly, PDF & Email, the company that offers ELK (Elasticsearch Logstash and Kibana) as a cloud service, has just unveiled Cognitive Insights™, a new analytics solution that combines the power of machine learning technology with human experience in order to identify issues in IT. The goal of this new solution is to help companies give their customers a better experience by uncovering data issues before they become problems.

The biggest challenge when analyzing machine generated data is extracting important information from the data and distinguishing the critical messages from the rest. With billions of lines of machine generated log data created every day, it’s not realistic for any organization to review each and every line. Currently, the only attempts to develop a method of sifting through such data have been mathematical anomaly detection. However, this solution has many issues and is often ineffective. Cognitive Insights aims to combine the knowledge from data experts around the world with the power of machine learning in what the company refers to as Unified Machine Intelligence (UMI).

UMI uses machine learning to provide accurate search results for companies in much the same way as popular services such as Google search or Amazon recommendations. UMI crawls thousands of data sources and discussion forums, ingesting millions of messages every single day to come up with the most common patterns of interesting log events. Additionally, UMI learns the way users interact with the results provided in order to offer the most relevant information to the unique circumstances.

We generate hundreds of millions of events every day,” said Joe Miller, a system engineer at Pantheon, a fast-growing website management platform in San Francisco. “We only know to look for issues that we know have caused problems for us in the past. Cognitive Insights alerts us also on issues that happened for thousands of companies around the world without us needing to explicitly search for them. With Cognitive Insights, we were already able to discover a production issue within moments of enabling the feature.”

Imagine that a company has the ability to read all the threads, discussions, source codes, version releases, and issues reported on a product like Hadoop or Kafka, and can then extract key patterns to use in its log searches. The company can then apply this to its streaming data in real time in order to identify these important events. The system is created to mimic a team of the best IT professionals reading every log line that is created, matching it against their internal knowledge, and alerting the company as to what needs attention.

After almost two years in the making, I am excited to announce the release of Cognitive Insights — the first step in our overall vision of leveraging AI to help companies run growing, complex IT environments,” said Tomer Levy, Co-Founder and CEO of “We realized early on that no matter how many tools and products a company has, it always comes down to the human experts to know what is really going on. No organization can have expertise in every underlying infrastructure or product that it uses. We created Cognitive Insights to enable organizations to leverage the collective expertise of the best engineers around the world and apply it to their own environment.”


Sign up for the free insideBIGDATA newsletter.

Speak Your Mind