Sign up for our newsletter and get the latest big data news and analysis.

The Power of Machine Learning in Cybersecurity

sven-krasserIn this special guest feature, Dr. Sven Krasser, Chief Scientist at CrowdStrike, focuses on what machine learning can solve and why its crucial in today’s cybersecurity landscape and why the data input is the most important factor. He currently serves as Chief Scientist for CrowdStrike, where he oversees the development of endhost and cloud-based Big Data technologies. Previously, Dr. Krasser was at McAfee where he led the data analysis and classification efforts for TrustedSource. He is the lead inventor of numerous key patented and patent-pending network and host security technologies and is the author of numerous publications on networking and security technologies.

Machine learning is a buzzword that has picked up steam across several industries, and especially in the cybersecurity space we see more and more companies adding machine learning capabilities as a differentiator into their marketing materials. However, machine learning has been around for decades, and many security companies have been using it under the hood for a while. So what is changing, how much is hype and what shows promise?

To better understand the value of machine learning, it’s important to take a step back at how most security tools work. The traditional workhorses of cybersecurity have been signatures and heuristics. Signatures (also called “Indicators of Compromise” or IoCs) can be as straightforward as a hash value or a byte sequence that a product is looking for. Heuristics are often created by human analysts as a set of rules that, for example, describe malicious traits. They cover the ground for a larger number of signatures and create some resilience against basic modifications an attacker might attempt. The challenges that these traditional methods impose on the security industry are twofold. First, the number of threats and their variations is exploding, and deriving signatures one-by-one for each of them is a losing proposition. Second, modern threats blend into the environment and only subtly differ from legitimate usage patterns. Detecting them requires looking at a larger amount of data.

On both counts, machine learning can help. For example, let’s take a look at malware detection. There are on average more than 10 million new malware files every month, so signature or IoC based approaches are no longer viable, and human-derived heuristics can barely keep up with the amount of new malware families. These approaches commonly rely on data files hundreds of megabytes in size that need to be updated daily. In contrast, machine learning based approaches do not attempt to recognize individual malicious files; instead, they model which traits make a file malicious. In the real world, this is analogous to the difference between learning the right letters to check on a multiple choice test and actually understanding the questions. For comparison, a machine learning model file on the order of 10 megabytes updated every few months can accomplish the same level of detections for known malware as traditional approaches. And because detection is not based on knowledge of individual files, this approach works very well for new and unknown malware.

The majority of advanced threats do not rely on malware. Attackers use exploitation techniques, stolen credentials, and generally “live off the land.” It is very challenging for traditional approaches to spot these types of intrusions — in part because there are so few artifacts left behind that can be spotted using IoCs. Furthermore, attackers quickly move laterally from machine to machine, effectively reducing their cross-section for solutions that can only look at individual endpoints or parts of the network. A remedy to this problem is to not look for artifacts but rather to model intent and objectives by observing events and activities, so called “Indicators of Attack” or IoAs. Machine learning brings two advantages to the table: analyzing data at a larger scale and more breadth than can be achieved by a human analyst.

More scale means being able to pull larger amounts of data into the analysis. Especially for advanced threats, a lot of data is needed before trends emerge and abnormalities can be spotted. Cloud-based solutions have a distinct advantage here as their vantage point allows analyzing larger amounts of data from various systems at the same time. For example, Netflix (think cloud) can give you better movie recommendations than your local Blockbuster clerk (think appliance) because it has vastly more data at its disposal.

Next, machine learning allows working with data at more breadth. On the one hand, this means being able to look at many different attributes concurrently; on the other hand, it means being able to combine data from a gamut of sources. To use the same metaphor: knowing a user’s book preferences could be used to improve their movie recommendations.

In summary, what changed to make machine learning especially appealing against today’s security threats? First, there is vastly more data available, and more data is needed to spot advanced threats. Second, the cloud allows solutions to effectively handle data at sufficiently large scales without the constraints imposed by individual machines on the user’s network. Lastly, computation costs have become much cheaper, making more complex approaches viable now. Naturally, machine learning can be applied at smaller scales as many companies still do; the big payoff, however, lies in Big Data.

Sign up for the free insideBIGDATA newsletter.

 

Leave a Comment

*

Resource Links: