How Automation Reveals the Signal in the Noise

To keep their networks safe, organizations often adopt a “more is better” cybersecurity posture: collect every bit of data and then figure out what to do with it. That’s why security teams review fewer than one in a million of the logs collected in their SIEMs or data lakes. They simply don’t have the time to look at it all.

What’s more, most of those logs have no “signal.” That is, they don’t indicate whether they represent malicious or normal activity. A signal indicates that malicious activity – rather than just random noise – is taking place. When SOC teams use best practices in decision automation to sift the signal out of the noise, they collect less data, save a lot of money and only look at what matters.

Sorting through the massive amount of noise

At the most basic level, even the smallest security companies and programs aggregate logs and security sensor data. While security teams may use logs after an incident to forensically reassemble what happened, the primary use of log data is to detect when an attack is happening and respond rapidly and appropriately to prevent real damage.

Security teams collect this data to inspect it for malicious activity, but there’s so much data that it needs to be screened in an algorithmic way. SOCs typically use simple rules based on Boolean logic that identifies known suspicious activity. These rules reduce the volume of events down to a level that humans can realistically monitor. These algorithms seek the “signal” – some kind of indicator that can be used to derive specific information about malicious actors and malicious code. 

Evaluating the different types of signals  

1. Signatures: This type of signal matches a string contained in a network packet or seen in a file on a device. Longer signatures are more specific; shorter ones more general and, thus, prone to false positives. Telling the difference between what’s truly malicious and what’s just a nuisance is a challenge, especially at scale.

2. Heuristics: Heuristics are how you solve problems; in the SOC, it’s the process you take to determine whether a pattern of behavior indicates malicious activity. If it looks like a hacker, and it acts like a hacker, it’s likely just your admins playing with hacker tools. Until it’s actually a hacker. Heuristics do basic pattern matching and, again, it takes lots of additional information to separate the good from the bad. These are also subject to change over-time making them a challenge to maintain relevance.

3. Hashes (malware and variants, known good files): Hashes are uniquely useful, as they can be both “known good” and “known bad.” Known good is often far more useful, as variants are constantly emerging for known malware, which changes the hash.

4. Behaviors (system, telemetry, human): Human nature is your friend when it comes to behaviors. Detection avoidance behaviors are very common as humans seek to hide actions they are taking that are malicious or suspicious. Also, systems and sensors have normal behaviors and exception behaviors. Again, you have to separate the exceptions, and they can often occur at a similar frequency making this difficult.

5. Activity logs: These are typically benign records of system and user activity. It takes a ton of work to recognize patterns in them that indicate suspicious or malicious activity is happening. Most logs are purpose-built for system troubleshooting and not for security detection. It is useful to simulate common and modern attacks to build predictive log profiles, and again this has to be maintained as hacker tactics, techniques, and procedures evolve. 

6. Indicators of compromise (IP-based, domain): Only one percent of the time is IP-based intelligence ever valid. The other 99 percent of the time, it’s not present. IP-based intelligence isn’t worth paying attention to. It barely even qualifies as a signal. Domain reputation is a reasonable contextual signal in the presence of other alerts or indicators.

7. Timing and patterns: There are specific time windows and patterns in logging that can indicate suspicious activity. These, like activity logs, are metadata detections that identify use outside of normal hours and other abnormal patterns that should be investigated. Metadata is critical as you interpret things like recon differently when they are lateral or low and slow.

8. Anomalies (Clusters and outliers, failures):  The sheer volume of anomalies makes it difficult to use them for detection. They do have some security benefits but won’t help with detecting bad guys. The way we connect and administer the modern enterprise and cloud systems together causes a ton of anomalies. 

By using specific data where you understand its detection purpose, you reduce the size of the haystack so you can sift through it faster. It's more efficient and more effective – like fishing in an ankle-deep stream as opposed to in a lake.

Decision automation and signal analysis

This is where decision automation comes in. By leveraging automation, SOC teams can process massive amounts of data – sifting the wheat from the chaff – letting evidence be evaluated in the right context for making an appropriate judgment call.

These best practices for automating the signal identification process will help you get started on your own journey.

When looking for malicious activity, it’s key to only collect what matters. Identify the information that reduces your uncertainty and pull it into a decision automation solution. Every piece of relevant evidence you gather further reduces your uncertainty, and the weight of evidence makes a rational decision. This allows teams to recognize potential incidents, investigate them, and then learn and reinforce which information would have enabled them to detect those incidents in real-time. Don’t collect and automate everything; only what you can use to improve your detection capability.  

Using decision automation to monitor, analyze, case-build, and escalate ensures that junior analysts aren’t being overwhelmed by sifting through mountains of useless data in search of the signal. Decision automation is able to supplement human SOC analysts at machine speed, sorting the signal and contextual information from the surrounding noise to provide a long-tail depth of analysis in seconds.

What matters most

The impulse to analyze every scrap of data is strong, but it’s ultimately counterproductive and expensive. Because logs lack the signal of malicious activity, they take up space and create a never-ending headache for security analysts. This is where decision automation can make a difference. By focusing on the signals that matter, decision automation helps streamline business processes, decrease burnout, and increase security operations efficiency. With decision automation, you can pull the signal from the noise and protect your organization more effectively.