Watch Out for That Log!

4:05 PM -- In two blogs last week, I offered some thoughts on how to collect and aggregate security log data. Now let's step back and ask, "What do I do with all these logs now that I've got them in one place?"

You could ignore them until you have a security incident that requires log analysis to figure out what happened. If you go that route, make sure you archive the logs for many months. You don't want to be tracking the activity of an attacker only to run out of logs because you rotate them every three months.

Ideally, you should take the path of proactive security and monitor the logs for suspicious activity that could indicate an attack -- or worse, a compromise. The problem is that logs can become overwhelming quickly. What events do we really need to keep track of, and which ones are indicative of malicious activity going on within our network?

For the sake of completeness, I personally like to collect everything. The downside of this approach is that I then have to weed out the events that are just the noise of typical system activity, so that I can focus on interesting events that may require further attention.

I gave Microsoft some heat last week for not providing native log forwarding and collection. But Microsoft does provide valuable resources on its security site for identifying events of interests and weeding out the noisy ones. Appendix A of Microsoft's "Security Monitoring and Attack Detection Planning Guide" includes a listing of "unnecessary events" that can be excluded to reduce storage needs and make it easier to review the meaningful events.

Using Microsoft's list as a basis, it may take awhile to figure out what is unnecessary in your environment. Log monitoring is just like intrusion detection systems (IDS): It requires tuning to be valuable. Ever install Snort and enable all the rules? Same thing applies to logging.

To help you figure out what all the events mean, the Digital Forensic Institute has published a Security Event ID Cheat Sheet, and EventID.net has a searchable online database.

Automation and event correlation are the ultimate goals of any log monitoring solution, whether it is developed in house or purchased off the shelf. Your organization may not have the time or capabilities to develop the tools and methodologies to be effective at log monitoring.

If that's the case, check out the commercial solutions from companies like LogLogic and Intersect Alliance. They provide automated alerts when they recognize events that might indicate incoming attacks or hardware failure. They also offer advanced search capabilities.

As Nike says, "Just do it." I've never met anyone who complained about having a well-designed and effective logging system. But I know many who regretted not having a better log management strategy -- usually after their systems were compromised.

— John H. Sawyer is a security geek on the IT Security Team at the University of Florida. He enjoys taking long war walks on the beach and riding pwnies. When he's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading