Network Forensics

The I-Team

Successful incident handling begins with a properly trained team (to find educational resources, see "Get Smart" on page 38). Your information security department may consist of a few trained individuals who carry out many roles, sometimes referred to as the "one person, 1,000 hats" infrastructure. Regardless, your infosec-response guidelines must clearly define the responsibilities of each business unit. Everyone from those in the executive suite down plays a part. Here's a breakdown of the roles organizations must fill, and their functions:

• The infosec executive steering committee sets direction for the information security department as a whole. This is where overall business risk is assessed and included in infosec policy development. Charged with making big-picture decisions, the committee evaluates the progress of and adherence to initiatives designed to protect the organization, for example, whether business units are complying with security policies. It usually comprises upper-level managers who are capable of performing cost-benefit analyses and providing direction based on the results--is the extra security offered by tokens, for example, worth the hardware and helpdesk costs? Most security pros would say yes, while those tasked with supporting end users might disagree.

• The information security department maintains all policies and standards, including the overall incident-response process. This team performs routine audits and assessments, including investigation of reported incidents. All new-project development must be evaluated and approved by this department, including changes or additions to both internal and external network presence and infrastructure. For instance, if HR wants to create a database of all employee contact information that's accessible to the entire company, it must obtain approval from the information security department before proceeding.• Corporate security and HR work together any time an investigation is warranted. Security investigates fraudulent activity and policy violations that result in loss, working closely with the infosec department if any of those violations include information assets. Corporate security and HR conduct personnel investigations and maintain physical security policies and processes.

• Network administrators put the policies designed by the infosec department into practice. They keep networks and systems safe and implement new development initiatives; if you decide to install a forensic tool like EnCase Enterprise, you'll need network admins' help to install client servlets on production systems. All admins must be trained as first responders, as they're typically the first ones to be aware of anomalies.

• End users must be trained to report strange or abnormal activity to the information security department and their local network administrators. Users must comply with company security policies, including reporting suspected incidents--easy to say, hard to accomplish. Our best advice is to have the directive come from the top. If you see a user violating a security policy, notify his or her supervisor. It's an uphill battle but one worth fighting.

Keep these roles in mind as you document, assign and manage information security policy development and processes. Of course, each team's level of involvement will vary based on the type and severity of an incident. In general, the infosec team should perform all technology-related investigations, working with those who are affected by the incident to gather information. Every step taken during an investigation should be documented, with the understanding that it may be used as evidence in a legal proceeding.This might be a good place to state a couple of obvious concepts. First, it's extremely important to document only known facts--any unsubstantiated information could invalidate even legitimate claims. Copies of all relevant e-mail and voicemail should be saved to ensure that they're properly entered into any incident report. Second, make sure the incident response plan you develop is reviewed by counsel, and if a court case is even a remote possibility, secure the advice and guidance of your company's legal team.

When developing a policy, bear in mind that incident response consists of three stages: analysis, coordination and handling, and resolution.

During analysis, strive to collect every scrap of evidence related to the incident. Remember that evidence can be obtained from several sources in addition to the primary system under investigation. For instance, if documents have been illegally transferred over the network through e-mail or alternative means, logs of network traffic or connections can be invaluable in identifying the parties involved, as well as the relevant dates and times. Your infosec policy should dictate logging and archiving transactions throughout the network, including but not limited to IDS logs, proxy logs, system firewall logs, and SMTP or mail server logs.

All systems on the network must be considered suspect until their integrity can be verified--that is, guilty until proven innocent. Initial analysis consists of examining every questionable system to determine the scope of the incident. It's imperative that only trusted tools be used in examination to avoid deception by modified system utilities, such as a rootkit. This process should be managed by the information security team, in cooperation with the network administrators responsible for the machines in question.

After your initial investigation has determined the range and severity of the incident, coordination and handling begin. Management must be notified of all known details; from there, the level of escalation will depend on the severity of the breach. If the incident is an acceptable-use policy violation, for instance, only direct management need be involved. If, however, an incident has impacted customer relations or financial data, escalation to the CIO and above may be necessary.The main concern in developing a handling strategy is containment. With a network security incident, this means ensuring that the attacker cannot gain additional access to compromised data. The same is true of some internal policy-violation incidents: The employees under investigation should not be allowed continued access to compromised data. In short, keep the sand in the sandbox to limit damage.

If the identity of the perpetrator is unknown, decide whether to slam the door immediately or continue to allow access for tracking and evidence collection. If prosecution is a possibility, hard evidence is a necessity. If you decide you want to keep access open, management approval is imperative.

I Arrest My Case

Incident handling must include forensic-level response strategies if your evidence is to stand up in court. Every pertinent fact must be documented, and chain-of-custody protocol must be observed. Although many incidents are never reported to law enforcement, in part to avoid damage to corporate reputations, you must approach every investigation as if you'll have to defend your evidence to a jury. This is where the exact documentation and clearly defined roles and responsibilities will save time, money and, potentially, the prosecution itself.

A forensic response entails creating an exact bitstream copy of the hard drives on any affected systems. For obvious reasons, this step must occur prior to running backup/restore operations. Images obtained as forensic evidence must be backed up immediately to optical media for preservation. Remember that even opening a file will change its access time. If a system is suspect, make a copy before touching anything.Once evidence has been collected from all involved systems, the resolution--getting the system running again without chance of reinfection--can begin in tandem with continued analysis of collected evidence. Resolution implies significantly more than just restoring a system to a recent backup. A backup does not eliminate the risk of continued unauthorized access because the vulnerabilities that allowed the initial violation will still be present.

Depending on the type of incident, the vulnerability may be related to current versions of network services, or perhaps incorrectly set privileges on shared directories. The team investigating the collected evidence must be in close communication with the team restoring and securing the violated system.

If the incident involves network penetration, investigators must determine the root cause. Many cases are lost at this point: Evidence is modified or destroyed in a rush to determine the source of the attack. Anyone who has witnessed the frenzy that takes over the IT department the morning after an intrusion can attest to this.

A well-trained team should maintain investigative policy even when tempted to take "just a quick peek" at a suspect system. This policy includes verifying current versions of all available network services and checking log files for evidence of service exploitation. Hashes of current files and services can be compared against known-good versions of those same files and services, as an aid in discovering any rogue programs or processes running on the exploited system. Tripwire's eponymous product is one of the best-known tools; Guidance Software's EnCase Enterprise Edition (see review on page 43) does this automatically as well.

Additional log files collected from network servers, such as firewall or VPN logs, must be examined for source addresses correlating with times of exploitation as recorded by logs and file access times on the exploited system.Once the root cause of the incident has been identified, make sure the system won't be victimized again. This may mean upgrading to a newer patched version of a particular network service, or reviewing and modifying or updating existing firewall software or rule sets.

If the incident involved an internal policy violation, there is less danger that an outside entity could further victimize the system, so restoration is not as vital. It's more important to assess the scope of the incident and the damage. For instance, if an employee has stolen intellectual property, such as R&D data or customer lists, and is using it for personal gain, you must define exactly what's been taken and, if possible, where it's been transferred. Although a bitstream copy of the system should be obtained, systems used in internal crimes should be transported to a secure location for imaging and investigation.

Again, all rules for accumulating forensic evidence must be followed to the letter to ensure evidence integrity in the event of a trial. The information security team must work closely with the HR department to identify and deal with everyone involved. Furthermore, team members must maintain strict confidentiality about an incident. If mention of Joe passing R&D plans to his old employer becomes water-cooler gossip, you've got a problem.

Bag It and Tag It

Chain-of-custody records must track all evidence from the first response through trial. A log of every action taken on suspect systems, who performed the action and when, must be kept to avoid allegations of tampering. Each piece of evidence must be marked with a tag and recorded. We've seen cases dismissed because of inaccurate or incomplete chain-of-custody documentation. This is because of the nature of electronic crime: Unlike the knife in a murder case, electronic evidence can be tainted, destroyed or altered long after the crime has been committed and evidence has been collected.Records must include the full names of all persons involved in transferring evidence, as well as their signatures. Provide location addresses and exact transfer times. All tagged evidence should be described in detail within one document; include as much information as possible. This document can initially be as simple as a Microsoft Word file, but for integrity it should always be kept in hard copy. Once the evidence is tagged and documented, it can be transported to a secure location for further investigation.

Prior to electronic forensic analysis, physically examine the hardware; even a Post-It note on the system could provide clues. In one case, a government employee was suspected of compiling information about operations above his clearance level. Third parties came forward to admit transferring classified documents, but an investigation of his hard drive produced nothing incriminating. As an afterthought, his computer case was opened, and a second, disconnected hard drive was discovered that contained all of the documents attested to and more. Those were enough to fully prosecute the individual.

Call in the Cavalry

It's important to know when to involve a third party. If an incident is serious enough to severely affect public image and customer privacy, or to justify a trial, consider contracting the investigation to a computer forensic specialist. Using a forensic investigator makes it less likely you'll be accused of evidence-tampering, in the same way that an independent accountant can reduce the risk of an audit. Even a well-intentioned member of the IT staff can alter the findings of an investigation in a misguided attempt to prove guilt.

Consider a recent case of wrongful termination: An employee of a Fortune 500 company was fired for using the Internet to view decidedly inappropriate content in violation of company policy. In attempting to investigate, IT employees found "evidence" of a crime that would escalate the prosecution to a federal level, prompting the hiring of an investigator to verify the findings.As it turned out, the "evidence" had actually been generated by the untrained IT workers in their attempted investigation. They had opened Internet history files on the live system to verify contents, which generated new cache files for Internet pop-ups and Web sites that contained the evidence in question. So, in fact, the evidence they were using to prosecute the former employee had been generated by the people tasked to investigate the incident!

When in doubt, bring in a professional. In this instance, improper procedure almost turned into a wrongful federal prosecution.

Back to Basics

Common-sense network-management policies and information security guidelines must be followed consistently. Responding to an incident can be extremely difficult if your day-to-day procedures are all over the map.

For example, restoring an exploited system to a trusted state without data loss can be expensive and time-consuming if regular system backups have not been performed. Log files must be kept throughout the network, as they become invaluable in determining where and how an incident occurred.Many readers responding to our poll say they would have had more success with investigations if log files had been more comprehensive. One respondent says that poor interoperability among different log mechanisms and locations--firewall, DMZ, IDS, servers--made it difficult to piece together a complete picture.

Many companies do not develop information security policies until after an incident. This is like remembering your seat belt after a collision. A business does not operate without systems and services in place to serve its customers. Likewise, it shouldn't operate without the knowledge and tools necessary to protect those systems and services.

Get Back Up For a guide to getting your systems up and running after an incident, including a real-world example of how our University of Wisconsin partner labs recovered from a nasty rootkit attack on 300 Linux systems.

Marisa Mack is a security consultant for Neohapsis, a Chicago-based security consulting firm. Write to her at [email protected].

Create and enforce an information security incident investigation policy and designate a team to respond to breaches before you learn that a former employee took your customer list to a competitor. While only one-third of respondents to our e-poll maintain relationships with law enforcement, all organizations should have the capabilities to investigate internally and produce evidence that will stand up in court.To do this, you need people, processes and investigative tools. In "CSI: Enterprise," we outline how to keep your assets and reputation intact by building a partnership. Everyone from the executive suite down plays a part, including the information security and IT departments, corporate security and human resources, network administrators, and end users. An investigative process generally follows a three-step plan: initial analysis followed by coordination among those involved, a decision on how to proceed and an effort to put affected systems back into production safely.

In "Elementary, My Dear Watson!" on page 43, we discuss a range of investigative tools, from full-featured remote image acquisition to specialized applications that can dig deep into text or mail stores. Rather than award an Editor's Choice, we focused on covering the broad spectrum of forensic offerings. Finally, if you need training on how to put these investigative powerhouses to good use, see "Get Smart".

Training for IT and infosec personnel tasked with forensic security-incident handling can range from obtaining a full four-year degree to attending a one-day seminar on the basics. Many companies benefit from sending some or all of their IT/IS staff to at least one or two seminars or conferences to stay current in the information security and forensic industries. The trick is finding the one that's best for you among the thousands of training courses available.

A good starting point is the SANS (SysAdmin, Audit, Network, Security) Institute, which is well-known and trusted for computer security training and research and offers several levels of certification. SANS also holds annual week-long conferences for security pros.

Very basic CCE (Certified Computer Examiner) certification can benefit those working on the perimeter of an investigation by introducing fundamental forensic concepts, though it's not recommended as a primary source of training for anyone leading an investigation.Many colleges offer degrees in digital forensics for a more in-depth education. Champlain College Online, for example, offers two- and four-year Computer and Digital Forensics degrees, as well as a certificate program. A list of colleges and universities with courses and degree programs in computer forensics is available here. North Carolina Wesleyan College also maintains an exhaustive list of computer forensics links at faculty. ncwc. edu/ toconnor/ 426/ 426links. htm.

Vendors also offer training. Guidance Software's courses are geared toward the EnCase product line, while ASR Data holds training sessions on Linux-based forensic response.