Workshop: Recovering From an Attack

Another attack earlier this year did more widespread damage and took longer to repair. About 300 Linux systems were hit by sophisticated and fast-moving attacks that used rootkits. Some 200 of these targeted systems were high-performance cluster machines, and it took the university more than a month to get one of the clusters back online.

We recommend watching traffic at the border so that you can keep abreast of any suspicious activity. Sometimes, other organizations can help pinpoint a problem. A sister university, for example, may alert us to a problem and send snippets of logs from machines under attack.

Recovery Mode

Once you've concluded that something is amiss, you can no longer trust any program installed on the infected machines, nor the operating system kernel. Assume every binary has been replaced and is either the problem itself or is no longer able to discover the source of the problem. Depending on your organization and the nature of the attack, you may want to call the authorities before attempting any reconnaissance or repair. (Don't forget to contact your legal counsel first.)

In the spring Linux incident at the university, a rootkit called SuckIT was surreptitiously installed to gain root access and steal user names and passwords, as well as to set off denial-of-service attacks using the unsuspecting machines. This attack is an example of how intruders have plenty of tools--including rootkits, key loggers and back doors--to choose from once they have access to your machine. It's difficult to determine whether the intruder has removed the software he or she installed or if it's still on the machine, so be suspicious of anything on that computer. With black-hat attacker tools like SuckIT and Wolff, intruders can burrow themselves and their code deep into your machines.Malicious software can be distributed within a single executable file. One little file (less than 100 KB) may not seem like much, but it could contain code capable of opening many back doors to bypass system security. If you eliminate that rogue file, you won't know what other code the intruders may have inserted while the back doors were open. A key logger, which logs all keys typed on the keyboard, could be running in the background, or another back door could be left to execute sporadically or at a future date--all of which are tough to detect. We recommend monitoring any victimized computer for at least two weeks after the attack is discovered and doing a reinstall as soon as possible.

Some spyware programs use "brother" processes that place them in two locations in memory. You can't kill them--even if you kill one, the second process copies itself into a different location with a different name. These programs also drain resources from your machine.

And there's no way to know what was replaced on your computer, either. Even cmd.exe has been replaced on some Windows computers. The best solution: Always do a complete reinstall with a formatted disk. If you're using an x86 environment, remember to format the boot record (fdisk /mbr) because that can be used to reinfect the machine as well.

A complete reinstall requires first pulling the power cord from the back of the machine. An operating system keeps part of its memory on disk in page or swap files, which contain parts of running programs on your computer. Most OSs slowly delete information no longer needed off the hard drive, fully synchronizing only at shutdown. Vital information may be hidden here--for example, tracks that were covered or files that were created and deleted. So interrupting the power keeps this information on disk for you to recover and sift through.Next, use recovery tools to boot the machine off a CD-ROM. Our top picks are Knoppix-STD (security tools distribution) for Linux and FIRE (Forensic and Incident Response Environment) for Windows (biatchux.dmzs.com). These are live CDs that let you manipulate the data on the infected computer with known good tools and a known, uncompromised kernel. We prefer STD for Linux-based intrusions because it has lots of useful forensic tools, but we keep FIRE handy to help with Windows intrusions.

After booting the computer in a recovery environment, create a disk image of the infected computer's hard drive and store it for later examination (see "Use Forensics To Help Crack the Case," below). The standard tool dd is a very good choice--it copies every bit of the hard drive. Tools like PartitionMagic, which copy only the "good" parts of the drive, aren't as helpful.

Removing unauthorized, implanted code in a Windows system is simple, right? Wrong. The danger of getting rid of the code is that it can trigger other malware. Some spyware, for example, comes with an uninstall tool. In our recent recovery efforts, we found this so-called tool to be a wolf in sheep's clothing--it lets the spyware reinstall itself in a different location. We've also had spyware insert itself into the network stack so that when it is removed, the computer won't boot and must be reinstalled.

Make sure you have known, clean copies of tools, such as netstat.exe, cmd.exe and dir that exist on the file system, as well as freeware tools like Process Explorer, TCPView and Streams from Sysinternals. If your goal is to get the system back online quickly and a complete reinstall is not an option, then a quick and dirty fix is the way to go. This is risky, however, and is not a replacement for a reinstall.

Here are some tips for handling a Windows intrusion:• Write down the computer's information, including IP address, hardware address and clock skew.

• Disable the System Restore functionality in Windows XP. It may be hiding another copy of the problem.

• In Windows Explorer, make sure that "Show System Files" and "Show Hidden Files" are both checked. A lot of crafty executables try to cloak themselves as system files.

Using known, good copies of all the utilities, run netstat -ano from a command line or fire up TCPView to see the compromised system's network activity. It may be a denial-of-service or a server-type process, with ports listening that shouldn't be. After finding the problem network traffic, for instance, note the PID (Process ID) associated with the network connection.

Next, fire up Process Explorer. Find the PID associated with the bad network traffic and collect all the information you can about the process, such as the name of the file and where it resides. Then halt the nefarious process and see if the network anomalies stop as well.Now that the situation is temporarily under control, figure out how to keep the attack from being loaded next time the system is booted. Manually search the 14 different places where files can be run from the Windows XP registry or use an automated tool like Autoruns. If you see any entries associated with the rogue file, remove their registry keys.

One of the latest Windows exploits involves ADS (alternate data streams), which Microsoft uses to store thumbnails of images. Unfortunately, ADS can also be used on NTFS systems to hide data in the "description" of another file so attackers' keystroke logs and executable files can evade easy detection. In one case, we found keylog files attached to the winnt.exe file; in another, FTP servers were attached to the C:Windows directory.

Look for these types of files with a utility that's designed to find and clean them, such as Streams' LADS or the latest version of Lavasoft's AdAware. And check your hosts file in C:Windowssystem32etc to make sure all the entries there are legit. Think of the fun an attacker could have by redirecting windowsupdate.microsoft.com to www.i-am-a-hacker.com.

The safest method is to first remove the intruder from the system. Unlike with Unix, there are few recovery tools for booting into a live Windows recovery environment. You can use Knoppix, but this doesn't give you access to the Windows registry. You can boot the system with a powerful tool from Winternals called ERDCommander, but it costs $150 for a workstation and $300 for a server emergency download. Another option is to grab a screwdriver, pull the hard drive out of the afflicted system and mount it on another Windows system.

Now you can regain control of the operating system, as long as you don't execute anything from the compromised drive. Because you can't use Task Manager and netstat to find out what's going on, you'll need to be creative. Consider using Microsoft's Sigverif tool, which checks for digital signatures by examining the compromised drive and investigating any unsigned files.Bear in mind that a file may not be what it seems. A DLL file, for instance, may be a text config script for an FTP server lurking on the system.

Although the automated tools for analyzing the registry won't work, you can load the specific hives of the registry on your system and manually inspect it. Now delete the bad files, check for ADS, scan the system for viruses and check for spyware.

If you've experienced an attack on Unix or Linux, the process is similar but more involved (see "Trouble in Unix and Linux Land").

An Ounce of Prevention

The best first response when you're under attack is to reinstall the computer from a healthy backup. But if you don't have backups or are pressured by management to repair the computer rather than reinstall, things can get dicey: Most IT shops use standard images or install methods. Unfortunately, this leaves every machine in your data center at risk to an attack from the same tool.And as soon as you know you've been infiltrated, have your users change their passwords. During the Linux attack at our university, we dutifully formatted the systems and patched them. During the password change, however, we unknowingly let users enter their same passwords. No more than 15 minutes went by before the attackers logged back into those systems with the same passwords and proceeded to attack the system from the inside. Lesson learned: Keep password histories, and prevent the reuse of passwords for two years.

Jeff Ballard is a Unix systems manager and Dave DeCoster is a network security administrator for the Computer-Aided Engineering Center at the University of Wisconsin-Madison. Send your comments on this article to [email protected].

A forensic disk image is an attack autopsy report, so to speak. It helps you figure out how the attack emerged, how it progressed and what the attacker was hiding. You can create this image using dd, which comes standard with Unix and in tools like Knoppix-STD and FIRE.

The image helps you determine whether the attack was an isolated occurrence. Check a copy of the disk image you made to determine similarities in attack methods with other incidents. Last July, at the University of Wisconsin-Madison, where we work, attackers installed rogue FTP servers on several computers, rotating the infected machines to evade detection. Once we found one sick system and fixed it, another would appear.

By examining stored disk images of the compromised computers, we discovered that the attackers used a unique signature in the FTP servers' banners as well as a unique port: They had hidden the servers on Port 9998 rather than in FTP's usual spot, Port 21. This let the attackers use our servers to host pirated games and movies.If you don't find anything suspicious in your investigation and suspect the attack was held only in the system's memory, dig through the swap file to see if the malware resides there. A memory-resident type of attack tries to avoid discovery, so if you reboot the computer, evidence of the attack and attacker is wiped out.

To locate any files the intruder may have deleted from your system, run file-recovery tools on the image to restore them. Bring your dd image of the victim hard drive to another Unix-type box and use The Sleuth Kit on it. If the kit's command-line interface makes you uncomfortable, use the GUI of The Sleuth Kit, called Autopsy.

If you end up calling your local FBI field office for help, agents may be able to use your forensic image to detect deleted sections of log files and other traces of code to track the attacker.

1. Maintain regular backups of your machines, and keep your software patched. Worms, viruses and intruders often enter through these holes, so make sure you take all the necessary precautions to fill them.

2. Install antivirus software on your Windows-based systems. Keep the virus definitions current.3. Install a host-based firewall. Even if you have a network firewall, add another layer of protection. Allow only the traffic that has to come through, and turn on logging so you can monitor incoming packets regularly.

4. Install a host-based intrusion-detection system. Consider installing IDS software such as Snort. This free software, which can be used on both Windows and Unix systems, alerts you to attacks that are reaching the system and not caught by other security tools.

5. Enforce strong passwords. Make sure all your users devise strong--not easily guessed--passwords. Ensure that user names and passwords are always encrypted when they go over the wire. Use tools like the John the Ripper password cracker on Unix or Linux, or @stake's LC 5 on Microsoft Windows, to strength-test your passwords.

6. Check your routing equipment. Make sure rules are set up so that the router isn't passing bad traffic. Your border router should not allow the IP address range of 192.168.* as a source address, for instance, because this range isn't supposed to be routed over the Internet, as stated in RFC 1597.

Given the complexity and powerful multiuser nature of Unix and Linux machines, regaining control of these machines can be a hard fight.What typically happens when a Unix or Linux machine is compromised is that the intruder first gains user-level access to the computer. Then, he or she can:

• Directly attack other machines--particularly if this machine is behind your firewall.

• Use this machine to obfuscate the origins of this hacker (institution-hopping).

• Serve illegal content on the Internet.

• Look for private company data accessible by legit users.There are some well-known, remotely attackable services, such as old versions of many different types of Web, FTP and SSH servers, but attackers most commonly gain access to Unix and Linux systems by cracking weak passwords. They typically use these passwords to then break into other systems.

Once the intruder has user-level access on the machine, he or she next tries to gain root--the Unix and Linux superuser--privileges. Never underestimate user tendencies to resort to the same (or at least similar) password on every system at their disposal. At the University of Wisconsin-Madison and several other higher-education institutions recently, intruders used passwords learned from a compromised computer at another peer institution.

After going from user-level access to root access via a rootkit, an intruder is likely to alter log files and install a password-cracking tool. Rootkits have one goal: stealth. These automated tools not only escalate privileges, but also let the intruder quickly and effectively hide his or her activities from view, attacking the standard administration programs running on the machine (such as ps, grep and find) or attacking the kernel to hide from any other program on the machine. Install a tool to find rootkits--Rootkit Hunter from Rootkit.nl, for example, scans your computer to find and remove most rootkits.

If you have a separate host collecting your log files, use that host's logs; otherwise, look at the files on the local machine, keeping in mind that your attacker has likely covered his or her tracks.

Now thoroughly check the machine for additional accounts and configuration files in /etc. Of particular interest are /etc/inetd.conf and all files in the /etc/init.d or /etc/rc.d directories or equivalent on your system. Infected PCs and servers are often set to resume the intruder's back doors when the machine is rebooted. Look over your standard programs in /bin, /sbin, /usr/bin and /usr/sbin against known good sources to ensure they weren't replaced.Finally, reboot the machine and monitor it closely. If you've missed something, the intruder may come back and you'll have to repeat this entire process. To find recently changed files, try using The Sleuth Kit from www.sleuthkit.org. And make sure to install the very latest vendor patches.