Southern Co. Seeks Smarter Storage Strategy

The process of backup and recovery was getting downright painful for Southern Co. When an accounting server failed during a month-end close, it took the Atlanta-based utility 30 hours to recover the server's 300 GB of financial data.

"We were seeing our risk increasing with the size of data that had to be backed up," says Roger Park, the company's manager of messaging, server support and enterprise storage.

Much of what Southern had been diligently backing up nightly was outdated or extraneous data, such as casual e-mail messages. The company, which produces about 30,000 megawatts of power for more than 4 million customers in the Southeast, has seen an explosion in data-storage requirements as its business has become more automated and has expanded into new areas, such as gas power. The Windows NT data alone grew from 27 TB to 50 TB in the past year and a half, some of which came from Microsoft Exchange data that its users had moved to Microsoft Outlook Personal Folders on the company's SANs (storage-area networks).

So archiving has become the new mantra at Southern. The utility is now test-running an archival strategy based on Veritas Software Corp.'s Storage Migrator. Until now, Southern has archived its data by storing manual information on spreadsheets, for example, on an IT staffer's C:/ drive. But by instead aging off its data to archived tape, the utility expects to save 40 percent of its storage capacity.

"By simply archiving data that's used less frequently, we expect the backup-and-restore process to speed up by a factor of six," says Gail Commer, manager of enterprise storage at Southern. Backup has been a hot button at the highest levels of the organization for some time (see "The Hard Sell," below), and the archiving strategy was a natural next step.CONSERVATION MEASURES

It was the company's new policy to limit e-mail storage on its messaging servers and shared drives that finally drove Southern to build an archival system. Employees now must delete any e-mail that's not business-critical and use Outlook PST files sparingly to avoid bogging down Southern's storage.

"A lot of our data growth has been from users cleaning up their e-mail boxes because they're limited to a megabyte of storage for Exchange," Commer says. "One thing we want to stop is the storing of messages outside Exchange." That's because Southern can better control and retrieve business-critical messages if they're stored on the Exchange servers, she says.

The company's hefty internal applications have also put the squeeze on data-storage space. Southern's homegrown TCMS (Trouble Call Management System) application, for example, uses heavy graphics and GIS (geographical information system) data to help the company detect where a power outage has occurred, Commer says. When a customer calls in an outage, the location of his or her residence is illuminated on an electronic map so the utility can determine the location of the downed substation or transformer.

Southern also runs several applications that store geographic and hydroelectric information, and several applications that map the lakes and land the utility manages for its power generation and resources.The company backs up about half of its corporate data across three SANs, which are located at data centers in Atlanta, Birmingham, Ala., and Inverness, Ala. The amount of data backed up to these SANs is increasing, while remote sites are starting to back up their data to regional hub sites instead of locally. Backing up their data to the SANs via the corporate WAN requires too much bandwidth.

SANity CHECK

Southern is in the process of consolidating its SAN software with Veritas' Storage Migrator. It's also moving its backup to a single software platform, Veritas NetBackup, replacing separate packages for its legacy and Windows NT data (Arcserv and Legato). Storage Migrator manages storage parameters, backup and restore, and archiving. The company says it hopes to finish converting its roughly 400 corporate servers, as well as its remote sites, to regional hub backups within the two years.

The Veritas backup software already has sped up backups for the utility. It used to take four days to back up a 3-TB Windows NT server cluster; with Veritas NetBackup, it takes just 24 hours, Park says. "And we still want to shrink that window even further," he adds.

Moreover, the new software will let Southern finally manage backups centrally, Park says. The goal is to have software agents sitting on remote-site servers and feeding data back to a central management console to handle outage alerts and other problems. That way, the company can eliminate as many local backups as possible and prevent backup-and-restore problems before they occur."There have been cases where backups were not as current as we would like them to be, which is not acceptable," Commer says.

Today, the company uses a mix of automated storage-management tools and manual processes, with spreadsheets that track the owners of the data and the applications that run on each server.

SAN and storage management, however, don't typically get as much attention in budget planning because they're already getting done, albeit manually, Park says. Southern developed some in-house storage-reporting tools, including those for inventory management and file-system management. The company also runs Configuresoft's Enterprise Configuration Manager for tracking and managing storage resources for its Windows servers.

WHAT'S NEXT

ISCSI storage technology, meanwhile, may be on the horizon for remote sites where technicians aren't stationed full-time. That way, Southern could set up more reliable backups that don't require hands-on support. The utility already has installed a Veritas VeriStor iSCSI disk subsystem at one of its remote sites."We're looking for something that's easy to connect and disconnect, and iSCSI just plugs in like any other device and any server there can look at it," says Mark Rawson, a senior storage analyst at Southern. "iSCSI has a better remote capability. Someone down there just plugs it into an Ethernet hub, and we do the rest from here."

WHEN ARCHIVING IS ESSENTIAL

Archiving isn't the easiest storage project to sell. It's not sexy like a SAN, and it doesn't have a clear bottom-line payoff. But prior to the installation of their archival system, the IT team at Southern Co. couldn't meet its backup windows. Failures were common.

The company's vulnerability hit home last winter when several executives at the Atlanta-based utility lost their shared folders during a backup. That was the last straw.

Plus "the amount of data being backed up was growing so quickly," says Gail Commer, manager of enterprise storage at Southern Co. These factors made the archival piece of the storage strategy a simpler sell. The power company didn't disclose what it spent on the archival system, which is based on Veritas Software Corp.'s Storage Migrator.Although selling the CIO on the company's SAN integration project was economically justified, archiving isn't necessarily a matter of economics. It's a matter of mitigating risks, which in turn helps prevent data and financial loss.

Once Southern's backup-window times improve and become more reliable with the new archival system, the company should see some significant payback, according to Roger Park, manager of messaging, server support and enterprise storage at Southern Co.