Southern Co.: A Smarter Storage Strategy

So archiving has become the new mantra at Southern. The utility is now test-running an archival strategy based on Veritas Software Corp.'s Storage Migrator. Until now, Southern has archived its data by storing manual information on spreadsheets, for example, on an IT staffer's C:/ drive. But by instead aging off its data to archived tape, the utility expects to save 40 percent of its storage capacity.

"By simply archiving data that's used less frequently, we expect the backup-and-restore process to speed up by a factor of six," says Gail Commer, manager of enterprise storage at Southern. Backup has been a hot button at the highest levels of the organization for some time (see "The Hard Sell," below), and the archiving strategy was a natural next step.

Conservation Measures

It was the company's new policy to limit e-mail storage on its messaging servers and shared drives that finally drove Southern to build an archival system. Employees now must delete any e-mail that's not business-critical and use Outlook PST files sparingly to avoid bogging down Southern's storage.

"A lot of our data growth has been from users cleaning up their e-mail boxes because they're limited to a megabyte of storage for Exchange," Commer says. "One thing we want to stop is the storing of messages outside Exchange." That's because Southern can better control and retrieve business-critical messages if they're stored on the Exchange servers, she says.The company's hefty internal applications have also put the squeeze on data-storage space. Southern's homegrown TCMS (Trouble Call Management System) application, for example, uses heavy graphics and GIS (geographical information system) data to help the company detect where a power outage has occurred, Commer says. When a customer calls in an outage, the location of his or her residence is illuminated on an electronic map so the utility can determine the location of the downed substation or transformer.

Southern also runs several applications that store geographic and hydroelectric information, and several applications that map the lakes and land the utility manages for its power generation and resources.

The company backs up about half of its corporate data across three SANs, which are located at data centers in Atlanta, Birmingham, Ala., and Inverness, Ala. The amount of data backed up to these SANs is increasing, while remote sites are starting to back up their data to regional hub sites instead of locally. Backing up their data to the SANs via the corporate WAN requires too much bandwidth.

SANity Check

Southern is in the process of consolidating its SAN software with Veritas' Storage Migrator. It's also moving its backup to a single software platform, Veritas NetBackup, replacing separate packages for its legacy and Windows NT data (Arcserv and Legato). Storage Migrator manages storage parameters, backup and restore, and archiving. The company says it hopes to finish converting its roughly 400 corporate servers, as well as its remote sites, to regional hub backups within the two years.The Veritas backup software already has sped up backups for the utility. It used to take four days to back up a 3-TB Windows NT server cluster; with Veritas NetBackup, it takes just 24 hours, Park says. "And we still want to shrink that window even further," he adds.

Moreover, the new software will let Southern finally manage backups centrally, Park says. The goal is to have software agents sitting on remote-site servers and feeding data back to a central management console to handle outage alerts and other problems. That way, the company can eliminate as many local backups as possible and prevent backup-and-restore problems before they occur.

"There have been cases where backups were not as current as we would like them to be, which is not acceptable," Commer says.

Today, the company uses a mix of automated storage-management tools and manual processes, with spreadsheets that track the owners of the data and the applications that run on each server.

SAN and storage management, however, don't typically get as much attention in budget planning because they're already getting done, albeit manually, Park says. Southern developed some in-house storage-reporting tools, including those for inventory management and file-system management. The company also runs Configuresoft's Enterprise Configuration Manager for tracking and managing storage resources for its Windows servers.What's Next

ISCSI storage technology, meanwhile, may be on the horizon for remote sites where technicians aren't stationed full-time. That way, Southern could set up more reliable backups that don't require hands-on support. The utility already has installed a Veritas VeriStor iSCSI disk subsystem at one of its remote sites.

"We're looking for something that's easy to connect and disconnect, and iSCSI just plugs in like any other device and any server there can look at it," says Mark Rawson, a senior storage analyst at Southern. "iSCSI has a better remote capability. Someone down there just plugs it into an Ethernet hub, and we do the rest from here."

Tell us about you Network and we may profile it in a future issue. Send e-mail to [email protected] or call (516) 562-5914.

Post a comment or question on this story.

When Archiving Is EssentialArchiving isn't the easiest storage project to sell. It's not sexy like a SAN, and it doesn't have a clear bottom-line payoff. But prior to the installation of their archival system, the IT team at Southern Co. couldn't meet its backup windows. Failures were common.

The company's vulnerability hit home last winter when several executives at the Atlanta-based utility lost their shared folders during a backup. That was the last straw.

Plus "the amount of data being backed up was growing so quickly," says Gail Commer, manager of enterprise storage at Southern Co. These factors made the archival piece of the storage strategy a simpler sell. The power company didn't disclose what it spent on the archival system, which is based on Veritas Software Corp.'s Storage Migrator.

Although selling the CIO on the company's SAN integration project was economically justified, archiving isn't necessarily a matter of economics. It's a matter of mitigating risks, which in turn helps prevent data and financial loss.

Once Southern's backup-window times improve and become more reliable with the new archival system, the company should see some significant payback, according to Roger Park, manager of messaging, server support and enterprise storage at Southern Co. Roger Park: Manager of messaging, server support and enterprise storage, Southern Co., AtlantaRoger Park, 56, manages the servers and desktop systems group that supports server hardware and operating systems for Southern Co.'s 750 Windows and 250 Unix platforms. His group also develops automated installation scripts and images for the utility's 1,000 servers and 25,000 workstations. He's been in IT for 34 years and with Southern Co. for 27 years. Park holds a bachelor's degree from Utica College of Syracuse University.

Next time I'll: Implement a different backup and recovery strategy at distributed sites--one aimed at reducing the amount of data to be backed up at such sites. I'll avoid getting so close to the bleeding edge and ensure that I choose technology that provides the best balance between robustness and cost.

Most ill-timed backup failure: No backup failure is good, but the restore experiences stand out most. Restoring 200 GB of data to a server hundreds of miles away simply takes too long for anyone to be happy. That's why we're pursuing our storage strategy.

Why direct-attached storage is out: When hundreds of servers are involved, managing direct-attached storage and the associated backup activities is not as cost-effective as the alternatives.

Biggest mistake made in technology circles today: Forgetting that technology exists to serve the business.Best advice: Hire people smarter than yourself and then get out of their way.

Biggest bet I ever made: Moving to Atlanta sight-unseen in October 1976. I landed my job with Southern Co. during my second week here and I've loved the job and the city ever since.

For fun: Reading, cooking, genealogy research

Wheels: 1991 Honda Accord with 95,000 miles on it