Under Pressure

Just hours after Delta Air Lines Inc. advertised new discounted fares and financial incentives to book tickets online instead of by phone, the airline's Web site last week crumpled under the weight of heavier-than-expected traffic. The site fluctuated between slow and inaccessible for much of the day. "They made a big splash about simplifying their fare structure," says Norm Rose, president of Travel Tech Consulting. "I'd think they'd try to anticipate."

Delta is only one of a handful of organizations recently caught off-guard by surging transaction volumes. A week earlier, the International Red Cross' Web site ground to a halt following an unprecedented influx of online contributions for tsunami relief. On Christmas Day, Delta subsidiary Comair had to cancel nearly all its 1,160 flights when its crew-scheduling system went down after a snowstorm necessitated more changes than the aging system could accommodate. And in the days leading up to Christmas, retailers Amazon.com Inc. and Walgreen Co. experienced significant computer snafus they said were related to high volumes: Amazon customers were left uncertain whether purchases had gone through, and Walgreen inadvertently double- and triple-billed millions of customers' credit-card and bank accounts.

How is it possible that companies still have scalability and reliability problems in such mission-critical systems? The reasons range from system complexity to inadequate planning to plain old bad luck. Whatever the reasons, the recent technological gaffes at companies with years of experience honing their E-business efforts send a warning to others that may be ready to risk skimping on the time or attention or money needed to maintain a scalable IT infrastructure. "You can never pay enough attention to scalability," says Marc West, global CIO of H&R Block Inc. "I believe [these problems] can be avoided and should be avoided. It puts a black eye on the whole prospect of conducting business online."

The companies that faced recent problems declined to discuss details about what went wrong. Many Amazon customers in early December didn't get messages verifying that their purchases had been completed because of "complicated systems that have problems from time to time," an Amazon spokesman says. Delta's site was hit with triple the normal amount of traffic, an airline spokeswoman says. Walgreen says difficulties arose when unusually heavy credit-card transaction volume caused a piece of software to resend transaction batches to a processing vendor. The retailer corrected the errors. "We weren't aware that volume would create this problem," a spokesman says.

Across companies and industries, the pressure to squeeze savings out of computer maintenance to fund new business-technology initiatives has been intense. Add to that the costs of keeping up with compliance and security demands. All that can hurt other areas, including the need to upgrade software or add more servers for Web sites and other customer-transaction systems. "CIOs are very frustrated with having to make zero-sum trade-offs of one priority versus another," says Bob Suh, chief technology strategist for Accenture. "Maybe companies need to stub their toes a bit to realize they need additional capacity."E-business provides more visibility to customers and comes with incredibly high expectations, says Bob Reeder, CIO of Alaska Airlines Inc. Now, as in the past, high transaction volumes can result in customers waiting longer on the phone to buy tickets. Yet when customers go to a Web site and wait minutes for pages to load, the problem is more noticeable and irritating. "There's always a constraint somewhere. You have a buffer of capability, but you know at some point you might exceed it," Reeder says. "We've tried to have an excess capacity on our Web site. We use the heaviest day we've ever had, times three. You hope that's enough." Upon hearing that Delta's site crashed under the strain of traffic that was triple its normal levels, Reeder pauses. "Maybe three times isn't enough."

In many cases, the answer is more rigorous testing, Reeder says. Alaska Airlines' staff recently found that a new piece of code tested fine in the lab, but when it was used on a small number of servers, a slight performance problem popped up. Visitors didn't notice, Reeder says, but it limited the number of transactions the application could accommodate. The airline pulled it down and made upgrades. "There's nothing like a load test on the world's Internet," he says.

Casino resort operator MGM Mirage erred on the side of caution as it approached New Year's Eve weekend, its busiest of the year. CIO Glenn Bonner had his staff analyze and test all of the company's transactional systems by early November to be sure they could handle the onslaught that comes with 99% occupancy at its resorts on the Las Vegas strip. He concluded that continuing to run those systems on the same storage array that supports the company's data warehouse might create some problems, so he acquired a new storage array to minimize the risk.

But other CIOs might not be allowed such flexibility, having been beaten into the ground with cost-cutting, compliance, and security priorities in recent years, Bonner says. Transaction capacity hasn't been a priority among many top business executives, and warnings from IT execs sometimes fall upon deaf ears, he says.

Given intense cost pressures, unpredictable demand, and complex E-business infrastructures that exist at many companies, more such problems are likely. Says Bonner: "They're going to learn that lesson the hard way."-- with Steven Marlin