A New SAN Improves Application Performance And Saves Paydays

Imagine gathering all your employees together at 5:00 p.m. on payday to tell them their checks haven't arrived. After applying bandages to your wounds, imagine the phone call you'd make to the payroll processor who blew the delivery. Jack Rahner has never sat on the receiving end of such a call, and never intends to. But AlphaStaff, where Rahner is VP of IT operations, runs a legacy enterprise resource planning (ERP) system, and sometimes latency in the system has meant the human resources outsourcing company had to hand-deliver clients' payroll to planes waiting on the tarmac. Rather than continue to play it so close, Rahner sought out a new SAN with fast, solid state disks to improve ERP performance.

AlphaStaff has grown from 6,000 employees to 80,000 in the past five years. Data has grown alongside the employees, climbing from 1Tbyte to 60Tbytes now under management, and the company was outgrowing its original storage array and wanted to upgrade. In particular, SSD support was important because AlphaStaff's enterprise resource planning (ERP) application runs on older technology--a Unidata database. The database was written back in the days when memory was very expensive, so all the temp files are stored on disk rather than in memory. Rahner says the ERP system would benefit from high-performance SSDs. Besides SSDs, the SAN would have to support ISCSI, as AlphaStaff wanted to stick with its iSCSI backbone.

So about a year and a half ago, Rahner and his team began the search for a new storage array. "Our data has expanded rapidly, and storage had become harder to manage," says Rahner. He says the organization has outgrown the small and mid-size IT tools that the company implemented years ago. "What was supposed to scale out, we topped out."

Rahner considered several options, including Compellent and NetApp. He participated in a month-long bake-off that Compellent performed in its Eden Prairie, Minn. headquarters. Rahner brought his existing array and the ERP app to Eden Prairie and did simulation testing using an I/O meter, and ran simulations to forecast for growth. He says during the testing Compellent was 60 percent faster than his existing system and less expensive per gigabyte.

Rahner was also impressed with Compellent's automated tiering capabilities, which moves frequently accessed data to high-performance disk, while older data is moved to less expensive drives. This lets AlphaStaff use a mix of drive types and speeds to balance data storage needs. With automated tiering, AlphaStaff has moved 80 percent of its data to lower tiers of 7.2K 1TB and 2TB SAS drives.

That automation also meant Rahner didn't have to hire another employee. The company used to have a full-time administrator whose primary job was to reconfigure storage to keep data on the most appropriate tier to maintain high performance. Rahner has now been able to move that administrator to new projects. "There is a real ROI," says Rahner, who notes the additional employee would've cost the company up to $60,000. "The savings were almost immediate."

The Compellent SAN has had other benefits as well. A feature called Lightweight Snapshots lets the company back up and recover data that changes often. The snapshot captures block-level changes, so backups take up less space and occur faster. The SAN is configured to take a snapshot once an hour, and retain each snapshot for 14 days. Rahner says that with the old array, backups would have to be restored from tape, which could take up to 24 hours. Now that backups are on disk, they can be restored in ten minutes. In addition, reports that used to take two days can now run in an hour.

Perhaps the best benefit is the boost in performance of its mission-critical ERP system. The SSDs have cut latency from 22 milliseconds down to two milliseconds in almost every single transaction associated with processing client payrolls. This latency reduction is key, because many of AlphaStaff's clients still deliver physical checks to their employees. AlphaStaff has to ship those checks on time or people don't get paid. "We've never missed a client's payroll in our organizational history," Rahner says, noting that the company has even hand-delivered payroll to its clients during a hurricane. "But although we never missed a payroll, we were reaching critical mass and chasing airplanes." He won't have to do it any more.