Overcoming Application Performance Issues

Managing IT for Orangefield Columbus, a global financial services organization with operations in four countries and more than $5 billion in assets, involves complying with countless international requirements and delivering the highest levels of performance and reliability to meet customer expectations.

We recently faced a major challenge when significant data growth led to storage performance latency and degradation. Our service levels were being hampered, and customer expectations were not being met. We needed a much higher-performing storage solution.

For background, our business has traditionally focused on hedge funds, but as we've moved into private equity, our growth has accelerated. As a result, the demands on our IT team to deliver improved quality of service in terms of performance, capacity, and efficiency have increased. These demands mean our team must be at the forefront of adopting emerging technologies to keep pace with customer demand and gain competitive advantage.

Our two most critical applications are a Virtual Portfolio Manager (a portfolio accounting product) from SunGard Financial Services and a Backstop Solutions Group tool called Customer Manager Services CRM. Both are Microsoft SQL-based applications. These mission-critical applications and others we use are virtualized using VMware vSphere 5.1.

Put simply, we receive financial data from numerous sources, compile it into our systems, and generate essential reports. Our CRM and accounting systems are our mission-critical applications, but we run many other apps, including a number of SQL databases, Exchange, Citrix XenClient, web portals, and FTP portals. And because not all applications are created equal, it's important that our IT infrastructure meet the specific performance and capacity requirements of each.

The recent significant growth within our multiple databases led to serious performance degradation in our storage system and hurt our customers' experience. We found ourselves out of space and in need of a much higher-performing storage system. Our initial system used HP's LeftHand Network RAID for data integrity, but network RAID consumes a good amount of network bandwidth and capacity. As our data volumes grew, we saw degradation in application performance. Something had to change.

Implementation and challenges
In evaluating storage systems, we focused on products from EMC and Fusion-io. We ended up selecting the Fusion ioControl Hybrid Storage product, which better matched our needs and was economical.

Our IT staff implemented the product after a few hours of training. First, a pilot project was initiated to practice what we had learned, so that we knew how to provision space and add drives in VMware. Second, we had to evaluate all our data. Previously, we had about 100 LUNs on our SAN to manage data replication to our disaster recovery (DR) system very granularly, but the Fusion ioControl and Zerto DR, another technology we deployed, did not have the same limitations.

Also, the Fusion ioControl has priority access to the SAN, and we wanted to take advantage of that. As a result, we arranged our VMware drives into new groups based on the virtual servers' desired priority within the SAN.

Since this was new to us, we did some testing of the different priorities groups using staging servers to test the performance for our applications. Once we were satisfied with how to assign our VMware LUNs to the SAN's priority groups, we could start the migration.

The servers were moved one by one using VMware's server migration function, which moved the servers' drives into the Fusion ioControl system. We moved our staging servers over first and ran tests on them. Once staging passed testing, we were able to move over production servers. Though VMware can "live migrate" a server, we found it more efficient to migrate the servers during off hours.

Our biggest challenge during implementation was configuring our SAN to replicate to the DR site. We did not have the budget to implement a SAN for DR; instead, we chose a DR service that provides space, processing power, and network access. This service charges us only for what we use. Data on the SAN was replicated to the DR site using Zerto for VMware.

The results
Since implementing the new system, we've experienced a huge performance improvement. With our original system, we were maxing out at 3,000 IOPS, but with the Fusion system, we can hit up to 100,000 IOPS. We haven't come close to hitting the max yet, and this ample performance gives us confidence that we can meet our customers' requirements.

We've eliminated system contention during our peak demand times (before the market opens and after it closes), and we have the capacity we need to continue our accelerated growth. Overall, we've seen a considerable improvement in system performance and, as a result, we're able to deliver high levels of customer service and meet our customers' expectations.

Additional benefits we have experienced include decreasing time spent on storage management from five hours per month to one hour per month. The new system also consumes 33% less power.

The key to our successful implementation of Fusion ioControl was understanding how to use the priority groups for our environment. Though each organization will have specific requirements for its applications and special needs depending on its environment and business objectives, the process we went through may be beneficial to other organizations looking to overcome application performance issues.