Two Companies Cope With Storage And Transfer Of Large Files

Increasingly large, unstructured data files are inundating corporate data centers storage capacity and WAN connections. For companies that are using more and more videos, photographs and graphics, file sizes are naturally expected to grow, but in a recent Network Computing State of Storage survey [registration required], the majority of respondents said storage growth was 10-24 percent year over year. All of this contributes to the ongoing saga of what to do with this burgeoning data as it meets up with internal storage repositories and communications bandwidths that were designed for smaller files of day-to-day transaction processing, not for large data payloads.

The dilemma is two-fold for organizations: coming up with enough bandwidth so you can optimize large file transfer at rates that are acceptable for the business  and potentially redesigning storage schemes to accommodate a growing population of large files.

Dave Lunemann, client services manager at Fiber Utilities Group,  a private network outsourcing management company with over 8,000 miles of fiber optic network, says "Years ago, the thought of a private network was not as earth-shattering as it is today. This is because so many companies now are Internet-dependent for their wide area network (WAN) needs."

That tide is starting to turn, however, with large enterprises and companies whose main business focus is in the large files of data that they work with. "Large enterprises have highly demanding quality of service (QoS), security, latency and redundancy needs," said Lunemann. "This is what is making them return to private networks that don't subject them to bandwidth constraints from standard Internet."

The results can be impressive. The response time on a fiber optic private network allows a customer in Utah to see a bank in Iowa as a local presence, and this is good for business. A west coast genome research operation can transmit a very large file to its east coast collaborator securely and quickly, without wasting scientists' time. A multi-state healthcare company can deftly transfer MRIs and x-rays, optimizing patient care.The Mill is a London-based visual effects company with developers in London, New York and Los Angeles, and clients like Nike, Turner Classic Movies and the BBC. "We actually use a private MPLS (Multiprotocol Label Switching) network to link up all of our offices," said computer system manager Jonathan Brazier."

Of course, the large file management quandary does not stop at navigating through network bandwidth constraints. The next step for companies is to craft a data center design strategy for storing the data in a way that permits rapid access and cost-effective resource utilization.

Ilion is a Spanish video production house that creates feature film animation for a worldwide market. "At the peak of production, we were using over 350 people," says Gonzalo Rueda, Ilion's CTO and director of technology. Ilion started as a 20-40 person operation, so the extreme ramp-up of personnel coupled with the large animation files that were being produced daily placed enormous pressures on its video rendering system.

"We hired consultants and we looked at several different options to address our large file storage needs," says Rueda. "We were interested in more than just how many IOPS (input/output per second) the storage system was capable of handling, or the sheer speed of throughput. We wanted a solution that was compatible with the way that we did business. For me, this meant that we had to rethink our storage infrastructure so we could move it forward to better align it with the needs of the business."

Ilion ultimately decided to upgrade storage with a two-head, BluArc Titan 2200 storage cluster. "We had 90 terabytes of data that we were working with on our active network stations and another 110 terabytes of data that we needed to access less frequently, and that we had to store in archives," said Rueda.  "We also had to redefine data best practices for redundancy and hot spares."The result was a combination of FC and SATA drives, with metadata and constantly accessed data being placed on the faster, more expensive FC drives, and less accessed data being stored on slower, more affordable SATA drives.

"Since we cut over to this solution, we have experienced a 3-4x increase in speed, and we have seen that our decisions on what to allocate to FC versus SATA have worked well," said Arueda. "It has enabled us to support heavy production workloads where we have 70 animators working at the same time and in real-time with files without degradation. We also have scalability. This is the kind of production capacity we need to support our growing business."

Video production houses like Ilion have unique requirements that are not typical for other SMBs, in the same way that multi-national banks do not characterize the bandwidth needs of all large enterprises, but they illustrate that managing large data files will become everyone's challenge. This is principally due to a move toward more files with rich media content, and abundant use of Internet and emerging Internet forums for companies such as social networks. All produce large files, and all demand sizable bandwidth.

What does this portend for corporate IT in enterprises and SMBs? That if they haven't already done so, it is time to review and rethink storage and network strategies. Fiscal conservatism in the data center isn't about to change, so few organizations will discard everything they have and opt to buy new. However, revisiting road maps now, matching these up with hardware and software asset retirement cycles and taking a fresh look at how well data center storage and bandwidth are keeping up with the business make sense for 2011 budget planning.