Big Data Means Big File Sharing

In this age of Gmail, CloudApp, and Dropbox, it seems like sharing files across organizations and geographies would be a solved problem. Just email it, or upload it and send a link. But what if you're a video producer on location needing to send the day's clips, at about 500 GB per hour for 1080p video, back to Hollywood for editing? Or perhaps a medical researcher sharing with colleagues DNA sequences clocking in at 3 GB per genome, or even a road construction company getting 60-MB CAD file design updates from the local highway department?

Email is usually out of the question, since even Google limits attachment sizes to 25 MB. Ditto for Dropbox; even if you don't mind the $20-per-month price tag per 100 GB quota, the service caps uploads at 300 MB. Likewise for CloudApp, which caps file sizes at 50 MB. And even assuming you find a place in the cloud to share the data, you've still got to get it there over network pipes and file transfer protocols like FTP or WebDAV, which were designed for browser sessions or small downloads, not gigabyte uploads.

This is just the problem Aspera is looking to solve with its Aspera On-Demand virtual appliance, which facilitates moving data to and from Amazon Web Services at up to 350 Mbps, dependent on your WAN link. So, if you're an enterprise that has moved data- and processor-intensive applications into Amazon's cloud--particularly apps where users are constantly manipulating or updating the data--you now have a means of accelerating their access to near-LAN-level speeds.

From its roots in the media and entertainment market, Aspera is targeting the new offering to address the growing need for efficient "big data" transfer in industries ranging from defense to pharmaceuticals. Its first product, locally installed server software with an associated client application, amounted to little more than an FTP replacement--FTP 2.0 if you will, albeit one with sizable benefits, including orders-of-magnitude increases in throughput, in-flight AES encryption, and bandwidth-usage controls. But it's aimed at businesses with the IT resources and mega-file volume to justify purchasing yet another piece of server software and distributing another unfamiliar client application to end users. Smaller businesses that can't go that route but still need to collaborate on large data sets were out of luck--until now.

Aspera's niche was, until recently, a corner of the IT market of interest primarily to the media and entertainment industries. The cloud, and the simultaneous but largely unrelated growth in rich media generally, has changed all that as companies across the industrial spectrum now need to move large amounts of data across the Internet. And this problem of transferring unique data sets from disparate, sometimes mobile sources (like the aforementioned video producer on a shoot) is one that neither WAN-optimization appliances nor content-distribution networks are designed to solve. Aspera's technology is designed to improve content delivery in a number of scenarios--not only ad hoc file transfer and synchronization, but also data transfer into the cloud, from both PCs and, now, mobile devices.

Historically, file transfer optimization products like Aspera's have used a client-server software architecture. You run a proprietary server "receptacle" in the enterprise and distribute client applications to users who transfer big data. This works fine for the ad agency scenario filming a new car ad at the Bonneville Salt Flats (the middle of nowhere, Internet-wise) needing to copy the day's best clips back to Hollywood for editing. But cloud services introduce another, and likely much more common, use case, one in which you don't directly control the server hardware and are using the cloud to facilitate working with people outside your company. Yet using the cloud when the object of discussion is a 100-plus-MB video clip, scientific data set, or CAD file can feel like trying to drain a swimming pool with a drinking straw. Here is where Aspera's new virtual appliance, which turns its server software into an Amazon EC2 instance tied to an S3 storage pool, comes in handy.

Aspera isn't alone in addressing the need for more efficient transfer of and access to cloud-resident data. RocketStream, which has since been acquired by Tibco and incorporated into its Managed File Transfer service; FileCatalyst; and RepliWeb offer similar managed data transfer services. If you're in a business that routinely manages large data files, this new generation of services that move data transfer optimization technology into the cloud is worth investigating. Speeding up file transfers by one or two orders of magnitude does wonders to your ability to efficiently collaborate across organizations and geographies.

Get lessons from five companies on the front lines of implementing unified communications. Also in the all-digital supplement of Network Computing: Mike Fratto on how to make the case for UC. Download the supplement now. (Free registration required.)