Aspera on demand big data for Amazon Web Services

Big data can mean big problems in the cloud.

Aspera's Bhavik Vyas and I had a long conversation about the impact cloud computing is having on users of "big data." Organizations have been trying to deal with an ever expanding mass of structured and non structured data and trying to move all or some of that data into a cloud computing environment, such as Amazon's Web Services (AWS) creates some pretty serious challenges. These organizations simply have too much data to move into and out of cloud-based solutions to use traditional file movement tools.

In Aspera's words, cloud services offer the illusion of infinite computing resources and also eliminate up front commitments to specific hardware resources. When an organization takes the suppliers of these services up on their offers, they soon find that getting data into and out of the cloud-based implementation of their application can become a serious bottleneck.

So, they've stuck a deal with Amazon AWS to offer their technology to accelerate the movement of data into and out of the cloud.

Snapshot analysis

I've posted about Aspera's file movement software in the past (see Aspera - High speed data transfers for more information.) The company has realized that traditional data movement technology was based upon assumptions about network performance, network latency, network error rates and the like that are no longer true of today's high speed networks. This means that the traditional file movement protocols and tools don't offer enough performance when used on today's networks. This really becomes visible if an organization places an application into an Infrastructure as a Service (IaaS) environment such as that offered by Amazon.

Although only a segment of Amazon AWS users will run into the data movement bottleneck, those do will find it quite painful.  It may mean the success or failure of a cloud services implementation. Aspera's products might be a very useful addition to an IT developer's tool kit.