Migrating to Amazon Web Services: The blueprint

Migrating to Amazon Web Services: The blueprint

Summary: Moving your infrastructure to Amazon Web Services--or any other cloud platform--sounds like a no-brainer in many respects. You can scale up or down as needed and pay only for the computing and storage you use.


Moving your infrastructure to Amazon Web Services--or any other cloud platform--sounds like a no-brainer in many respects. You can scale up or down as needed and pay only for the computing and storage you use. The big question, however, is this: How exactly does a company migrate its infrastructure?

Helpstream, a software as a service customer service and relationship management company, recently completed a migration to Amazon Web Services. Bob Warfield, executive vice president of products at Helpstream, recently disclosed the company's move on his blog, his corporate blog and to the Enterprise Irregular mailing list. On Tuesday, Warfield, along with Helpstream CTO Dan Hardy and other executives, hosted a call walking me and Tom Foydel (see Tom's take) through the migration to Amazon Web Services (see all articles).

Helpstream's service has a customer service portal, community module, solutions management and checklists. The company is a big Oracle partnehelpstreammug.pngr and has about 140 clients and 90,000 users. In other words, Helpstream (right) isn't a just-hatched startup, but it's not exactly General Electric either.

Here's a look at Helpstream's blueprint, which took about six months to implement.  In those six months, Helpstream moved its infrastructure to  Amazon Web Services--primarily the Elastic Compute Cloud (EC2) and Simple Storage System (S3). The migration is notable given that Helpstream has never talked to an Amazon person--everything is done online--and billing is handled via a corporate credit card (an interesting quirk of dealing with an e-tailer.

Before we get into the step by step, there are a few high level takeaways:

  • Helpstream's previous infrastructure was operating at 10 percent utilization in most cases unless there was a spike. It had a co-located data center. With Amazon it can add an incremental server in three minutes. Amazon works well for testing,  betas and other experiments. For instance, Helpstream can use Amazon to add capacity so customers can get a sneak peak at an upcoming deployment.
  • Amazon's prices were a big selling point. The margins are narrow for SaaS companies and Amazon's prices were attractive. Warfield reckons that the top half of public SaaS companies growth at a 35 percent clip, but spends 27 percent on service cost.  In other words, the margins stink relative to traditional software. Helpstream reckons it will cut its IT infrastructure costs by about 59 percent. Meanwhile, Amazon's margins on Web Services are much better than e-tailing. Translation: Amazon doesn't have a burning desire to gouge you since the margins on Web services  are infinitely better than its core business. It's a lot like the guy that lives on  Long Island and decides to buy a house in North Carolina. You're not going to sweat  a few pennies given you're shack on Long Island equates to a 5 acres and a lake in the Carolinas.
  • Helpstream was very cognizant of conflict of interests and Amazon was about as Switzerland-like as you could get. Amazon is primarily an e-tailer and that was appealing to Helpstream for a few reasons. For starters, Amazon isn't going to start making SaaS apps--something that could be a worry if you decided to put yourself in Salesforce.com's cloud for instance. Warfield noted that vendor lock-in was a concern, but Amazon was more advanced than other cloud providers and is "very neutral like Switzerland" compared to Helpstream. A company like eBay obviously wouldn't host apps on Amazon Web Services due to potential conflicts and competition.
  • If you are going with something like Amazon Web Services make sure you have analogs with other vendors just in case. Avoid anything proprietary. For instance, the services Helpstream uses are offered by other companies so "we could move to another cloud if we had to," said Warfield.
  • Helpstream was big on transparency, security and other key features that Amazon needs just to run its business. "Amazon has done a lot more than most small companies can do," said Warfield. One key point: Helpstream has a reporting tool outside of Amazon's systems so it can notify customers in the event of an outage.
  • There are cultural issues to ponder. I couldn't help but wonder how this migration would work out for larger companies with legacy applications and a rat's nest of systems (essentially most of you reading this). Helpstream's approach lined up with Amazon Web Services well. To wit: Helpstream's infrastructure is open source based--Apache, MySQL etc.--and lines up with Amazon's. Culturally, Helpstream is OK with not talking to a human. In addition, Helpstream is a SaaS company and understands the cloud. It's unclear whether other companies could build their infrastructure without any handholding let alone move terabytes of data to Amazon's cloud.

aws3.pngWith that background out of the way let's go to the phases. Helpstream's goals were pretty straightforward. The company wanted to cut risks, avoid a move in one big swoop and improve performance.

Phase 1: Backups and test servers

This phase was really a "get to know Amazon" stage. Helpstream's first step was to move its backup copies from tape to Amazon's S3 storage service. The perks were fast backups, redundant physical copies and Helpstream didn't have to pay anyone to visit the datacenter cage.

From there, Helpstream began experimenting with Amazon's EC2 service. Hardy noted that it took less than an hour to bring up a pod, essentially a data center cage. Helpstream began using EC2 for test servers and trials. This phase took about two months and the main benefit was practicing on Amazon's infrastructure.

Phase 2: Move most of storage to S3

In the second phase, Helpstream's goal was to move its BLOBS (binary large objects) such as attachments and knowledge base documents from MySQL to S3. Warfield noted that Helpstream kept the client code unchanged to minimize bugs. It didn't serve documents from S3 initially. Ultimately, Helpstream moved 85 percent of its data to S3 and found that its MySQL and servers became more efficient. By moving most of its storage to S3 it set the stage to move to EC2 without as much data.

One big help in this phase was Amazon's move to offer elastic block storage. The primary benefit: You could mount storage to EC2 and have it mirrored. The process took about a month.

Phase 3: Moving to EC2

With the first two phases under Helpstream's belt it was all about planning for the final move. Here's where Helpstream's size helped. Although Helpstream had a detailed plan, it didn't have hundreds of terabytes to move. A larger company would have had more steps. "There's always something to tease apart," said Warfield. "Having talked to large enterprises, the scary thing to them is the big terabyte data cluster. You have to do it a step at a time."

Helpstream's last phase came down to five hours over a weekend. Here were the steps:

  1. Shutdown the service for maintenance;
  2. Configure EC2 servers;
  3. Move remaining 15 percent of data to EC2 servers;
  4. Test;
  5. Point DNS at new servers;
  6. Leave proxy running in data center.

Fortunately for Helpstream, it went well. Hardy noted that the biggest risk was getting to step 5 and then having to roll back the changes. What moved the process along was migrating most of Helpstream's data ahead of time. Simply put, the more data you need to move the higher the risks.

The payoff (projected):

Since Helpstream has only been on Amazon Web Services for two weeks, its ROI figures are essentially projections. Here's a look at the monthly cost breakdown for Helpstream's data center:

  • It will save 100 percent on servers, switches, VPNs and other infrastructure;
  • 21 percent on bandwidth, cage space and other Amazon charges;
  • 59 percent on monitoring, server administration and in-cage work.

The biggest pop is clearly hardware costs. Helpstream reckons it will deliver savings of 59 percent overall by moving to Amazon. Anecdotally, Warfield noted that customers have seen a performance improvement.

Going forward, Helpstream is looking to install more automation and control panels to scale up infrastructure and manage assets better.

More reading:

Topics: Emerging Tech, Amazon, Software, Networking, Hardware, Enterprise Software, Data Centers, Cloud, Browser, Storage

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Now this is where it makes sense.

    In this sense the so called "cloud" is just an evolution of the common hosting scenarios. But its the only "cloud" scenario that makes sense to me.

    Good article. I'm planning on looking at EC2 for a couple of future ventures. The biggest selling point is the dynamic scaling. It reduces some of the guesswork planning and commitment of resources on the front end.
  • Keep local backups.

    The whole idea of backups is to have data in several places. S3 should complement, not replace, existing backups.

    Even if S3 is in the business of keeping data replicated, I still consider them "one" place, because Amazon may decide to shut down the service if they're financially burdened or go out of business - and in today's financial conditions, that is certainly very possible!

    Same goes for everything else: No matter how large the basket is, you simply do not put all of your eggs in one basket. Hosted services should be in addition to, not replacing, existing services.
    • Why would hosted services imply....

      ....that backups are not being kept or the lack of a disaster recovery plan? Though I do see "the cloud" as just buzzword hoopla it also irritates me when people assume that organizations wishing to make use of the cloud loose all common IT sense when doing so. Rackspace can go out of business just like Amazon. Does anyone question the sanity of an organization using Rackspace managed hosting in this manner?
  • RE: Migrating to Amazon Web Services: The blueprint

    You take excellent notes Larry! Good post and the next question that we all must ask is this: Is there any chance that companies with on-premise software are going to move it to the cloud or are we more likely to see only SaaS companies like HelpStream make the move? If on-premise companies run in the cloud, and gain a lot of cost reduction, then doesn't this hurt SaaS forecasts overall. Or, if only SaaS companies make the move then doesn't it make on-premise just flat out uncompetitive.
  • Amazin' Amazon

    I've long admired the flexibility and scalability that Amazon brings to the table with their cloud services. They've been doing this for several years and basically, have the infrastructure and interface down. The only real problem with them is that they're strictly Linux. Any business software that runs on a mainframe, mid-range or Windows server simply can't play in that arena - and that's 99% of all business software.

    Which of course, leads the the real problem with cloud computing: Converting all your software to run on another platform. In many cases, the software is proprietary and a conversion ain't gonna happen. Even if you are capable of rewriting your business software to run on another platform, who's going to be able to justify that cost? Not to mention the success rate of large software projects and their impact on a companies bottom line!

    Basically, the migration to the cloud is going to be just that: a migration. There's simply no way that businesses can afford the costs and pitfalls of conversion regardless of all the benefits of the cloud platform.

    Steve G.
    • RE: Amazin' Amazon

      You raised good points, but here are my "thinking out loud" thoughts:-

      [i]who's going to be able to justify that cost?[/i]
      Companies working on updates of their software versions, and enhancements which tend to happen every couple of years, can do so and take this into consideration.

      [i]There's simply no way that businesses can afford the costs and pitfalls[/i]
      Thinking of the organization I work for, with over 220 offices in 130 countries, I imagine the cost of IT staff in each office, rental space used for data centers, electricity, hardware, maintenance, ...etc will more than cover the cost of converting many proprietary applications to run on Linux or any other OS.

      Just my $0.02
      • While this all sounds cool, cheap, etc., but..

        for major corporations critical apps where there is sometimes 10 year's or more of data to be converted, validated, and retained. Flipping the switch to a new software environment is easy, but there are usually legal, IRS, SOX, HIPPA, etc. requirements for data security and retention that seem to be overlooked until it comes down to actually doing the implementation. In my experience, the data conversion and validation takes 80% of the $ for a conversion project. If you don't convert historical data, then you have to keep the old system around and the end-users now have to use two systems to look up data.

        Data and record retention is rapidly becoming a major issue and cost for most companies of any size. If there is any litigation, the courts are not looking favorably upon the inability to produce requested documents, emails, etc. as evidence.

        • And thus your company becomes uncompetitive

          Sure this is a big issue for established companies. Your points are completely valid. But once your competitors do it, specifically newer more nimble competitors they will have lower operational costs and thus a bigger competitive threat to your company. You just can't ignore 20 - 50% cost savings. It's a game changer. Better start cleaning up that data now while you have the time. Otherwise, polish up the resume.
    • Dig out of the mud

      Note to self, CTO, CIO, and Purchasing:

      1) Avoid applications that only run on one platform. Unless there is no alternative or the product offers an order of magnitude greater capability over others, keep looking.

      2) Inform current vendors that the organization will be seeking an alternative to their single-platform application. It does not have to be SOA or cloud, but something that does not bind the company's budget to a single operating system or hardware platform.

      3) Ensure that internal projects are multi-platform or have a plan to be multi-platform from a single set of source code. Developers have MANY cross-platform options from which to choose. Java, Python, Perl, etc.

      4) Size correctly. Do not expend capital where 90% of the resources are unused. Prepare to scale according to business requirements. Either way, up or down.
    • Amazon not strictly Linux

      EC2 does offer Windows server 2003 instances for a bit more per hour.
  • RE: Migrating to Amazon Web Services: The blueprint

    Steve, just a heads up that Amazon began offering Windows servers a month or so ago.


  • your article

    Nice job - linked to http://www.TheSOABlog.com
  • RE: Migrating to Amazon Web Services: The blueprint

    Don't you think this is a security risk keeping too much data in one place by so many companies? Sorry about the paranoia. I live in India!
    Nagesh Tummala
    • No reason why you can't keep local backup

      How ever you will have to pay for transfering data to and from EC2.
  • EC2 also good for one time projects

    EC2 is ideal for trying new things and not disrupting your existing servers. For penies you can create a virtual Linux or windows server, log into it and play. When you are done, just destroy the instance. There is no contract and no obligation on your part.

    The problem with Amazon is that many mail serves block incoming mail from amazon virtual servers due to spamers using them.
  • RE: Migrating to Amazon Web Services: The blueprint

    Amazon needs at least one viable competitor in this space, just as salesforce.com does in the CRM SaaS space. Without it, you are locked into the vendor and at their mercy on pricing and service. Microsoft take note. And EMC. Etc.
  • RE: Migrating to Amazon Web Services: The blueprint

    One of the things I found which I moved into the
    amazon cloud in November of last year is that any
    number of domains automatically reject email messages
    coming from mail servers located inside amazon's ip
    space. Example: All GoDaddy domain that use their
    secureserver.net mail service (most, if not all) get
    blocked with no warning. What's particularly
    frustrating is there is no indication, messages come
    back marked No such user x@y.tld .

    Most of these sites blocking the messages make it
    exceedingly difficult to contact anyone in order to
    get the messages flowing.