Refining architecture: American Airlines keeps e-com airborne with site upgrade

Here's your dilemma: Traffic and sales on your e-commerce site are soaring—so much so that the site has begun to crash regularly. Do you live with it or try a major upgrade of the back end that, if unsuccessful, could endanger your e-commerce revenue?

Better the devil you know than the devil you don't?

That's the question Scott Hyden asked himself last fall. The managing director of interactive marketing for American Airlines Inc., in Fort Worth, Texas, Hyden knew the airline's site (www.aa.com) was unstable because it was down for hours every month. But as the second-biggest seller of airline tickets on the Web, American's online presence was also responsible for millions in sales every day. Hyden finally decided on the upgrade—but only after finding sophisticated testing technology that allowed him to see exactly how the site's new servers and databases would respond to the load and the site's high level of personalization.

"We would never have implemented any changes unless we'd been able to test it first," he said.

Hyden's dilemma is being repeated at Web sites across the country, experts say. As pioneer e-commerce sites age, technology advances, and so do the demands of Web-savvy customers, who expect 100 percent uptime and quick-loading sites. A single negative experience can be enough to alienate a customer.

"A lot of big-name sites are rebuilding," said Billie Shea, an analyst for The Newport Group Inc., in Barnstable, Mass. "When the site is not working well, you're missing a lot of opportunities and business, and you don't know how much."

But fix one thing, and other things can break. And for sites that are personalized, such as American's, the opportunity for broken links is even greater, since each page is dynamically generated for a particular customer.

That's why Hyden's decision last fall to revamp American's site was so wrenching. To improve site performance and reliability, American went from one server to four. That created redundancy, but it also increased potential points of failure. "This was a scary thing for this company to go through," Hyden said. "I couldn't tell [upper management] that this was a straightforward thing. There were several high-risk factors here."

The prospect of making big changes was particularly scary, he said, since business on the popular site, despite its instability, is growing at 115 percent per year. The Web accounts for 5 percent to 7 percent of American's total business.

American isn't the only airline relying on the Web for more business. Business travelers, in particular, are making increased use of airline-specific sites. According to Forrester Research Inc., in Cambridge, Mass., 44 percent of business travelers will book some travel online this year. And, Forrester reported, airlines such as American will see 9 percent to 10 percent of their bookings coming from online sources by year's end.

With more travelers accessing American's site, Hyden's goal was simple: "I wanted customers to not notice anything is different" after the site was rearchitected and redistributed, he said.

The first steps were to understand what the problems were, Hyden said: "Our architecture did not have built-in redundancy in the database, Web and server applications. We had way too many single points of failure." At its worst, he said, the American site was down several hours a day—something he had to fix.

In addition to splitting the transaction load between two server facilities in California and two in Virginia, American upgraded the server software on which the sites were based. The American site was built using BroadVision Inc.'s One-to-One Enterprise 3.0, which creates a customized experience for users. But the applications under it "needed cleaning," Hyden said. So he decided upgrade to Version 4.1 of BroadVision's One-to-One. The airline didn't upgrade the applications, since they were custom-built, but they were recompiled to run on the newer version of software. "It was a very unsexy thing to do," Hyden said. "But without this stuff, you can't conduct commerce at all."

All that was left was testing. Hyden needed to make sure that, in redistributing the site's back end, he hadn't introduced new points of failure.

"American is a huge site, and there are so many layers. That means there are so many places things could go wrong," Newport Group's Shea said.

The airline put out a request for proposal for a testing suite, and with the IT department leading the way, put contenders through a rigorous process, including a "cook-off" to determine which tools would work the best with the size and high level of personalization of the site. American, in conjunction with its Web integrator, MarchFirst Inc., chose RSW Software Inc.'s e-Test Suite and service.

RSW, of Waltham, Mass., started the testing process by looking at the usage and load patterns on the site. "Having data to work with helped us create realistic models," said Steve Caplow, director of marketing and business development for RSW. Then the challenge was to isolate and hit each segment of the site to locate its weak spots. Caplow said there were fundamentally three challenges: accurately determining the load, making sure the firewalls were architected correctly for the load and making sure the servers could handle the increased load created by personalization.

"It was crucial that we determine the load accurately because that's where most of the problems can occur," Caplow said. The existing data came in handy for modeling, helping make sure the load tests were accurate. Then RSW ran a series of tests, checking the firewalls and recommending changes where needed.

"We did a lot of successive runs to see what made the load balancing better or worse," Caplow said. "We really needed to be able to correlate the end-user experience with what happens at each tier of the back end."

Finally, the tools had to simulate the personalization experience by creating unique user IDs in Netscape Communications Corp.'s Navigator or Microsoft Corp.'s Internet Explorer, then requesting different pages. Then the pages had to be checked to make sure they held the right information.

"American wanted a stark, realistic view of their site. That's what we gave them," Caplow said.

And it worked. The newly architected site has been up since the beginning of last month and—other than scheduled main tenance and one human error—it's been running just fine, Hyden said.

Now he's free to consider new features for the site. Over the next three months, Hyden hopes to give users the ability to book frequent-flier tickets online. In addition, some customers have told American that they are scared off by the current online ticket purchasing process, which requires registration. So Hyden is looking for a way to give visitors the opportunity to buy tickets without registering. "They'll be able to do guest bookings, and we won't collect any personal information from them," he said. In addition, American is adding a "paging" capability that will let users access the American Airlines site from Palm Inc.'s Palm VIIs, pagers or cell phones. And in the longer term, he'd like to make the site "less cluttered."

But for right now, he's just glad the transition is over. "All I hope is that consumers tell us, 'Hey, the site is up more now,'" he said. "That would be great to hear."