Visa pulls it all together

With the economy still sagging, many retailers don't know exactly how much business to expect this make-or-break holiday shopping season. Fortunately for them, they can expect a major electronic partner, Visa International, to be ready to handle its usual peak load of 10 million-plus transactions per second, thanks to the recent successful completion of a major legacy system re-engineering and consolidation project that took three years and the efforts of 600 IT professionals.

The BankCard Clearing consolidation effort--which involved merging two large chunks of Visa's huge VisaNet worldwide transaction system and rewriting parts of the system that had been in Assembler code--is indicative of pressure many large enterprises are feeling to revamp large legacy systems to cut maintenance expenses and speed the rollout of new e-commerce capabilities. In Visa's case, the consolidation project will allow the San Francisco company to cut dramatically the time it takes to support future mobile and other e-commerce features with its core systems, officials said.

But the process of upgrading was no simple task. For one thing, transaction systems such as Visa's cannot afford downtime. Complicating the process was that some of the systems Visa was re-engineering were 27 years old, meaning that programmers familiar with them were practically impossible to find. As a result, the project required detailed project management expertise to rival what Santa must go through every year to get toys to all the good girls and boys.

Everywhere you want to be

VisaNet is the world's largest commercial transaction processing system. Running on 25 large mainframe computers and more than 230 midrange systems and comprising roughly 300 applications and more than 50 million lines of code, the system connects Visa's 21,000 member banks, 21 million acceptance locations and nearly 570 million card holders worldwide.

In 1998, Visa and Inovant Inc., the company's technology arm, began implementing a phased plan that would consolidate the two clearing applications in VisaNet--BASE II (Bank Americard Service Exchange) and SMS (Single Message System). The BASE II system, launched in 1974, provides preferred clearing services and authorizes a card holder to make a purchase. In contrast, on SMS, launched 15 years ago, a full financial transaction occurs. The system automatically authorizes and provides enough information to Visa to automatically settle payment.

While the VisaNet clearing systems worked well, the fact that they were aging and that there were two of them drove up maintenance costs and made e-commerce enhancements difficult. By integrating the systems, Visa would have to make upgrades and changes to its settlement systems only once using a single set of tools and development languages. (Visa uses its own proprietary development tools.)

Joel Mittler, senior vice president for strategic projects at Inovant, in San Francisco, said he also expects the re-engineered and consolidated systems to more easily accommodate new e-commerce-oriented payment channels. Mittler said that thanks to the consolidation, such projects should take half as long.

"The focus wasn't on getting transactions through at a faster rate but on how we could respond faster and cheaper to changing business requirements," Mittler said. "The challenge in this business is keeping up with new channels such as mobile payments, e-commerce payments and virtual systems."

To tackle such a large project, Inovant executives took over a videoconferencing room at Visa headquarters and turned it into a command center. From there, 600 Inovant and Visa employees over the course of the three-year upgrade carefully determined its requirements and came up with a general and a detailed design. The project was split into phases. After each phase, the team reassessed the project's complexities, issues, timelines and schedules.

Two factors constantly came into play. During the implementation, transactions needed to continue uninterrupted for Visa's worldwide member financial institutions.

Recoding and testing

The first major phase of the clearing system project--completed after the first 12 months--involved the replacement of thousand of lines of Assembler code with a modular set of proprietary applications and tables developed in-house. Visa, which had struggled to find Assembler mainframe programmers over the last few years, chose to write everything in C--an easier skill set to find among programmers. The entire upgrade meant millions of lines of code were replaced; thousands of lines of new code were written in C; and 8,000 tables, managed by an IBM DB2 database, were built.

That might seem like a lot of work to save a legacy system, but Visa officials said they never seriously considered moving the re-engineered parts of VisaNet off mainframes. That's because Visa's investments in mainframe technology meant it did not make sense to move to a client/server computing environment, officials said.

The second major phase of the project involved testing the system. To ensure the deployment of the new architecture would go smoothly and not interrupt concurrent financial transactions, Visa implemented a series of tests to ensure that the new systems would be able to speak with the systems of member financial institutions. Employees all over the world and selected member institutions oversaw the results of the tests to make sure all transactions were processed. Inovant ran more than 600,000 test scenarios to validate all possible transactions. It also engaged in full-scale production validation with up to 70 million transactions, which were then compared with transactions being conducted in real time by the existing system.

The final deployment, which began at midnight Greenwich Mean Time on July 14, took 12 hours. The videoconferencing-equipped command center was manned 24-by-7 by 50 employees. With officials looking on anxiously, the new system processed 109 million transactions valued at US$3.5 billion in its first two days. During the deployment, both the old and the new systems ran concurrently.

"Everyone at Visa and Inovant was ready for the worst, given the complexity of the upgrade, and the deployment basically exceeded all of our wildest dreams," Mittler said. "Our success, I believe, has to do with communication. We shared every issue--every bad thing, every good thing."

Not that Mittler is resting on his laurels. Inovant and Visa have begun to work on re-engineering VisaNet's profile manager system. The system holds the profiles of all 21,000 member financial institutions, and the project will involve the integration of two separate repositories--one for authorization, another for clearing and settlement.

"As the number of virtual payment channels increases, Visa has to be able to deliver reliability and upgrades to our computing environment," Mittler said.

Case file: Visa International

• Location: San Francisco
• Challenge: Upgrade to new consolidated financial transaction clearing applications for all Visa's worldwide processing systems (VisaNet), which date back to 1974; deploy the applications while continuing to process millions of transactions for Visa's merchants and member financial institutions
• Solution: To prepare, Visa ran more than 600,000 test scenarios, engaged in full-scale production validation with up to 70 million transactions per run and conducted extensive testing; the project involved 750,000 lines of replaced code, 8,000 tables and 600 Visa employees
• What's next: Continue to upgrade VisaNet, including a migration to Internet protocols and a move toward distributed computing