IT failure at Heathrow T5: What really happened

Anyone having the misfortune to pass recently through Heathrow's new Terminal 5 knows the huge project debuted as a case study in full-blown systems failure. Most reports have focused on stranded customers, a crazed supermodel, and Queen Elizabeth's grand opening speech, which called the terminal "a 21st Century gateway to Britain."

Here's my rundown of the major failures and why this debacle happened.

Project scope. Terminal 5 represents an enormous investment in buildings, systems, technology, systems, and human processes. From ComputerworldUK:

Terminal 5 has cost BA and the British Airports Authority £4.3bn [$8.5 billion] to build and outfit. BA says around £75m of these costs are for technology, while BAA invested at least another £175m in IT systems.

The work has involved 180 IT suppliers and seen 163 IT systems installed.

The sheer physical size of the project is impressive. From the official brochure:

Terminal 5's technical complexity matches its physical size. According to the British Airports Authority (BAA), "It has taken 400,000 man-hours of software engineering just to develop the complex system, and coding work is set to continue even after the initial installation begins." From CIO UK:

The project is a complex one. T5 will involve 180 IT suppliers and run 163 IT systems, 546 interfaces, more than 9,000 connected devices, 2,100 PCs and “enough cable to lay to Istanbul and back”. It will contain 175 lifts, 131 escalators and 18km of conveyor belts for baggage handling. [British Airways CIO Paul] Coby says that even the construction of T5 involves: “creating a small town with a full telecommunications network for the construction workers, merely to enable the terminal to be built.”

Political fallout. The impact of the problem has hit the highest levels of British politics. The Guardian reports:

Foreign secretary David Miliband also joined the criticism of BA as it emerged that the crisis could cost the airline £50m. Miliband said a furious EU ministerial counterpart, whose bags had been lost at T5, had asked him to reprimand BA and BAA. Writing on his blog, the foreign secretary said he had been harangued at a gathering of EU foreign ministers, days after the terminal's opening was marred by a botched baggage handling operation and multiple flight cancellations.

UK Under-Secretary of State for Transport, Jim Fitzpatrick, testified before the House of Commons under harsh questioning:

The management of terminal 5 is an operational matter for British Airways and BAA, but that does not mean that the Government are not taking a keen interest in seeing that the difficulties T5 has faced since it opened last week are addressed and resolved as quickly as possible. On it first day in operation, T5’s bespoke baggage system was affected by a number of issues. First, there were technical software problems, but more significantly BA’s challenge was integrating teams of staff, and it has been addressing this as a priority....

[N]ational pride has been dented. Her Majesty opened T5 to national fanfare and delight in early March. The following week, when the A380 arrived for its first flight, many of us believed that Heathrow had turned a corner and that the bad publicity of recent years would turn into positive publicity, notwithstanding the importance of scrutiny as regards the expansion. Clearly, that was not the case.

Baggage systems failure. Stranded travelers immediately understood how problems with the baggage handling system affected them personally. A passenger told the BBC:

"Apparently the computer software told the baggage people that the flight had taken off. So everyone in the plane just watched as all their suitcases were taken back into the terminal instead of being loaded on."

One baggage worker told the BBC the situation was "mayhem" and that the technical problems had been known about for some time. "They have been doing tests on the belt system for the last few weeks and knew it wasn't going right. The computer cannot cope with the number of bags going through."

Airport-technology.com describes the system:

The baggage handling system at T5 will be the largest baggage handling system in Europe for a single terminal. There will be two systems including a main baggage sorter and a fast track system.

The system was designed by an integrated team from BAA, BA and Vanderlande Industries of the Netherlands, and will handle both intra-terminal and inter-terminal luggage and will actually process 70,000 bags a day.

Bags undergo several processes on the way through the system including automatic identification, explosives screening, fast tracking for urgent bags, sorting and automatic sorting and passenger reconciliation.

Vanderlande adds technical detail:

The baggage handling system has a total of 12 transfer break lines and 132 check-in desks according to the ‘walk through’ concept for easy flow of large number of passengers. Bags are automatically read, screened and sorted to their final build location via two electrical HELIXORTERS™. Early bags are stored in a BAGSTORE™ warehouse with 4,000 positions, allowing individual access to each bag any time. Late bags can be transported at high speed via BAGTRAX™ directly to the head of stands where the awaiting aircraft are parked.

Vanderlande Industries forms part of the T5 Strategic Team and is supported by BAA's Baggage Delivery Team. Vanderlande Industries is fully responsible for design, software development, manufacturing, installation, commissioning and integration of this baggage handling system.

Computer Weekly reports that IBM supplied software components:

The system was designed with Dutch company Vanderlande and IBM. The IBM software works out where the bags are supposed to be going, and logistics software works out the best way to get there.

Car park problems. Among all the other problems, the BBC reports that terminal staff had trouble new parking garages, which exacerbated the other issues. Given everything else, this one's just icing on the cake.

Privacy concerns. The British Airports Authority eliminated biometric security measures (fingerprinting all passengers) immediately prior to the terminal opening due to legal questions. Computer Weekly reports:

Heathrow Airport owner BAA is pulling a biometric fingerprint system at the new Terminal 5 the day before the building opens to the public, after the Information Commissioner’s Office (ICO) raised concerns about the system.

The system would have taken the fingerprints of all domestic passengers as they passed through the security gates, and was intended to prevent illegal immigration.

Computing elaborated on this issue:

The Information Commissioner’s Office (ICO) expressed concern that the move may breach data protection laws – even though BAA planned to delete fingerprint data after 24 hours.

"We have concerns about the routine collection of fingerprint information from passengers and we will require reassurance from BAA that the data protection implications of the proposals have been fully addressed," said an ICO spokeswoman.

BAA will now use a single digital photograph to confirm passengers at the gate are the same as those that check in – a system used in most other international UK airports.

THE PROJECT FAILURES ANALYSIS

Canceled flights, lost baggage, and substantial delays were symptoms that arose from failures in management, planning, and testing on this expensive and complex project.

As reported in ComputerworldUK, the T5 systems represented a massive business transformation effort for British Airways:

The technology has let British Airways create what is believed to be one of the airline industry’s first Order Data Stores (ODS). The ODS will combine customer information from across the complete booking-to-fulfillment lifecycle in a single unified data layer.

The software means British Airways can now implement new business processes that bridge disparate global distribution systems and passenger service systems, enabling it to action sophisticated workflows against end-to-end customer orders. BA said the capability is facilitating “the implementation of new business models and help[ing] the airline accelerate its transformation of the IT function into a business enabler to support key business goals and initiatives.”

British Airways CIO, Paul Coby, described the process changes the company is trying to incorporate into T5. From CIO UK:

Originally designed for the automotive industry, the lean model is popular in manufacturing circles as a process management philosophy that focuses on standardising processes in order to reduce costs and time waste and improve efficiency. T5, according to Coby, has many qualities that are similar to a factory, in that it aims to eliminate waste, simplify and streamline processes, create flow and create a culture of continuous improvement.

BA uses the lean model to assist the carrier to reduce overheads and manage the enormous task of handling business change and the massive integration challenge involved in this large-scale project. Coby explains that lean involves using simple, repeatable processes and only adopting complex technology when there is no alternative.

Training issues. Given the new system's complexity, staff training and preparation was clearly lacking. BA acknowledged the critical importance of training in the kind of business transformation program that T5 represents. From Computing:

Training and familiarisation procedures were carried out for a year so that staff could test the new IT systems in place throughout the building.

“Delivering more than 200 IT projects was easy when compared to the people element of the plan,” said BA’s programme head of information management for T5, Glenn Morgan.

“Getting people to understand the new role of technology and buy into the new proposition was our biggest challenge,” he said. “For some projects, people were asking for new systems to support their work, but we actually changed the process and brought no IT.”

BA blamed the glitches on problems with "staff familiarisation", which had a knock-on effect on bags and flights. A BA spokeswoman said delays in getting staff screened for work and other technical difficulties had caused the airline problems.

Planning, testing, and software quality assurance (QA). British Airways started system testing a year ago. From CIO UK:

BA’s CIO, Paul Coby told CIO UK [in March 2007] the IT work to support such a large-scale, new-build project was also going well. “Devices are deployed, connections are being integrated and 2007 will be testing year. The airline is moving onto the T5 systems, so they run for a year ready to operate at the new terminal when it opens in 2008. This is the year we put the IT infrastructure and systems in place to manage our people and passengers,” he said.

However, BA underestimated the testing, integration, and release planning required to achieve a trouble-free launch. CIO UK described BA's misplaced sense of invincibility:

Almost every new airport has opened late, cost more and had major operational issues, and, according to [Nick Gaines, director of business critical systems and IT at the British Airports Authority], “system integration problems” are often blamed. “This will not be the case for T5. BAA and BA plan working together for system integration over two years, we’ve been focused on bringing systems together. Integration is not just plugging technology together; it’s about people, processes and systems working together.”

In a moment of understatement adding insult to injury, British Airways CEO, Willie Walsh, commented:

We are working hard to tackle the difficulties we have had with the terminal's baggage system. From time to time problems have developed that were not encountered during the extensive trials.

The systems incorporated in T5 severely taxed BA's planning, testing, and deployment capabilities. In an irony perhaps better suited to a BBC sitcom, the British Airports Authority recently solicited for software quality assurance vendors. In February, 2008 ComputerworldUK reported:

Airports authority BAA has begun looking for a service provider to work with it to help develop consistent IT systems at Heathrow and across the authority’s airports. It has put out a tender for a framework agreement to find suppliers able to risk-assess any new systems it develops – and to monitor and maintain those same systems once they are in use.

The airports authority wants any suppliers to maintain software quality by putting in place rigorous development processes and testing procedures prior to systems go-live.

And it said it also wants firms to “conduct risk and integrity analyses of proposed system architectures and provide guidance on assurance issues.”

My take: there's never enough time or budget to test, train, and prepare properly for launch. But after the deluge, when problems have been exposed and the bigwigs embarrassed by failure, time and money magically appear. Geez, isn't that always what seems to happen?