IT sizing error delays Boston commuters
The automated machines that dispense subway tickets in Boston went down for two hours during a recent rush hour. This situation offers an example of IT failure in an unusual setting.
According to the Boston Globe:
The breakdown affected Charlie Card fare-dispensing machines that take credit and debit cards. Between 7:40 and 9:24 a.m., the machines "started experiencing intermittent disruptions of service" that typically lasted between five and seven minutes, but worked fine after that, T spokesman Joe Pesaturo said.
[Scheidt & Bachmann, the manufacturer, is] going to come back to us with a corrective action plan," Pesaturo said, which could include beefing up computer processing power to handle a crush of monthly pass and fare sales like yesterday's.
THE PROJECT FAILURES ANALYSIS
Automated fare systems are designed to help passengers quickly purchase tickets with minimum delay. This user simplicity is matched by back-end complexity, as shown below in the manufacturer's system diagram:
IT deployments of this type require proper sizing to determine hardware requirements, based on anticipated transaction levels, system requirements, and so on. If the system isn't properly sized, outages can occur during times of peak demand.
Given the manufacturer's proposal to add processing power, it seems likely the hardware was not adequately sized during the initial deployment. Since Boston subway ridership levels are carefully tracked, and the backend system requirements are well-known by the manufacturer, this failure should not have occurred.
From my perspective, there are two possible explanations:
- Ridership spiked beyond any reasonable expectations, causing hardware overload. Given the available historical data, this scenario seems unlikely.
- The implementation was not planned or managed properly. I must conclude this is the likely cause of the failure.
How and why a technical exercise like sizing could be so screwed up is hard to say. Perhaps Boston wanted to save money on hardware, hoping for best. Regardless of the reasons, we can definitely say this is another government IT project gone awry.