Analysis: IBM and Pennsylvania share blame in massive IT train wreck

Analysis: IBM and Pennsylvania share blame in massive IT train wreck

Summary: A huge IT failure illustrates problems that arise when customer and system integrator do not fulfil their responsibilities and obligations.

SHARE:
22

After significant delays and cost overruns, Pennsylvania has terminated a contract with IBM to provide new software for the state's unemployment compensation system.

IBM and Pennsylvania share blame in massive IT train wreck
Photo credit: Michael Krigsman

The state's Secretary of Labor and Industry, Julia Hearthway, made the decision following an $800,000 independent study [pdf download], conducted by Carnegie Mellon University’s Software Engineering Institute, that blames both IBM and the state. At the time of cancellation, the project was 42 months behind schedule, with a $60 million cost overrun, based on an original budget of $106.9 million.

InfoWorld reports that Hearthway cancelled the project because, "the problems we've identified cannot be solved." IBM said it was "surprised by today's announcement," adding, "In complex information technology implementations, there is accountability on both sides for system performance and service delivery."

The department seemed incredulous that the cancellation surprised IBM: "We've been meeting weekly with them and they know full well all the problems. That's ridiculous that they say they're surprised. They must be in denial."

Untangling the mess

Large IT projects virtually always involve highly technical, labor-intensive activities performed by numerous people and multiple companies with competing priorities. Making matters more complicated, these big budget projects tend to create their own center of gravity in which procurement decisions are politicized and managers must coordinate diverse resources for extended periods.

The Carnegie Mellon report makes clear that this project failed because both sides had lapses in fulfilling their own responsibilities. As the state and IBM wrangle blame in legal proceedings that may take years to resolve, it is apparent that both sides share culpability. The report findings focus on three themes: software quality; procurement and implementation; and program management.

Software quality. The report explains that IBM's software development plan followed "industry and company standards and practices," although the company lacked the "discipline to execute." Among other deficiencies, IBM did not perform sufficient testing and quality assurance, leaving the system at risk for technical failure upon deployment.

The report also faults the state for not providing sufficient oversight on IBM in key areas such as testing. For example, the state "prematurely" accepted software into production that had "known defects impacting performance."

Another example of the state mismanaging its obligation to oversee the details of IBM’s work, the department approved requirements, "without fully understanding what they were approving.” The state's inexperience, combined with IBM's "lack of rigor and discipline," created an environment in which the software could possibly fail if released.

Procurement and implementation. The assessment report identified "four major weaknesses” in the department's purchasing, saying the RFP contained:

(1) unprioritized and often ambiguous requirements, (2) lack of detailed and objective source selection criteria, (3) the lack of specification of detailed... operational system attributes, performance measures, associated metrics, and a risk management framework, and (4) the absence of an explicit risk assessment of bidder plans, schedules, and assumptions.

The incomplete and ambiguous procurement specifications created a poor foundation for the entire project. Poorly defined projects are common when an organization lacks the experience to foresee potential pitfalls.

Although the department should have created better specifications, it appears IBM breached its duties and responsibilities by accepting this poorly defined project. Enterprise buyers engage large system integrators like IBM precisely to bring the experience needed to construct the project properly.

This situation calls IBM’s judgment and fitness to execute into question and raises the important question why they accepted this flawed project in the first place. One can speculate that IBM accepted the project despite being aware of the flaws, assuming that subsequent change orders would cover whatever issues might arise after the project started.

In addition to taking on a project defined ambiguously, the report says IBM made three important and unrealistic planning assumptions that created additional risk:

  1. There would be no legislative or rule changes affecting the project
  2. Unemployment claim demands would remain constant
  3. There was no need to analyze the code, business logic, or database of the old system

Because of these wrong assumptions, the department and IBM "had different expectations and understandings about the planning and execution of the program at contract award." In plain English: this project was screwed from the start.

In summary, Pennsylvania wrote a poor requirements document and IBM accepted a flawed project despite the presence of substantial risks that should have been obvious.

Ineffective governance and program management. Throughout the project, neither the department nor IBM established clear metrics and expected outcomes against which they could measure progress.

In addition, both sides failed to provide the resources needed to handle the work required on this large project. Aside from saying the department failed to "provide the level of DLI [Department of Labor and Industry] staffing assumed in the Contractor's [IBM's] proposal," the report calls out the department's lack of resources:

From the start, DLI was not able to provide adequate resources to staff its planned management and governance approach. Further, DLI made no formal delegation of roles, responsibilities, and authority for management of the program. DLI's approach to managing the UCMS [unemployment compensation] program from contract award to early 2011, led to a situation in which no one in DLI was accountable and responsible for the administration of the program. As a result, there was no effective oversight or timely action to make definitive decisions to mitigate the systemic risks that were continually highlighted by the IV & V contractor.

The report also presents a scathing indictment of IBM's resource management on the project, which started in 2006:

The size and churn within the Contractor's workforce has also contributed to program discontinuity. Since the start of the project, 638 different contractor staff members have worked on the project with the majority of the workforce having less than one year on the project and 75 percent having less than two years.

Regarding governance, the report concludes:

Overall project governance and management were insufficient for the complexity of the effort necessary to achieve project and mission goals.

The leadership challenge

Unfortunately, cases such as this happen all the time. High rates of failure, attest to the complexity and challenges associated with running large IT projects. It is worthwhile noting that IBM is also at the center of a massive IT failure in Australia, in which an Au$6.19 million project somehow required Au$1.2 billion to “develop and operate properly.”

These cases raise an important question regarding relationships among enterprise buyers, software vendors, and system integrators. The collaborative nature of enterprise implementations demands that all parties work together to serve the customer’s best interests. Successful projects rely on a give-and-take in which the parties cooperate in good faith.

However, the collaboration breaks down when the parties succumb to their own pressures, limitations, and self-interested agendas. For example:

  • The customer may not possess sufficient expertise to put together a solid request for proposal, as apparently happened here. In this case, the department effectively pushed significant project ambiguity onto IBM.
  • The system integrator may bid the project low, hoping to get in the door and later create change orders that push up the price. We can only speculate as to IBM’s willingness to accept an ambiguous project, but these things come to mind.
  • In some situations, although not discussed in this case, the software vendor’s sales people may suggest a product that does not adequately fit the customer’s needs.

Each of these illustrative points presents a situation in which conflicting goals and agendas can drive IT failure. Because these dynamics are so prevalent, I have written extensively about the IT Devil’s Triangle, which describes conflicting relationships among customer, software vendor, and services provider.

Perhaps the department and IBM could have salvaged the project had each side met its own responsibilities and obligations. Unfortunately, that did not happen.

Topics: CXO, Enterprise Software, IBM

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

22 comments
Log in or register to join the discussion
  • This is how all the big contractors play the game.

    The rule is to get the contract signed first, and then try to figure out if the project can actually be delivered. I would bet money that IBM outsourced or offshored a significant portion of the work, since there is no way IBM will hire any full time workers in the US, thus the huge churn in personnel, which leads to poor communication, lack of proper handoff to newcomers, and a certain doom to the project as a result. None of this is unique or surprising, and from my personal observation, EDS and others basically do the same thing - bid low, then jack the price through change orders. Nice little racket they have there, but sometimes they get found out, usually after the failures become catastrophic and irreversible.
    thetwonkey
    • Not Just the Big Ones

      It's not just the big contractors that do this. I worked for a small firm that would bid first and then figure out if they actually had the time and resources to complete it. The usual next step involved an announcement from management that the project MUST BE COMPLETED at all costs even though it was unrealistic. Bonuses for sales & management for landing the sweet contract and exhaustion and burnout for the ones actually doing the work.
      LimeGreen416
    • IBM doesn't need to outsource...

      ...they have sufficient capacity to offshore it within IBM.
      jaykayess
  • 42 months behind schedule

    Sounds like a typical IBM project.
    LBiege
  • The Glove Fits in this case

    This definitely fits with every experience I've had with IBM; even the blaming the customer for the problems part. Sure customers can and do sometimes derail a project but too often it's just an excuse foisted by IBM who as a rule never ever takes any responsibility. People should be getting a clue that yes you can make a mistake in going with IBM and yes people do often lose their jobs because of it although usually there will be a lot of collateral damage to others first.
    The IT industry has been ruined by politics, manipulators, and BSers masquerading as IT professionals, IT Project Managers etc. many of which work for the likes of IBM. IBM was already kicked out if Indiana, Florida, and others and BANNED from Queensland. Why anyone would sign a contract with them is beyond comprehension.
    greywolf7
    • This happens with ALL vendors...

      ...large, medium and small. Accenture, HP, IBM, Capgemini, PWC, Atos, Tata, Wipro, Infosys, HCL... You name it, they have all had their own share of failed projects. It's par for the course. It's like the credit card business, where a certain percentage of fraud is factored into the cost of doing business. Similarly, a risk margin is built into every project cost... it's just that some projects need to invoke the margin and most don't.
      jaykayess
  • Just buy a Windows Server

    Setup it, connect some Windows 7 machines with Office and be done with it. If you need some cloud, ask Microsoft be done! Geez, we all know at the end of the day, most of these state employee intense work schedules involve 1 page documents in Word and lots of solitaire with majority of time Facebook and Twitter.
    adacosta38
    • That's it

      I am always puzzled when I see projects like that. It all seems to be over complicated and overpriced, when all the people need is a shared calendar and a simple way to share documents. Since you couldn't charge millions for that, they need to figure out a way to make it more complex. They include smoke and mirrors and suddenly you deal with a $100 million project.
      dcdavy
    • Thats just being stupid.

      None of the work would be done.
      jessepollard
  • That's a Boston crash scene

    That's a Boston crash scene, looks like Commonwealth Ave. near Boston University.
    Just sayin'
    Reality-based
  • It is the environment that creates these projects that is at fault

    In the article they stated no one had analyzed the old system, or studied the future needs. Both of these are projects in their own right, that could have led to a solid specification. More importantly, depending on what was found, the design potentially could have been an incremental approach of a number of smaller and easier managed contracts that would have upgraded the original system.

    The real problem is that the cost (time, money and effort) of getting a project through a govement agency and legislature does not scale with the size of the project. This drives people to these huge projects that are bound to fail.
    oldsysprog
    • Bingo.

      Very likely the government never had any idea of what they were asking for in the first place, then wouldn't budget time + money to find out what it was doing, then gave a bogus RFP. And IBM walked into a trap where the accepted proposal had nearly nothing to do with what was needed.
      jessepollard
  • Governance and Accountability

    Being a former employee of the Commonwealth's Office of Administration, I noticed many deficiencies in Governance and especially accountability. Many managers were more concerned about their "smoke" breaks than managing projects.
    weatherwarrior
  • how many H1B's on the project?

    When I was collecting unemployment in PA after getting replaced by an H1B, I thought the existing web based system worked great. Can't even imagine why they needed to replace it.

    The article didn't mention how many H1B's and L1's were on the project.
    blurkosphere
  • This is a common issue with governemnt accounts

    People in government seldom are held accountable for their errors, overspending, or lack of doing their job. And there are incentives to allow budget overruns, while in private industry the buyer does what they can to keep the costs down.

    Thus, I'm not surprised that "Pennsylvania wrote a poor requirements document." I'm not surprised that IBM bid for the business, knowing that, and seems to me IBM's assumptions were based on the fact they were missing from the requirements. Their assumption that these would be sorted out and specified later, with a potential revision to the contract, seems normal to me as well. Why shouldn't IBM assume that "There would be no legislative or rule changes affecting the project" when it's not specified in the contract and they are dealing with the government who'll be making those changes? Isn't it the government's responsibility to specify the amount of unemployment claims to process in the contract?

    Given the government has announced it's spending money on an IT project, to me it's their responsibility to start with a good requirements document. Lacking a good requirements document, is just a means for government to pass the blame to a vendor it hired with essentially an open ended undefined contract. Perhaps they planned it that way, because now they get to do it again and that means more spending on government IT budgets and salaries. Government often doesn't care about wasting taxpayers' money. It's not out of their pocket, and instead they get more work and more pay for mismanagement.

    If IBM didn't take the opportunity because the requirements weren't clear, then someone else would have. The blame resides with the government who initiated the project and has hired another contractor for $800,000 to pass the blame.
    MoreFreedom
  • Train wreak

    Looks like what happened to Windows 8.
    dcowsky@...
  • It's Not The First Time This Has Happened With IBM

    What you are describing sounds almost identical to what occurred here in Indiana in 2009 involving the Department of Public Welfare (see below). You would think that they would have learned something from their prior experience(s).

    http://www.inthepublicinterest.org/article/indiana-axes-welfare-contract-ibm
    JakeRader
  • This is a tale of two stories

    The first is the failure story.

    But second, is a story about what happens when business people think that the brand is an independent concept separate from the quality, value, and experience by the customer. The summary report by SEI pointed out the discontinuity in IBM’s implementation team from high turnover including key project management roles, in fact the report uses the term "churn."

    What business decisions within IBM drove this churn in team staffing? Were they running so lean that they moved people to other projects? Were they trying to lower their internal labor rates by moving to less experienced people and thought the project would not suffer?
    elizab
  • All too familiar

    Not with just IBM but with any large software/integration project. Too often public sector doesn't have/can't afford the expertise so they lean on Big names like IBM, Pearson, Microsoft. The big names are so sure of their expertise they rarely take the time to learn the individual nuances of the public sector projects, and try to cookie cutter a for-profit solution for them.

    It takes give and take, and alot of hard work to make an implementation like this succeed. And far too often, both sides don't count that in to the true costs. They then handicap their own success by trying to shave costs and time at the end.

    Too bad for PA. Their current L&I system definitely could use a rehab.
    mpompey0110
  • Know the facts

    For the UCMS project, PA, like many states, did not allow an off-shore or out-source model. However, IBM did implement UCMS releases 1, 2 and 3 with a contractor model, using fewer and fewer actual IBM employees.
    tedmc@...