The billion dollar web site you paid for

Summary:Whoever heard of a pure IT project that cost a billion dollars to build (so far)? A GAO investigation goes deep into just how bad the process of building HealthCare.gov was.

healthcare-620x388

Perhaps no news about HealthCare.gov, the Federal healthcare exchange website and supporting systems, is shocking anymore. We all know that it was an utter disaster at launch on October 1, 2013 and was completely unusable for some time thereafter. But eventually they got it to the point of being usable, so no harm no foul, right?

You may not think so after reading the recent GAO (Government Accountability Office) report HEALTHCARE.GOV — Ineffective Planning and Oversight Practices Underscore the Need for Improved Contract Management. The report is embedded at the bottom of this story.

Special Feature

Cloud Computing: Moving to IaaS

Infrastructure as a Service providers make a very compelling argument for businesses to stop running their own data centers and simply purchase server capacity on-demand and scale up and down as needed. This is our deep dive on IaaS strategy and best practices

Not only was the project a technical disaster — development was originally supposed to be complete October 1, 2013, but the schedule is now for the end of 2014 — but it has cost far, far beyond what was budgeted and far further than what could be called reasonable for such a system.

The report says (page 9) that, through March 2014, the total cost of the project was $946 million. $840 million of this was spent by the CMS (Centers for Medicare and Medicaid Services), with the rest by the IRS and Department of Veterans Affairs. But the development costs continue to rise and are likely already over $1 billion.

Clearly CMS was put in a bad spot having to build a major first-of-its-kind system in a compressed time frame. One implication of this was that the bidding process had to proceed without completed specifications. CMS made many risky decisions in order to meet their goals, such as the use of "cost-plus-fixed-fee" contracts in the bid process and an Agile software development model, which was new to CMS. As the report notes (footnote 23), in 2009 the Office of Management and Budget released a Memorandum (M-09-25) calling for a reduction in the use of such high-risk contracts.

A theme pervades the report: These decisions might have been reasonable, but the risks they created increased the requirements for oversight. The report finds that the agency failed utterly in its oversight responsibilities. Over and over, procedures called for the creation of quality assurance surveillance and other oversight mechanisms, but CMS did not do so. The result was huge cost overruns, the main potential downside of cost-plus-fixed-fee contracts.

The report doesn't go on to the next logical question, whether more senior HHS and Administration officials were exercising any oversight of the process, but the answer would appear to be that they were not, and there certainly is no evidence that they were. The surprise of everyone to the site's miserable initial performance indicates that senior HHS managers and the White House were unaware.

CMS's work consisted of two main projects: the FFM (Federally Facilitated Marketplace) and the data hub. The FFM accepts and processes data entered through HealthCare.gov and was intended to provide 1) eligibility and enrollment, plan management and financial management. The data hub "...routes and verifies information among the FFM and external data sources, including other federal and state sources of information and issuers. For example, the data hub confirms an applicant's Social Security number with the Social Security Administration and connects to the Department of Homeland Security to assess the applicant's citizenship or immigration status." See page six of the GAO report for more expansive definitions of these projects.

The oversight failings made it possible for failures in development to go unaddressed. Why did development fail? One reason, if not the top reason, was CMS's changing of requirements throughout the process. The following quote summarizes many of the systemic failures in oversight and management and their implications:

From September 2011 to February 2014, estimated costs for developing the FFM increased from an initial obligation of $56 million to more than $209 million; similarly, data hub costs increased from an obligation of $30 million to almost $85 million. New and changing requirements drove cost increases during the first year of development, while the complexity of the system and rework resulting from changing CMS decisions added to FFM costs in the second year. In addition, required design and readiness governance reviews were either delayed or held without complete information and CMS did not receive required approvals. Furthermore, inconsistent contractor oversight within the program office and unclear roles and responsibilities led CMS program staff to inappropriately authorize contractors to expend funds.
fig4
Figure 4 from page 20 of the study. Unsurprisingly, the panic payments accelerated as the October 1 deadline approached. Yes, the popup text says "definitize".

Of course, throwing people and money at an IT project tends to make things worse, not better. And it's almost always a better idea to delay the rollout of a project than to launch with significant problems. But a launch delay was politically impossible, no matter how badly the project was going. The law said it would launch on October 1, so it had to launch on October 1.

But even though the launch date was fixed, the problems in the project necessitated schedule changes. As Figure 5 from the study, included below, shows, the Requirements, Analysis and Design stage of the project went from the originally scheduled three months to a year, which they did mostly by cutting out features of the system which were not essential to the launch, such as the Financial Management system that sent payments to the insurers. Indeed, this part of the system is still not complete and, according to the report, "... is currently scheduled to be implemented in increments from June through December 2014."

fig5

Cutting features cut the Development and Test stage from nine months to six, and the Operational Readiness Review from seven months to one. Yes, one. They reserved enough testing time to realize just how bad things were, and then they launched anyway. No IT project can succeed this way.

Given how pathetic the government management of the project was, I'm inclined to be somewhat sympathetic to the contractors, who were in an impossible position. That would be naive, as government contractors are often in the business of putting themselves in impossible positions, figuring that cost overruns will more than make up the difference. It's hard to work up any sympathy for CGI Federal, the main contractor for the FFM, and their hundreds of millions of dollars.

On CNET

Back-to-School Tech Gift Guide

Whether you're shopping for a student or are looking to update your own tech lineup, our back-to-school guide will show you something worth your time and money.

Even so, the GAO report says that CMS "identified significant FFM contractor performance issues as the October 1 deadline approached" (i.e., problems that were the contractor's fault), but decided to let them slide. It wasn't until December, when the you-know-what had already hit the fan, that CMS began withholding payment to CGI Federal. In January CMS announced that Accenture Federal Services would replace CGI Federal on that contract.

In retrospect, it would have been politically impossible to dismiss or discipline CGI Federal severely in June 2013 when, says the report, CMS grew increasingly concerned with their performance. CMS even sent a letter in August listing the problems and suggesting that they would take corrective action, but the letter was quickly withdrawn at the order of CMS Chief Operating Officer Michelle Snyder (who fell on her sword shortly after the rollout).

The report made clear that CMS was well aware of what poor shape the site was in at launch, and yet the news of it did not leak out. If only the government were as good at keeping national security secrets. It's clear that nothing was going to stop the October 1 rollout.

The Accenture contract to take over the FFM development project was a one-year, sole source contract for $91 million for one year, and even that contract has exploded. As of June 5, CMS had obligated more than $175 million to the Accenture FFM contract.

The conclusion the GAO draws is that the organizational and process decisions made by CMS are still flawed and the problems remain. Ominously, they conclude "[u]nless CMS takes action to improve acquisition oversight, adhere to a structured governance process, and enhance other aspects of contract management, significant risks remain that upcoming open enrollment periods could encounter challenges going forward." Will the next open enrollment be as disastrous as the first? We'll know by October.

Things may be better this year, as the administration brought in someone from the outside world late in 2013 to try and make some lemonade out of HealthCare.gov. Mikey Dickerson, an operational engineer hired away from Google, didn't like what he saw when he got to Columbia, Md., headquarters for HealthCare.gov: "The government had none of the modern tools to track, second by second, visitors to the website. And it had no way to figure out why the site was crashing." The addition of tools to address these concerns certainly accounts for some of the runup in costs in 2014.

But even if the system were complete and working well, it still cost a billion dollars. I asked a few people familiar with the development of large, complicated internet systems and they all said a billion dollars is a ridiculous amount, even including the fact that hardware purchases were involved. Nobody would go on the record.

I expect government to do a bad job in general, and I'm not surprised that it's bad at building IT systems. What disappoints me is the lack of appreciation of just how bad a job the administration, an administration once reputed to be "tech savvy," did on their most prominent project. A couple of CMS officials were allowed to resign and the contractor was replaced (after taking in hundreds of millions of dollars), but I'd say nobody has really paid a price for the HealthCare.gov debacle — other than the taxpayers who paid for it.

GAO Report: HEALTHCARE.GOV - Ineffective Planning and Oversight Practices Underscore the Need for Improved...

Topics: Government, Government : US, Health

About

Larry Seltzer has long been a recognized expert in technology, with a focus on mobile technology and security in recent years. He was most recently Editorial Director of BYTE, Dark Reading and Network Computing at UBM Tech. Prior to that he spent over a decade consulting and writing on technology subjects, primarily in the area of sec... Full Bio

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.