The billion dollar web site you paid for

The billion dollar web site you paid for

Summary: Whoever heard of a pure IT project that cost a billion dollars to build (so far)? A GAO investigation goes deep into just how bad the process of building HealthCare.gov was.

SHARE:
healthcare-620x388

Perhaps no news about HealthCare.gov, the Federal healthcare exchange website and supporting systems, is shocking anymore. We all know that it was an utter disaster at launch on October 1, 2013 and was completely unusable for some time thereafter. But eventually they got it to the point of being usable, so no harm no foul, right?

You may not think so after reading the recent GAO (Government Accountability Office) report HEALTHCARE.GOV — Ineffective Planning and Oversight Practices Underscore the Need for Improved Contract Management. The report is embedded at the bottom of this story.

Special Feature

Cloud Computing: Moving to IaaS

Cloud Computing: Moving to IaaS

Infrastructure as a Service providers make a very compelling argument for businesses to stop running their own data centers and simply purchase server capacity on-demand and scale up and down as needed. This is our deep dive on IaaS strategy and best practices

Not only was the project a technical disaster — development was originally supposed to be complete October 1, 2013, but the schedule is now for the end of 2014 — but it has cost far, far beyond what was budgeted and far further than what could be called reasonable for such a system.

The report says (page 9) that, through March 2014, the total cost of the project was $946 million. $840 million of this was spent by the CMS (Centers for Medicare and Medicaid Services), with the rest by the IRS and Department of Veterans Affairs. But the development costs continue to rise and are likely already over $1 billion.

Clearly CMS was put in a bad spot having to build a major first-of-its-kind system in a compressed time frame. One implication of this was that the bidding process had to proceed without completed specifications. CMS made many risky decisions in order to meet their goals, such as the use of "cost-plus-fixed-fee" contracts in the bid process and an Agile software development model, which was new to CMS. As the report notes (footnote 23), in 2009 the Office of Management and Budget released a Memorandum (M-09-25) calling for a reduction in the use of such high-risk contracts.

A theme pervades the report: These decisions might have been reasonable, but the risks they created increased the requirements for oversight. The report finds that the agency failed utterly in its oversight responsibilities. Over and over, procedures called for the creation of quality assurance surveillance and other oversight mechanisms, but CMS did not do so. The result was huge cost overruns, the main potential downside of cost-plus-fixed-fee contracts.

The report doesn't go on to the next logical question, whether more senior HHS and Administration officials were exercising any oversight of the process, but the answer would appear to be that they were not, and there certainly is no evidence that they were. The surprise of everyone to the site's miserable initial performance indicates that senior HHS managers and the White House were unaware.

CMS's work consisted of two main projects: the FFM (Federally Facilitated Marketplace) and the data hub. The FFM accepts and processes data entered through HealthCare.gov and was intended to provide 1) eligibility and enrollment, plan management and financial management. The data hub "...routes and verifies information among the FFM and external data sources, including other federal and state sources of information and issuers. For example, the data hub confirms an applicant's Social Security number with the Social Security Administration and connects to the Department of Homeland Security to assess the applicant's citizenship or immigration status." See page six of the GAO report for more expansive definitions of these projects.

The oversight failings made it possible for failures in development to go unaddressed. Why did development fail? One reason, if not the top reason, was CMS's changing of requirements throughout the process. The following quote summarizes many of the systemic failures in oversight and management and their implications:

From September 2011 to February 2014, estimated costs for developing the FFM increased from an initial obligation of $56 million to more than $209 million; similarly, data hub costs increased from an obligation of $30 million to almost $85 million. New and changing requirements drove cost increases during the first year of development, while the complexity of the system and rework resulting from changing CMS decisions added to FFM costs in the second year. In addition, required design and readiness governance reviews were either delayed or held without complete information and CMS did not receive required approvals. Furthermore, inconsistent contractor oversight within the program office and unclear roles and responsibilities led CMS program staff to inappropriately authorize contractors to expend funds.
fig4
Figure 4 from page 20 of the study. Unsurprisingly, the panic payments accelerated as the October 1 deadline approached. Yes, the popup text says "definitize".

Of course, throwing people and money at an IT project tends to make things worse, not better. And it's almost always a better idea to delay the rollout of a project than to launch with significant problems. But a launch delay was politically impossible, no matter how badly the project was going. The law said it would launch on October 1, so it had to launch on October 1.

But even though the launch date was fixed, the problems in the project necessitated schedule changes. As Figure 5 from the study, included below, shows, the Requirements, Analysis and Design stage of the project went from the originally scheduled three months to a year, which they did mostly by cutting out features of the system which were not essential to the launch, such as the Financial Management system that sent payments to the insurers. Indeed, this part of the system is still not complete and, according to the report, "... is currently scheduled to be implemented in increments from June through December 2014."

fig5

Cutting features cut the Development and Test stage from nine months to six, and the Operational Readiness Review from seven months to one. Yes, one. They reserved enough testing time to realize just how bad things were, and then they launched anyway. No IT project can succeed this way.

Given how pathetic the government management of the project was, I'm inclined to be somewhat sympathetic to the contractors, who were in an impossible position. That would be naive, as government contractors are often in the business of putting themselves in impossible positions, figuring that cost overruns will more than make up the difference. It's hard to work up any sympathy for CGI Federal, the main contractor for the FFM, and their hundreds of millions of dollars.

On CNET

Back-to-School Tech Gift Guide

Back-to-School Tech Gift Guide

Whether you're shopping for a student or are looking to update your own tech lineup, our back-to-school guide will show you something worth your time and money.

Even so, the GAO report says that CMS "identified significant FFM contractor performance issues as the October 1 deadline approached" (i.e., problems that were the contractor's fault), but decided to let them slide. It wasn't until December, when the you-know-what had already hit the fan, that CMS began withholding payment to CGI Federal. In January CMS announced that Accenture Federal Services would replace CGI Federal on that contract.

In retrospect, it would have been politically impossible to dismiss or discipline CGI Federal severely in June 2013 when, says the report, CMS grew increasingly concerned with their performance. CMS even sent a letter in August listing the problems and suggesting that they would take corrective action, but the letter was quickly withdrawn at the order of CMS Chief Operating Officer Michelle Snyder (who fell on her sword shortly after the rollout).

The report made clear that CMS was well aware of what poor shape the site was in at launch, and yet the news of it did not leak out. If only the government were as good at keeping national security secrets. It's clear that nothing was going to stop the October 1 rollout.

The Accenture contract to take over the FFM development project was a one-year, sole source contract for $91 million for one year, and even that contract has exploded. As of June 5, CMS had obligated more than $175 million to the Accenture FFM contract.

The conclusion the GAO draws is that the organizational and process decisions made by CMS are still flawed and the problems remain. Ominously, they conclude "[u]nless CMS takes action to improve acquisition oversight, adhere to a structured governance process, and enhance other aspects of contract management, significant risks remain that upcoming open enrollment periods could encounter challenges going forward." Will the next open enrollment be as disastrous as the first? We'll know by October.

Things may be better this year, as the administration brought in someone from the outside world late in 2013 to try and make some lemonade out of HealthCare.gov. Mikey Dickerson, an operational engineer hired away from Google, didn't like what he saw when he got to Columbia, Md., headquarters for HealthCare.gov: "The government had none of the modern tools to track, second by second, visitors to the website. And it had no way to figure out why the site was crashing." The addition of tools to address these concerns certainly accounts for some of the runup in costs in 2014.

But even if the system were complete and working well, it still cost a billion dollars. I asked a few people familiar with the development of large, complicated internet systems and they all said a billion dollars is a ridiculous amount, even including the fact that hardware purchases were involved. Nobody would go on the record.

I expect government to do a bad job in general, and I'm not surprised that it's bad at building IT systems. What disappoints me is the lack of appreciation of just how bad a job the administration, an administration once reputed to be "tech savvy," did on their most prominent project. A couple of CMS officials were allowed to resign and the contractor was replaced (after taking in hundreds of millions of dollars), but I'd say nobody has really paid a price for the HealthCare.gov debacle — other than the taxpayers who paid for it.

GAO Report: HEALTHCARE.GOV - Ineffective Planning and Oversight Practices Underscore the Need for Improved...

Topics: Government, Government US, Health

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

85 comments
Log in or register to join the discussion
  • They should have just gone to Google.

    I use Linux and run almost everything Google makes, including their Public DNS.

    They have the backbone to run it properly, with over a million servvers and their data centers and load balancing are state of the art.

    As far as the code, Google most likeley would have an upper hand there also.


    Thank God they use Linux and not Microsoft, the security is there, that's the most important aspect.



    toolbar.netcraft.com/site_report?url=undefined#last_reboot
    Joe.Smetona
    • Correct Netcraft Link.

      http://toolbar.netcraft.com/site_report?url=healthcare.gov#last_reboot
      Joe.Smetona
      • Google reluctently agreed to sign a BAA

        meaning even Google has doubts or lack of confidence their offerings would maintain HIPAA compliance.

        They did so only because MS had always agreed to sign a BAA, as they were knew their offering did.

        I know, - just another fact you didn't want anyone to know.....
        William.Farrel
        • How's your Obama care working out, Joe?

          Eh?
          Raid6
        • Please explain

          I'd like to understand your point. What is a BAA and why is it important, in the context of the Healthcare.gov website?
          maria.tseng@...
    • Joe, as always, thanks for the entertainment

      Joe.Smetona wrote:
      "Thank God they use Linux and not Microsoft, the security is there, that's the most important aspect."

      Security is much more a function of the knowledge, skills and abilities of the system administrators and developers than it is the server operating system platform. There is no lack of articles at ZDNet which confirm this.

      Joe.Smetona also wrote:
      "They should have just gone to Google ... including their Public DNS"

      All I can say in response to this jewel is that privacy is of the utmost importance in health care. Have you heard of HIPPA? There are qualified companies with much better privacy records than Google (and here I don't include Facebook).
      Rabid Howler Monkey
      • "...privacy is of the utmost importance in health care..."

        Well, that leaves out anything from MS.

        Google has a much better privacy record than anyone else.

        Yes, they collect what you give. They collect what is offered for free.

        But you don't hear of them being broken into anywhere NEAR as much as anything from Microsoft.
        jessepollard
        • Really????

          You are talking about Google....as in the search giant that makes 96% of its revenue selling information?

          Please link us to the many or any break in's at Microsoft that amounted to information being lost????????????????????????????

          I am no fan of Microsoft but they actually sell products and not Information.....your information.
          ctopher5669
        • What are you talking about?

          Don't recall any major breaches with Azure as of late nor any with on prem Windows Server 2012 RU2.

          With that said, Google adheres to B2B contracts and government regulations just like Microsoft, IBM, and Amazon with their cloud services. The problem with Google is that they are not really yet a player in cloud services, entering in the segment in November this year.
          Rann Xeroxx
        • Of course not Jesse. You think Google would tell anyone?

          You're in their offices enough (I would imagine) and in all that time, the sentence "breeched our servers" was never uttered?

          Or only in whispers...
          William.Farrel
    • No Joe, no.

      Joe, I won't even go into why you are incorrect but look it up and look at Howler Monkey's reply. I wish people would stop posting misleading information.
      bwebmasta
      • He's likely paid to. He's taken over Deitrick Schmidt's place

        as Linux cheerleader and story teller.

        every time he posts, his "Linux computers used effortlessly at home" count goes up exponentially...
        William.Farrel
    • No thanks

      They need to go to a company that does not have a business model of spying and selling information. Linux is also not any more secure. That's fanboy nonsense.
      Buster Friendly
    • Yeah

      Google would have done it for free as long as you let them catalog all of that personal information.

      No thanks.
      ctopher5669
    • Did they hire Michelle Obama's friend's company?

      I read they didn't even put out a request for bids. They just hired a Canadian company that happens to be the company of one of Michelle Obama's friends. If that story is true it needs to be investigated. How about at least hiring Americans for government projects?
      BrianC6234
      • How does one arrange an impartial investigation?

        The Republicans would be trying to hang the Administration (preferably the President personally) regardless of the evidence; and the Democrats would try to exonerate it, regardless of the evidence. A reasonably fair Congressional investigation was possible in the 1970s, and even the 1980s, but almost any member of Congress who tried to be impartial now probably wouldn't be renominated, and might even lose his other committee assignments. I long figured that the ideal Congressional investigative committee would be headed by a conservative Democrat and a liberal Republican, but both are now in very short supply.

        The GAO is probably as impartial an investigative body as one's likely to find at the federal level nowadays.
        John L. Ries
        • Federal Inspector Generals (IGs) of the U.S. Department and Agencies

          should also be expected to be independent in their investigations. The role of an IG is to "detect and prevent fraud, waste, abuse, and violations of law and to promote economy, efficiency and effectiveness in the operations of the Federal Government".

          Here's the list of U.S. government IGs:

          "Inspectors General Directory & Homepage Links"
          http://www.ignet.gov/igs/homepage1.html

          The U.S. HHS IG was tasked (by the HHS Secretary) in December, 2013, to "to review the development of HealthCare.gov":

          "Building On Our Progress and Moving Forward: Three Initial Steps"
          http://www.hhs.gov/healthcare/facts/blog/2013/12/three-initial-steps.html

          So, where's the HHS IG's report on HealthCare.gov? The answer, the work is in-progress:

          http://oig.hhs.gov/reports-and-publications/archives/workplan/2014/work-plan-2014.pdf

          Search for 'HealthCare.gov' in the 2014 Work Plan and you will find "Contracts Planning, Acquisition, Contracting, Management, and Performance".
          Rabid Howler Monkey
          • IGs should be independent...

            ...but last I checked, they served at the pleasure of the President, which definitely makes them part of the administration, rather than independent of it. My own preference would be that they be appointed for non-renewable fixed terms (seven years is probably long enough) and then pensioned off as the most likely way to insure the appropriate distance between the IGs and the management of the agencies they serve.
            John L. Ries
      • re: Did they hire Michelle Obama's friend's company?

        You damn betcha.

        Indeed they did, on a no-bid contract. It was purely a means to enrich their friends and having a working website at the end was, at best, a secondary consideration. Of course Obamacare and all the previous gov't interventions in the marketplace have been a steadily worsening joke for decades, so why shouldn't the Obamacare website be a bad joke. We wouldn't want it to feel left out, would we?
        rocket ride
  • One of the billion reasons . . .

    . . . why we need government as far away from health care as humanly possible. Every problem with the US healthcare system that Obamacare was allegedly intended to solve was *caused* by government intervention in health care over the last 70 years.

    We need a Separation of Health Care and State. Not because the government is unbelievably inefficient, which it is, but because the provision of health care is not a function of government.
    FDanconia