Understanding Marin County's $30 million ERP failure

Understanding Marin County's $30 million ERP failure

Summary: Marin County voted to stop an ongoing SAP project, implicitly accepting that it wasted over $30 million on software and related implementation services from Deloitte Consulting.

SHARE:
102

The Board of Supervisors in California's Marin County voted to stop an ongoing SAP project and seek a replacement solution, implicitly accepting that it wasted over $30 million on software and related implementation services from Deloitte Consulting.

The dramatic decision to replace SAP comes after relations between Marin and Deloitte Consulting, the implementation consultant on this project, deteriorated to the breaking point. Marin filed a lawsuit against Deloitte "seeking actual and compensatory damages of at least $30 million, along with unspecified punitive and, or exemplary damages and interest." Deloitte has countersued Marin for approximately $550,000 in unpaid fees and late charges.

Related: Marin County sues Deloitte: Alleges fraud on SAP project

The Contra Costa Times described the situation:

After spending more than $30 million on a fancy computer system that never worked well, top Civic Center officials have concluded the program should be scrapped as soon as a more efficient system can be phased in.

The Marin Independent Journal offered historical context:

The Board of Supervisors bought the SAP system in 2005, then paid mounting consultant bills as staff grappled with a fiscal nightmare in which the program could not produce basic financial reports. SAP never performed as promised, bobbling the payroll, accounts receivable and other routine financial functions. A grand jury probe concluded it cost taxpayers $28.6 million as of April 2009

Computer World brought in SAP's perspective:

SAP "strongly believes in the value and performance of its software in use in Marin County," said spokesman Andy Kendzie. "Our software works exactly as it should, and any issues in this implementation in no way reflect on SAP. Our software is installed and functioning perfectly in tens of thousands of public sector agencies, including dozens in California."

Marin's decision to jettison SAP and start over with new software is based on an analysis performed by Phoenix Business Consulting, a firm that Marin hired at a cost "not to exceed $50,000." The statement of work is relatively vague and uninformative:

To review the SAP program currently implemented at the County of Marin and analyze whether it meets the needs of the County, including whether it was properly implemented by Deloitte. Review what additional costs and efforts are necessary for SAP to properly function to meet the needs of the County Departments. Review what efforts were taken to fix the SAP problems which occurred post-implementation.

STRATEGIC ANALYSIS

This situation offers a clear example of how mismatched expectations can drive poor communication and lead to complete failure. To help understand what really went wrong, here is a brief description of each party's role in contributing to this situation:

Marin role. In my opinion, having reviewed substantial documentation, Marin's decision to replace SAP seems intended primarily to strengthen its lawsuit position and push all accountability away from itself. Marin's position is extreme and not credible.

Marin's apparent lack of organizational and governance maturity, and its inability to absorb business transformation changes associated with this implementation, seem to be a basic driver underlying this failure.

A summary status report compiled by Marin's Information Services & Technology Department acknowledges significant lessons learned on this project that it plans to apply on a future ERP implementation:

At this stage, staff is recommending that we look at other system options and recommend the following approach:

  1. An incremental, phased approach to the replacement of SAP, rather than the "big bang" approach that was intentionally followed in the SAP implementation, as was advised by the outside consultants;
  2. IST involvement up front to guide the steering committee of employees from key departments in recommending a system and leading its implementation;
  3. Less reliance on outside consultants and more on County staff, who have a greater, vested interest in the outcome and success of the implementation; and
  4. Routine communication with employees, the public and Board of Supervisors in the form of regular status reports and meetings, including meetings with an oversight subcommittee of the Board for the project.

A town supervisor also emphasized the importance of lessons learned during a meeting in which the Board voted to stop the project (video).

Deloitte role. Most emphatically, I believe Deloitte shares equal, or perhaps even greater, culpability in creating this situation. Deloitte's posturing and unwillingness to accept even partial responsibility for the failure appears inconsistent with the facts. Deloitte seems focused on compensation arising from the implementation process itself, without regard to whether the client achieved successful results or outcomes.

Deloitte's position and actions stand in stark contrast to these customer-centric words from its website:

For some, the path to value is a steady march. For others, it unfolds through bursts of innovation. But for the best of the best, it’s always both. Disciplined improvement initiatives, marked by powerful leaps and breakthroughs.

SAP role. Neither Marin nor Deloitte have suggested that SAP software contributed in any way to the failure. However, a complete analysis must consider the role of SAP in setting client expectations regarding demands imposed by the implementation process.

The extent to which software vendors and integrators should take responsibility for decisions that buyers make during the purchasing process is a difficult question. As I have written, the solution lies in three areas:

  1. Enterprise customers should be more careful assessing their own capabilities before undertaking any complicated organizational change initiatives, including implementing ERP systems.
  2. Systems integrators and consulting firms must be more straightforward in explaining pitfalls and success requirements to potential clients. Some consultants paint an overly positive picture during the sales process; such nonsense has to stop.
  3. Software vendors should tie a greater percentage of sales compensation to ultimate customer satisfaction.

I have referred to relationships among enterprise customers, software vendors, and system integrators as constituting a "Devil's Triangle."

For more information on the Devil's Triangle see:

Exploring the Devil’s Triangle The IT failures blame game (part 1) The IT failures blame game (part 2) ‘Pain chains’ and the IT Devil’s Triangle

Understanding the numbers. The county's decision to throw away its $30 million investment bears scrutiny, especially during this period of budget cuts and economic difficulty.

To understand more, I spoke at length with Mark F. O’Connor, CEO and co-founder of Monadnock Research, which offers advice to enterprise customers on working with services firms. In an excellent analytic writeup, Mark says:

Marin may want to step back and take a closer look at the findings of the report it is using to support that conclusion...Analysis of those findings, however, appears to offer clues into apparent unrealistic expectations similar to what got Marin into its current predicament.

I asked Mark to elaborate this view:

Opponents to the SAP-based system implemented at Marin County are clearly aligned around getting rid of it as quickly as possible. But it appears the County may never have accepted responsibility for making it work in the first place. In fact, my reading of the agreement with Deloitte seems to indicate the County was trying to cede its responsibilities to Deloitte.

Responsibility without authority, however, always yields outcomes similar to what we see here in complex systems projects. There are literally thousands of important decisions that need be made by client staff during an implementation—while they continue to do their day jobs. I would caution against pre-deciding what will happen with SAP at this point, before the County’s alternatives are fully vetted. While there is political consensus to move off SAP, there are workflow and human factors to consider, and a lack of consensus across all departments as to whether they should initiate another disruptive systems project so soon.

Additionally, the financial justification that appeared persuasive to the Supervisors was remarkable to me only in its lack of detail. The County needs to ensure that it doesn’t make decisions to improve its position in the Deloitte lawsuit at the expense of its financial interests and ability to effectively provide services to Marin County residents.

I would hope Marin's analysis and assessment of the various alternatives includes addressing these twelve key questions:

  1. What are the estimated expenses for hard and soft dollar costs associated with preparing the five plans the Board of Supervisors just authorized?
  2. What is the estimated cost for managing the alternative assessment, selection, and implementation process?
  3. What are the estimated application acquisition costs for the various scenarios?
  4. What are the estimated costs for business process redesign?
  5. What are the estimates for implementation costs for the various alternatives?
  6. How many hours are being assumed for each?
  7. What hourly rates for consultants are the implementation assumptions based on?
  8. What are the estimated costs for migration and conversion in each of the alternatives?
  9. If a best-of-breed solution is chosen, what are the expenses for software infrastructure and services associated with application integration at a level similar to what would be present in an integrated application suite?
  10. What are the application maintenance and support expense estimates?
  11. What are the training costs for the various alternatives?
  12. What are the internal staffing costs for the alternatives during the project for the SAP environment and for transition scenarios with risks and probabilities assigned for the different concurrent environment scenarios?

The Marin Information Systems and Technology group appears to have concluded that fixing the Deloitte-installed SAP application will cost nearly 25 percent more over a ten-year period than buying, modifying, implementing, and migrating data over to a new system in a protracted multi-phase project, during which time they would continue to operate the SAP environment concurrently, until going live on the respective new system modules. That conclusion seems implausible to me

My take. No good comes from situations like this. Marin appears to have wasted $30 million dollars; Deloitte faces a lawsuit and bad press; and SAP's brand and reputation will suffer even though no one blames the software.

Note to software vendors and system integrators: A substantial number of your customers, especially those who are smaller, only purchase ERP once in many years; these folks require more education than you typically provide during the selling process. Situations like this make clear that something is deeply wrong with your approach to sales and consulting.

Note to enterprise buyers: There is no escaping that you are ultimately responsible for the success or failure of your project. That said, when you hire services vendors be sure the contract includes provisions that connect cost and time to achieving successful results. If your vendors don't have skin in the game, then go elsewhere.

I could add much more to the list of suggestions, but that would require a book-length blog post.

Image from iStockPhoto. Thanks to Francine McKenna for assistance in preparing this report.

Topics: Government US, Government, SAP

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

102 comments
Log in or register to join the discussion
  • Poor Governance in the Extreme

    Another great post Michael. But I wish the object of the failure was something other than an ERP system. I fear many folks responsible for deploying technology at their enterprises won't pay this event much notice because they believe these types of failures only occur with big ERP implementations. There are SO many lessons to be learned here and there is SO much good advice that applies to technology endeavors of all shapes and sizes. Anyone who cares about realizing value from their technology investments should read this post.

    Steve Romero, IT Governance Evangelist
    http://community.ca.com/blogs/theitgovernanceevangelist/
    Steve Romero
    • Blame all around

      Good post, Michael. It just goes to show that everyone's hands are dirty on these types of things. For one organization to absolve itself completely from blame merely represents legal posturing. Those in the know won't buy into this.

      I find it interesting that SAP won't be the system at Marin. I'd argue that Marin could have chosen another ERP and, absent the proper governance measures that you and Mark describe, the outcome would have been the same.
      phil_simon
      • RE: Understanding Marin County's $30 million ERP failure

        Phil, I completely that these types of failure are almost always independent of specific software package. Without governance, monitoring of the services provider, and intelligent commitment from the internal organization, the project is often doomed.
        mkrigsman@...
      • RE: Understanding Marin County's $30 million ERP failure

        @philsimonsystems
        Have to agree.
        I've been doing these for several years and unless the client embeds the business decision makers into the process the outcome is likely to be just like Marin's.
        rhonin
  • RE: Understanding Marin County's $30 million ERP failure

    Very good article and may be worth a follow-up article or a series based on this one. ERP is a difficult solution to implement and the SAP system is not for the faint of heart. This article covered a lot of the key points to consider and it is worth looking at the total lifecycle cost including on-going operational and support/upgrade costs. The $30M sunk cost plus additional transition costs to a new system should demand further investigation to see what could be done to salvage a base SAP system. Who is looking at the waste of tax payer dollars as even the consultants should have recommended a more phased in approach of the more difficult functions as the business processes and responsibilities did not seem to be clear in this case.
    emitretsam
  • Another SAP/Deloitte failure... what's new?

    The $30 million in sunk costs has to be weighed against the unknowable millions that could have been lost had they continued to pursue this project, which seems to have been irretrievably broken. In that respect maybe their choice to kill it was most appropriate.<br><br>The only exception I would take to your post is the idea that "SAPs brand and reputation will suffer even though no one blames the software." The reason I disagree with this idea is because the landscape is littered with the bones of failed SAP projects and they continue to hold the largest market share. They've consistently been able to defend their high rate of failure on the unwillingness of clients to follow the rules for implementation. And, for better or worse, they're usually correct.<br><br>As to Deloitte's culpability, Project Portfolio Excellence Truth #20 sums it up: The extent to which management allows itself to be misled by vendors and consultants is their own fault.<br><br>Bill Monroe, Project Portfolio Excellence
    wmmonroe@...
    • That's because SAPs "high rate of failure"...

      is no higher than any other ERP vendor. 99% of all ERP implementation failures fall squarely on the company that bought the software or the consulting firm they paid to manage the implementation. Implementing an ERP system is a long, difficult and largely thankless task. When done well, it pays for itself quickly. When done poorly, it can bankrupt a company. I have yet to see a scenario where the software itself, whether it be SAP, BAAN, PeopleSoft, JD Edwards, MS Dynamics or a hundred other ERP systems out there, was the reason an implementation failed. I've always felt that all ERP vendors should provide certification paths which at least help to demonstrate proficiency in implemeting their software. It's one thing to be able to support a software package that's already up and running, but another thing entirely to get it up and running, configured and get data imported from legacy systems so that the system is actually useful from day 1 of go-live. If SAP, and all ERP vendors for that matter, bear responsibility in failed implementations this would be it.
      jasonp@...
      • RE: Understanding Marin County's $30 million ERP failure

        @jasonp@...

        Too many times the responsibility is charged to the customer.
        If the Software vendor sells a product very sophisticated (see too much complicated), I think the responsibility must be shared.
        smarta@...
      • RE: Understanding Marin County's $30 million ERP failure

        @smarta@...
        I can't agree.
        SAP does offer several differnt consulting groups and has a company rep built into these projects.
        The decision comes down to the customer on what consulting firm to utilize and their use of their SAP representative.
        rhonin
      • RE: Understanding Marin County's $30 million ERP failure

        @jasonp@... " 99% of all ERP implementation failures fall squarely on the company that bought the software or the consulting firm they paid to manage the implementation."

        That is way to high of a figure. Especally when you consider that many smaller systems it is the company itself who are the consultants. It is funny I learned this in college but rarely see it practice because of the cost and people don't know what they want untill they see it. One should map out the whole process before the coding starts for any database. Yet when I looked at ERP systems for a medium size corporation no sales people seem to want to give me their detail process maps. I have seen when the requirments are properly mapped out and good their is good project managment one saves a lot of headachs. Even then I would not always say the projects are always successful.
        bcclendinen@...
      • Too high?

        If anything, 99% is too low but I try to avoid any claims of 100%. I have yet to see an ERP failure that was directly attributable to the ERP software. I've spoken with several other implementation specialists and none of them have either. From my experience, those failures are fairly equally divided between the implementation vendor and the customer. There are some vendors who should really get out of the implementation business altogether...one in the New England area in particular I've advised customers against doing business with many times.
        jasonp@...
    • RE: Understanding Marin County's $30 million ERP failure

      @wmmonroe@... I have no experience from the viewpoint of someone who recieves SAP output. From the viewpoint of one who inputs to a SAP system; I have never seen a more user antagonstic piec of software
      annnon
      • RE: Understanding Marin County's $30 million ERP failure

        @annnon You are so correct. SAP is the most user antagonistic piece of ERP software... except for all the other ERP software suites.
        hawks5999
    • RE: Understanding Marin County's $30 million ERP failure

      @wmmonroe@... in other words, "never give a sucker an even break". customers are bozos, service companies are either there to help the bozo or just make a fast buck. this strikes me as the latter.
      jiagebusen
  • RE: Understanding Marin County's $30 million ERP failure

    Michael, it's an interesting article of what went wrong at a high level but it's not clear what specifically isn't working and why. We all know that SAP is proven, state of the art software, and that it works when properly configured. So the root cause of the problem must be that the software is either not properly configured or that Marin County didn?t understand the implications of the configuration decisions and how the software would work based upon these decisions. In either case, the responsibility clearly rests with Deloitte and the County.

    Rather than dump a $30 million investment, my recommendation would be for the ?Devil?s Triangle? players (SAP, Deloitte and the County) to work together to figure out exactly what isn?t working and fix it. Start from the beginning and review each configuration decision. Is it correct? If not fix it. If it is properly configured, does the County understand how the system will work based on this configuration. Are changes in workflow and processes required and are the users properly trained on these new processes?

    SAP and Deloitte should agree to go through this process at costs and the County should agree to pay these costs, since it is probably that these costs will be lower than starting over. It?s not that hard to determine the vendors? costs, which would include salaries of the people assigned and eligible travel expenses (based upon County policies). Most likely there are bad relationships between the people who have worked on this project for the past 5 years, so it?s also a good idea to address this problem.

    Dumping SAP is a bad idea, since any alternative solution will cost the County an additional $30+ million at a minimum and be subject to the same problems. Moreover, there will be a whole new learning curve to surmount and the users will need to be trained on a new look and feel, a different workflow and new/different processes. Unless the County wins its lawsuit against Deloitte, which sounds questionable, the alternative proposed of working together to resolve the problems will be less costly and take less time. I would be amazed if the current problems couldn?t be resolved in less than a year if all parties work together as a team.
    bcrowell5241
    • RE: Understanding Marin County's $30 million ERP failure

      @bcrowell5241 - proven -state of the art!!! - hahahahahahahahahahahahahahahahahaha
      steve@...
      • RE: Understanding Marin County's $30 million ERP failure

        @steve@... yeah, those two descriptors gave me pause as well. I won't say that SAP isn't proven or state of the art compared to it's competitors but I'd never lead with the idea that SAP is either in general terms.
        hawks5999
      • RE: Understanding Marin County's $30 million ERP failure

        @steve@...
        Exactly...especially when ABAP is such a poorly designed language....that's a "tell all". SAP has just "bulldozed" the industry with a lot of "bull".
        msimms
      • RE: Understanding Marin County's $30 million ERP failure

        @steve@... Funny my 3.5 billion dollar gross sales company implemented SAP in 1999, was just 3 month late (Feb go-live vs. May), but was within budget tolerance. Since then we have SCM, BW, CRM, BOBJ and HR systems. All integrate and work just fine. In fact integration is SAP?s key selling point. So to blame the software is easy, and usually the lame way out. I usually find comments from the above and mssimms below are people that have failed to learn or know other software and are scared of SAP?s size or could/did not grasp SAPS learning curve. However, once grasped other software appears to be immature. The interesting thing once you learn one SAP package, all of the other ERP packages in the suite just fit. We run our IT show (Basis through functional) with ~80 people. And support 24X7 worldwide implementations, 3500 employees worldwide and currently bringing up new offices/distribution centers every 3 months. We looked at 4 other vendors all failed in comparison to SAP?s R&D, and the industry size. Are they perfect ? now that is funny. Are they state of the Art? no and thank goodness. State of the art is usually something that looks good fails to meet your needs can?t be configured and the company will be bought or chapter 11. Of the 4 other vendors, we looked at the two higher candidates are now no longer or were absorbed with someone else. My observations are the consultants. They off shore to save money and that leads to mistakes. Off shoring will lower quality of work by a certain degree. Unless it is text book, or they can plug into the "new dehli network" no thought or does not require ANY business acumen, then yes off shore does work because of the time differential and the brute force of doing something, like and upgrade. The failure comes in three areas with consultants from my experience on three major project ? ERP, SCM, BW and a very complicated IP implementation. 1) the consultants are young and out of school are sold as?experts? with knowledge in the tool and industry. They are Green as a new gourd telling seasoned people what to do 2) off shore which I mentioned above and they hit the sweat shops to pull in and provide local ?experts?. 3) if your project deviates from white papers or basic/text book installations (name a business that doesn?t have something special) then you are screwed. I have seen it time and time again. These are not boutique consultants either these are Delloites, Accenture?s (we called them Accidenture after that project ), PWC?s of the world. Management likes them because when you throw that much money around it?s easy to blame them and move on. Sorry I have 3 major 10 Million + projects I have been part of and watched this from the inside. Once the employee?s learned the software, the ?consultant?s? were nothing more than cheap contractors at $180/hr. All of this from a private company, I can only imagine the hell a Government ? let alone a CA office ? can be like, especially the current box of cra@p we have in office. So yeah agree with the above, the consultants are probably the bigger blame and being a Gov't office it's probably just as responsible.
        ItsTheBottomLine
    • RE: Understanding Marin County's $30 million ERP failure

      @bcrowell5241 seems like a good idea, but I can't see Marin accepting it. It is very obvious they lost all confidence in Deloitte and it will be very hard for them to continue to pay a consultant they have no confidence in. What might be a better solution would be to keep the software and hire a new consultant company to fix the problems they are having. It's probably cheaper than scrapping their current set up.
      Al_nyc