Understanding Marin County's $30 million ERP failure

The Board of Supervisors in California's Marin County voted to stop an ongoing SAP project and seek a replacement solution, implicitly accepting that it wasted over $30 million on software and related implementation services from Deloitte Consulting.

The dramatic decision to replace SAP comes after relations between Marin and Deloitte Consulting, the implementation consultant on this project, deteriorated to the breaking point. Marin filed a lawsuit against Deloitte "seeking actual and compensatory damages of at least $30 million, along with unspecified punitive and, or exemplary damages and interest." Deloitte has countersued Marin for approximately $550,000 in unpaid fees and late charges.

Related: Marin County sues Deloitte: Alleges fraud on SAP project

The Contra Costa Times described the situation:

After spending more than $30 million on a fancy computer system that never worked well, top Civic Center officials have concluded the program should be scrapped as soon as a more efficient system can be phased in.

The Marin Independent Journal offered historical context:

The Board of Supervisors bought the SAP system in 2005, then paid mounting consultant bills as staff grappled with a fiscal nightmare in which the program could not produce basic financial reports. SAP never performed as promised, bobbling the payroll, accounts receivable and other routine financial functions. A grand jury probe concluded it cost taxpayers $28.6 million as of April 2009

Computer World brought in SAP's perspective:

SAP "strongly believes in the value and performance of its software in use in Marin County," said spokesman Andy Kendzie. "Our software works exactly as it should, and any issues in this implementation in no way reflect on SAP. Our software is installed and functioning perfectly in tens of thousands of public sector agencies, including dozens in California."

Marin's decision to jettison SAP and start over with new software is based on an analysis performed by Phoenix Business Consulting, a firm that Marin hired at a cost "not to exceed $50,000." The statement of work is relatively vague and uninformative:

To review the SAP program currently implemented at the County of Marin and analyze whether it meets the needs of the County, including whether it was properly implemented by Deloitte. Review what additional costs and efforts are necessary for SAP to properly function to meet the needs of the County Departments. Review what efforts were taken to fix the SAP problems which occurred post-implementation.

STRATEGIC ANALYSIS

This situation offers a clear example of how mismatched expectations can drive poor communication and lead to complete failure. To help understand what really went wrong, here is a brief description of each party's role in contributing to this situation:

Marin role. In my opinion, having reviewed substantial documentation, Marin's decision to replace SAP seems intended primarily to strengthen its lawsuit position and push all accountability away from itself. Marin's position is extreme and not credible.

Marin's apparent lack of organizational and governance maturity, and its inability to absorb business transformation changes associated with this implementation, seem to be a basic driver underlying this failure.

A summary status report compiled by Marin's Information Services & Technology Department acknowledges significant lessons learned on this project that it plans to apply on a future ERP implementation:

At this stage, staff is recommending that we look at other system options and recommend the following approach:

1. An incremental, phased approach to the replacement of SAP, rather than the "big bang" approach that was intentionally followed in the SAP implementation, as was advised by the outside consultants;
2. IST involvement up front to guide the steering committee of employees from key departments in recommending a system and leading its implementation;
3. Less reliance on outside consultants and more on County staff, who have a greater, vested interest in the outcome and success of the implementation; and
4. Routine communication with employees, the public and Board of Supervisors in the form of regular status reports and meetings, including meetings with an oversight subcommittee of the Board for the project.

A town supervisor also emphasized the importance of lessons learned during a meeting in which the Board voted to stop the project (video).

Deloitte role. Most emphatically, I believe Deloitte shares equal, or perhaps even greater, culpability in creating this situation. Deloitte's posturing and unwillingness to accept even partial responsibility for the failure appears inconsistent with the facts. Deloitte seems focused on compensation arising from the implementation process itself, without regard to whether the client achieved successful results or outcomes.

Deloitte's position and actions stand in stark contrast to these customer-centric words from its website:

For some, the path to value is a steady march. For others, it unfolds through bursts of innovation. But for the best of the best, it’s always both. Disciplined improvement initiatives, marked by powerful leaps and breakthroughs.

SAP role. Neither Marin nor Deloitte have suggested that SAP software contributed in any way to the failure. However, a complete analysis must consider the role of SAP in setting client expectations regarding demands imposed by the implementation process.

The extent to which software vendors and integrators should take responsibility for decisions that buyers make during the purchasing process is a difficult question. As I have written, the solution lies in three areas:

1. Enterprise customers should be more careful assessing their own capabilities before undertaking any complicated organizational change initiatives, including implementing ERP systems.
2. Systems integrators and consulting firms must be more straightforward in explaining pitfalls and success requirements to potential clients. Some consultants paint an overly positive picture during the sales process; such nonsense has to stop.
3. Software vendors should tie a greater percentage of sales compensation to ultimate customer satisfaction.

I have referred to relationships among enterprise customers, software vendors, and system integrators as constituting a "Devil's Triangle."

For more information on the Devil's Triangle see:

Exploring the Devil’s Triangle The IT failures blame game (part 1) The IT failures blame game (part 2) ‘Pain chains’ and the IT Devil’s Triangle

Understanding the numbers. The county's decision to throw away its $30 million investment bears scrutiny, especially during this period of budget cuts and economic difficulty.

To understand more, I spoke at length with Mark F. O’Connor, CEO and co-founder of Monadnock Research, which offers advice to enterprise customers on working with services firms. In an excellent analytic writeup, Mark says:

Marin may want to step back and take a closer look at the findings of the report it is using to support that conclusion...Analysis of those findings, however, appears to offer clues into apparent unrealistic expectations similar to what got Marin into its current predicament.

I asked Mark to elaborate this view:

Opponents to the SAP-based system implemented at Marin County are clearly aligned around getting rid of it as quickly as possible. But it appears the County may never have accepted responsibility for making it work in the first place. In fact, my reading of the agreement with Deloitte seems to indicate the County was trying to cede its responsibilities to Deloitte.

Responsibility without authority, however, always yields outcomes similar to what we see here in complex systems projects. There are literally thousands of important decisions that need be made by client staff during an implementation—while they continue to do their day jobs. I would caution against pre-deciding what will happen with SAP at this point, before the County’s alternatives are fully vetted. While there is political consensus to move off SAP, there are workflow and human factors to consider, and a lack of consensus across all departments as to whether they should initiate another disruptive systems project so soon.

Additionally, the financial justification that appeared persuasive to the Supervisors was remarkable to me only in its lack of detail. The County needs to ensure that it doesn’t make decisions to improve its position in the Deloitte lawsuit at the expense of its financial interests and ability to effectively provide services to Marin County residents.

I would hope Marin's analysis and assessment of the various alternatives includes addressing these twelve key questions:

1. What are the estimated expenses for hard and soft dollar costs associated with preparing the five plans the Board of Supervisors just authorized?
2. What is the estimated cost for managing the alternative assessment, selection, and implementation process?
3. What are the estimated application acquisition costs for the various scenarios?
4. What are the estimated costs for business process redesign?
5. What are the estimates for implementation costs for the various alternatives?
6. How many hours are being assumed for each?
7. What hourly rates for consultants are the implementation assumptions based on?
8. What are the estimated costs for migration and conversion in each of the alternatives?
9. If a best-of-breed solution is chosen, what are the expenses for software infrastructure and services associated with application integration at a level similar to what would be present in an integrated application suite?
10. What are the application maintenance and support expense estimates?
11. What are the training costs for the various alternatives?
12. What are the internal staffing costs for the alternatives during the project for the SAP environment and for transition scenarios with risks and probabilities assigned for the different concurrent environment scenarios?

The Marin Information Systems and Technology group appears to have concluded that fixing the Deloitte-installed SAP application will cost nearly 25 percent more over a ten-year period than buying, modifying, implementing, and migrating data over to a new system in a protracted multi-phase project, during which time they would continue to operate the SAP environment concurrently, until going live on the respective new system modules. That conclusion seems implausible to me

My take. No good comes from situations like this. Marin appears to have wasted $30 million dollars; Deloitte faces a lawsuit and bad press; and SAP's brand and reputation will suffer even though no one blames the software.

Note to software vendors and system integrators: A substantial number of your customers, especially those who are smaller, only purchase ERP once in many years; these folks require more education than you typically provide during the selling process. Situations like this make clear that something is deeply wrong with your approach to sales and consulting.

Note to enterprise buyers: There is no escaping that you are ultimately responsible for the success or failure of your project. That said, when you hire services vendors be sure the contract includes provisions that connect cost and time to achieving successful results. If your vendors don't have skin in the game, then go elsewhere.

I could add much more to the list of suggestions, but that would require a book-length blog post.

Image from iStockPhoto. Thanks to Francine McKenna for assistance in preparing this report.