San Diego fires Axon over ERP implementation problems

The city of San Diego, CA terminated its software implementation contract with services provider, Axon, citing "systematically deficient project management practices." The project is running $11 million (27 percent) over-budget to date, a number which will likely increase.

San Diego's termination memo highlights key Axon governance and implementation process deficiencies. Tom Fleming, president of San Diego Data Processing Corporation (DPC), a municipal IT contracting agency, wrote the document:

[Axon failed] to adequately maintain the project plan in a manner that permits effective resource utilization, "critical path" management, cost control and ... overall status against agreed upon project timelines and deliverables.

Axon has not maintained a project risk register and issues register in a consistent manner.

Axon has generally not followed the change control process, issue escalation procedure or project control protocol as stipulated in the SOW.

The cumulative effect project management deficiencies are an estimated ten month delay in the completion date....

The termination memo also criticizes the quality, caliber, and skills of Axon's consulting resources:

[C]ertain Axon functional resources have not demonstrated sufficient knowledge of public sector business practices.

Axon was not following the best practices as promulgated by the Software manufacturer.

Axon has submitted incomplete work product for acceptance and payment.

[T]he division of responsibility and workload between Axon consultants and their City Counterparts does not meet the spirit of the contractual arrangement between DPC and Axon.

Mary Lewis, San Diego's Chief Financial Officer, wrote a memo explaining the city's decision to replace Axon with implementation consultants from SAP, the software vendor:

SAP is uniquely positioned to resolve complex implementation challenges and to complete projects where previous efforts by other firms have failed to meet client expectations. In fact, several other local, state and federal government agencies have recently experienced similar implementation difficulties and have turned to SAP to resolve implementation issues.

These Agencies include:

• City of Portland, Oregon
• Tarrant County, Texas
• Minneapolis Public School District, Minnesota
• State of Washington
• Northern Kentucky University, Kentucky
• United States Postal Service
• Army Medical Material Agency

A section of that memo raises a significant red flag, suggesting the city plans to customize the SAP software:

The engagement of SAP permits a direct line of contact to product development teams at SAP which will facilitate quicker resolution of product customization issues that are required because of business requirements unique to the City.

In general, customizing code in packaged software like SAP carries risk, as I explained in another blog post:

Custom code in a packaged solution creates a variety of evils, which taken together lead to cost and time over-runs downstream, aside from increased development costs and risks during the project.

• Related: 'Custom Code Over My Dead Body'

In a relatively interesting disclosure, city officials released a report describing San Diego's financial arrangements with SAP:

[San Diego] executed a single source fixed cost contract with SAP for an amount not to exceed $14.2 million, using a fixed price deliverable based payment and phased go-live approach. There is an additional $1.5 million time-and-materials contract for 90 days of "post go-live" support, for a contract total of $15.7 million. It is important to note that the Axon contract had only 30 days of post go-live support as part of their fixed price.

THE PROJECT FAILURES ANALYSIS

This project reads like a stereotype how a local government transformation initiative can go wrong:

• San Diego, the client, didn't understand the full scope and extent of the undertaking
• Axon, the system integrator, is accused of supplying inadequate resources, project governance, and oversight, all of which substantially caused the failure

Local government responsibilities. In general, it's not unusual to see an enterprise software implementation overwhelm a small local government's capacity to handle change. In an interview with the San Diego Union Tribune, the city acknowledged its own contributions to this situation:

Chief Operating Officer Jay Goldstone said in an interview that the city had underestimated the cost and been too aggressive about the timing at the start.

“Hindsight is always 20/20,” Goldstone said. “This, maybe on Day One, if we could have rewritten the script, might have been a $45 million project.”

A recent PeopleSoft implementation failure in Oak Park, IL shared similar issues:

On the surface, it’s easy to blame the original system integrator as well as PeopleSoft. Both of these participants should probably have worked more closely with the town to ensure that process change expectations were clear and realistic. However, it’s entirely possible the external folks explained everything to the town, but somehow the information didn’t sink in.

Marin County, CA faces another similar situation on its SAP project, as described in the local newspaper, Marin Independent Journal. [Disclosure: I coached the Marin reporter how to analyze and understand this failure]:

[A] complicated system more sophisticated than necessary was installed, based in the wrong department and implemented too quickly as officials bowed to advice from consultants they now question. Critics were all but ignored. Training was minimal. Management was erratic.

Top managers overseeing the project bailed out in midstream to join software firms. The county's in-house computer experts were shunned as consultants moved in, and the high-tech staffers remained out of the loop until auditors threw up their hands in despair.

"We acknowledge this hasn't been the best implementation," said County Administrator Matthew Hymel, conceding that mistakes were made as the county tried to do too much too fast.

System integrator problems. System integrators can also contribute to their clients' failure situations. My recent interview with Forrester analyst, Natalie Petouhoff, focused on exactly this issue:

System integrators have a long-term business model based on projects going on and on and on; it’s a little like foxes guarding the hen house. Completing projects quickly is not necessarily in the integrators’ best interest. Many consulting firms went out of business because their old model of writing change orders and blaming it on the client no longer works. Customer pushback has kicked system integrators in the butt.]

My take. Given tightly interwoven accountability on both sides, it's difficult to accurately dissect the strands of responsibility. I suspect both San Diego and Axon bear responsibility for this failure. Regardless of what happened in the past, the new contract puts SAP squarely on the hot seat to get this project back on track.

Update 2/25/08 4:00pm EST: I received the following comment from SAP. Note the emphasis on improving governance:

SAP and SAP Public Services are 100% dedicated to the success of the project. We recognize the importance of this project to the citizens and officials of San Diego and we"ll deploy the resources necessary to complete the project. We know that transparency and team work at all levels of both SAP and the city are necessary as we work together to complete this successfully and we're encouraged by the strong alignment between the City and SAP senior executives. We know this leadership team will guide the project through the various decision points.

We have brought two major changes designed to ensure the project's success: 1) Methodology and 2) A refined governance model to help ensure swift decision-making and adoption. We believe this will help provide the project with transparency and accountability as we move forward together. Our goal with this model is to bring the project into line with expectations on cost and schedule, user satisfaction and performance.

[Thanks to an anonymous reader for bringing this situation to my attention. Readers, I love you! Video clip summarizing the situation from the San Diego News. Neither Axon nor San Diego Data Processing Corp. responded to my requests for comment.]