IBM's Devil's Triangle: An enterprise software soap opera

IBM faces lawsuits and public embarrassment in the Philippines over a failed government project involving the company's DB2 database product. The situation offers a textbook example of the Devil's Triangle, and demonstrates the tensions and conflicts that arise between technology vendors, customers, and system integrators.

Background. The Government Service Insurance System (GSIS), a Philippine agency responsible for managing the pensions of government employees, installed DB2 in 2006. By early 2008, the system began showing signs of weakness. Local newspaper, the Philippine Daily Inquirer, describes what happened:

[GSIS chief legal counsel Estrella Elamparo] explained that the software started showing problems in early 2008, particularly in handling voluminous chunks of data.

“IBM upgraded its database system purportedly to enable it to handle unlimited volumes of data,” Elamparo said. “However, the reported upgrade only worsened the problem because instead of fixing the problem, the database began mishandling data and prevented the simultaneous use of data.”

The government threatened lawsuits in response, according to the paper:

The Government Service Insurance System warned...it would launch “a series of legal actions” against IBM for “incalculable damages” that the pension fund sustained due to defective computer software.

GSIS went on to purchase a full-page spot in major newspaper, the Manila Bulletin, and published an open letter explaining the situation. The Bulletin reports:

The Government Service Insurance System (GSIS) has put out an open letter to its active members and pensioners, detailing how the software installed by computer giant IBM in the GSIS system has, to borrow the Fund’s own term, “turned into a nightmare.”

GSIS has been having difficulties processing the claims and benefits of members, pensioners and other beneficiaries, as well as the glitches encountered in some instances in the posting of payments made to the GSIS.

Another Daily Inquirer article describes the technical environment:

Full swing implementation was in 2008 after GSIS tapped systems integrator Questronix Corporation. Under the contract, Questronix would implement a full database package using DB2 version 8 and an SAP business application.

However, the agency started seeing problems with DB2 version 8, particularly “bad pages” on the tables. The agency later discovered that the DB2 tables could only handle 256 gigabytes of data, which was lower than what GSIS needed.

The database software was then upgraded to DB2 version 9.1, which could handle 512 exabytes of data, or one billion gigabytes.

[Elamparo] explained their database table exceeded 2 terabytes (2,000 gigabytes) worth of transactions. “We expected DB2 to keep running since it was only 2 terabytes.”

GSIS is now pursuing various legal measures against IBM, including filing criminal charges accusing IBM Philippines of bid rigging in connection with the project. From local newspaper BusinessMirror:

Elamparo [said] that IBM Philippines may be held liable for possibly dictating the bids of its dealers as a third party who is technically not a bidder.

“By giving various discount levels to its dealers involved in the bid—from 30 percent to 80 percent —IBM Philippines is in effect empowering one dealer over another, since those who had gotten a bigger discount would naturally be able to bid lower,” Elamparo said.

She brought the National Bureau of Investigation's attention to Section 65 of RA 9184, which bars a bidder from employing schemes that restrain natural rivalry of bidders, refrain from bidding to provide another bidder undue advantage, and knowingly submit high bids to have the contract awarded to a prearranged lowest bidder.

IBM response. In a statement to local television station, ABS CBN news, IBM denied creating the problems, pushing blame back to GSIS and third-party system integrators:

"GSIS did not engage IBM in the selection, customization and implementation of this system. IBM was the OEM (original equipment manufacturer) provider to one of the technology vendors engaged by GSIS. GSIS does not have any maintenance or support contract with IBM," the statement read.

"Nonetheless, in view of our long standing relationship with GSIS and out of goodwill, IBM has been working with GSIS' solution providers to resolve GSIS' system issue."

IBM also denied violating procurement laws, according to another local station, GMA NEWS TV:

The Philippine unit of US-based IBM said “it does not agree" with the statements made by the pension fund which earlier said that the company asked one of its dealers to pull out from a bidding for an IBM Linux server.

"When IBM was notified of GSIS's concerns last year, we took immediate action to thoroughly investigate the matter and found there were no irregularities in the way we conducted our business. We shared these findings with GSIS," IBM Philippines said in a statement.

SAP distanced itself from the situation, saying it has no contract with GSIS:

SAP Philippines said that it had no contractual relationship with the agency in the implementation of its IT infrastructure software or consultancy services.

Instead, the software company stressed that GSIS licensed SAP software from Team Synergia, a local value-added reseller.

THE PROJECT FAILURES ANALYSIS

Despite the hyperbole, accusations, and denials, it appears we can ascertain these facts:

• GSIS purchased a DB2 system from IBM as part of a broader system implementation.
• IBM did not supply either implementation services or substantial software aside from DB2.
• Something went horribly wrong with the DB2 deployment, interfering with GSIS's ability to process data resulting in substantial negative impact on the agency's members.
• SAP supplied ERP software through a third-party reseller, but did not perform any contractual services directly to GSIS.

On the surface, IBM appears to be the bad guy, supplying a malfunctioning database and then refusing to help when things turned bad. However, large-scale system deployments are so complex that one cannot isolate the technology from the associated implementation services.

In this situation, we do not have enough information to divide responsibility between IBM, the customer, and third-party system integrators. This situation is a classic Devil's Triangle relationship, with tensions arising between vendor, customer, and integrator due to conflicts of interest that are present amidst shared goals and outcomes.

I asked IT expert and fellow ZDNet blogger, Paul Murphy, for his opinion on this confusing mess. Paul suggests GSIS owns substantial responsibility:

DB2 can handle the job -  but getting AIX to handle large quantities of non local disk is non-trivial and because this particular transition involves migrating people and skills from OS/390 to AIX I'd bet the farm that the technical mistakes causing the failures were made by deeply resentful mainframers doing jobs they don't understand and don't want to do - and those people, of course, work for GSIS, not IBM.

Thus senior IBM people in the sales channel probably bear some responsibility for helping the client get into this mess, but blaming either DB2 or IBM's technical people for it is likely to be wrong.

My take. It's almost impossible to accurately analyze this situation from a distance. In my opinion, whether IBM is ultimately at fault is somewhat beside the point. The company sold the software knowing that the customer's environment is complex and therefore should be deeply committed to helping make things right.

In addition, GSIS should disclose more details about how it managed the project. I suspect Paul's assessment is accurate and that there's plenty of blame to go around. The situation will likely remain a stalemate until GSIS is more forthcoming and accepts its own responsibilities.

Sadly, the victims of this enterprise software soap opera are those who rely on GSIS for loans and other forms of financial support. This is hardly a shining moment for enterprise software, IBM, GSIS, or the system integrators.