IBM's Devil's Triangle: An enterprise software soap opera

IBM's Devil's Triangle: An enterprise software soap opera

Summary: IBM faces lawsuits and public embarrassment in the Philippines over a failed government project involving the company's DB2 database product.

SHARE:

IBM faces lawsuits and public embarrassment in the Philippines over a failed government project involving the company's DB2 database product. The situation offers a textbook example of the Devil's Triangle, and demonstrates the tensions and conflicts that arise between technology vendors, customers, and system integrators.

Background. The Government Service Insurance System (GSIS), a Philippine agency responsible for managing the pensions of government employees, installed DB2 in 2006. By early 2008, the system began showing signs of weakness. Local newspaper, the Philippine Daily Inquirer, describes what happened:

[GSIS chief legal counsel Estrella Elamparo] explained that the software started showing problems in early 2008, particularly in handling voluminous chunks of data.

“IBM upgraded its database system purportedly to enable it to handle unlimited volumes of data,” Elamparo said. “However, the reported upgrade only worsened the problem because instead of fixing the problem, the database began mishandling data and prevented the simultaneous use of data.”

The government threatened lawsuits in response, according to the paper:

The Government Service Insurance System warned...it would launch “a series of legal actions” against IBM for “incalculable damages” that the pension fund sustained due to defective computer software.

GSIS went on to purchase a full-page spot in major newspaper, the Manila Bulletin, and published an open letter explaining the situation. The Bulletin reports:

The Government Service Insurance System (GSIS) has put out an open letter to its active members and pensioners, detailing how the software installed by computer giant IBM in the GSIS system has, to borrow the Fund’s own term, “turned into a nightmare.”

GSIS has been having difficulties processing the claims and benefits of members, pensioners and other beneficiaries, as well as the glitches encountered in some instances in the posting of payments made to the GSIS.

Another Daily Inquirer article describes the technical environment:

Full swing implementation was in 2008 after GSIS tapped systems integrator Questronix Corporation. Under the contract, Questronix would implement a full database package using DB2 version 8 and an SAP business application.

However, the agency started seeing problems with DB2 version 8, particularly “bad pages” on the tables. The agency later discovered that the DB2 tables could only handle 256 gigabytes of data, which was lower than what GSIS needed.

The database software was then upgraded to DB2 version 9.1, which could handle 512 exabytes of data, or one billion gigabytes.

[Elamparo] explained their database table exceeded 2 terabytes (2,000 gigabytes) worth of transactions. “We expected DB2 to keep running since it was only 2 terabytes.”

GSIS is now pursuing various legal measures against IBM, including filing criminal charges accusing IBM Philippines of bid rigging in connection with the project. From local newspaper BusinessMirror:

Elamparo [said] that IBM Philippines may be held liable for possibly dictating the bids of its dealers as a third party who is technically not a bidder.

“By giving various discount levels to its dealers involved in the bid—from 30 percent to 80 percent —IBM Philippines is in effect empowering one dealer over another, since those who had gotten a bigger discount would naturally be able to bid lower,” Elamparo said.

She brought the National Bureau of Investigation's attention to Section 65 of RA 9184, which bars a bidder from employing schemes that restrain natural rivalry of bidders, refrain from bidding to provide another bidder undue advantage, and knowingly submit high bids to have the contract awarded to a prearranged lowest bidder.

IBM response. In a statement to local television station, ABS CBN news, IBM denied creating the problems, pushing blame back to GSIS and third-party system integrators:

"GSIS did not engage IBM in the selection, customization and implementation of this system. IBM was the OEM (original equipment manufacturer) provider to one of the technology vendors engaged by GSIS. GSIS does not have any maintenance or support contract with IBM," the statement read.

"Nonetheless, in view of our long standing relationship with GSIS and out of goodwill, IBM has been working with GSIS' solution providers to resolve GSIS' system issue."

IBM also denied violating procurement laws, according to another local station, GMA NEWS TV:

The Philippine unit of US-based IBM said “it does not agree" with the statements made by the pension fund which earlier said that the company asked one of its dealers to pull out from a bidding for an IBM Linux server.

"When IBM was notified of GSIS's concerns last year, we took immediate action to thoroughly investigate the matter and found there were no irregularities in the way we conducted our business. We shared these findings with GSIS," IBM Philippines said in a statement.

SAP distanced itself from the situation, saying it has no contract with GSIS:

SAP Philippines said that it had no contractual relationship with the agency in the implementation of its IT infrastructure software or consultancy services.

Instead, the software company stressed that GSIS licensed SAP software from Team Synergia, a local value-added reseller.

THE PROJECT FAILURES ANALYSIS

Despite the hyperbole, accusations, and denials, it appears we can ascertain these facts:

  • GSIS purchased a DB2 system from IBM as part of a broader system implementation.
  • IBM did not supply either implementation services or substantial software aside from DB2.
  • Something went horribly wrong with the DB2 deployment, interfering with GSIS's ability to process data resulting in substantial negative impact on the agency's members.
  • SAP supplied ERP software through a third-party reseller, but did not perform any contractual services directly to GSIS.

On the surface, IBM appears to be the bad guy, supplying a malfunctioning database and then refusing to help when things turned bad. However, large-scale system deployments are so complex that one cannot isolate the technology from the associated implementation services.

In this situation, we do not have enough information to divide responsibility between IBM, the customer, and third-party system integrators. This situation is a classic Devil's Triangle relationship, with tensions arising between vendor, customer, and integrator due to conflicts of interest that are present amidst shared goals and outcomes.

I asked IT expert and fellow ZDNet blogger, Paul Murphy, for his opinion on this confusing mess. Paul suggests GSIS owns substantial responsibility:

DB2 can handle the job -  but getting AIX to handle large quantities of non local disk is non-trivial and because this particular transition involves migrating people and skills from OS/390 to AIX I'd bet the farm that the technical mistakes causing the failures were made by deeply resentful mainframers doing jobs they don't understand and don't want to do - and those people, of course, work for GSIS, not IBM.

Thus senior IBM people in the sales channel probably bear some responsibility for helping the client get into this mess, but blaming either DB2 or IBM's technical people for it is likely to be wrong.

My take. It's almost impossible to accurately analyze this situation from a distance. In my opinion, whether IBM is ultimately at fault is somewhat beside the point. The company sold the software knowing that the customer's environment is complex and therefore should be deeply committed to helping make things right.

In addition, GSIS should disclose more details about how it managed the project. I suspect Paul's assessment is accurate and that there's plenty of blame to go around. The situation will likely remain a stalemate until GSIS is more forthcoming and accepts its own responsibilities.

Sadly, the victims of this enterprise software soap opera are those who rely on GSIS for loans and other forms of financial support. This is hardly a shining moment for enterprise software, IBM, GSIS, or the system integrators.

Topics: Data Management, Data Centers, Enterprise Software, IBM, Software

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

27 comments
Log in or register to join the discussion
  • Where's the architect?

    Who is the person that modeled the (today and future) capacity - and thought that 256Gb tables were adequate? Obviously in 2006 the tables were not this big and things ran fine until 2008 when the tables started exceeding this threshold.

    Nothing works like a well architected project with milestones and "gate" reviews. The capacity "issue" should have been discussed and the 256Gb limitation would have to have been agreed upon.

    In any event, if the system was working "fine" for 2 years - then the "soundness" of the design was "proven". Although upgrades can be messy, this looks like a simple upgrade would fix the issue. And as anyone that has EVER talked with a support tech - the "upgrade" issue ALWAYS comes up. "What version are you running? Hmm, I would recommend that you upgrade ...".
    Roger Ramjet
    • hmmm

      Architecture 101: design spec matches requirement spec, for the lifecycle of the product.

      It exists and has been dutifully (routinely) ignored. Or, it might exist under clause 33b(i) "wish-list".

      Cute comment from Paul Murphy, re: GSIS 390er's sticking a (virtual) spanner in the works. I've got 5 bucks on GSIS management having the design nouse of a gnat and the veto power of the Philippine Gov more than a bunch of niggly sys-admins++...
      muzza2005
  • Too many chiefs

    I have seen this more than a few times. Often, the interests of big companies, vendors, and consulting firms do not align enough to ensure a successful implementation. I have worked on projects with more than one vendor and more than one consultancy to boot, making these things even more problematic.

    I agree that it's tough on these projects to determine who's at fault without being in the room.

    Great post. Very well researched.
    phil_simon
  • RE: IBM's Devil's Triangle: An enterprise software soap opera

    <blockquote>In my opinion, whether IBM is ultimately at fault is somewhat beside the point. The company sold the software knowing that the customer?s environment is complex and therefore should be deeply committed to helping make things right.</blockquote>

    sorry wrong. IBM makes it clear that there was no direct consultation with them by GSIS in the purchasing of DB2 - that was accomplished through a reseller. Nor did GSIS have any maintenance or implementation contracts with IBM. IBM only works with contracts: no contracts, no service which is the way it should be in a business relationship.

    My money is on Paul Murphy's take.
    Doug_Alder
    • Philisophical disagreement

      I understand your point, but disagree. Exactly where does IBM's responsibility begin and end?

      A major DB2 deployment is not like buying milk at the grocery store. IBM must have had some sense of its customer and the overall environment into which the product was being integrated. They sold the product and I believe should take more responsibility in getting things right.

      IBM had a choice whether or not to make that sale. Now let them help get the damn thing to work properly.

      mkrigsman@...
      • System Integrator!

        You're talking about the system integrator roll... not the OEM vendors.

        Who made the claims, and had the contracts for implementation and support?

        You're making a lot of assumptions, and ignoring parts of the article that discussed those exact things.
        dominigan
        • Hardly

          My position is clearly stated: IBM was probably not primarily responsible for the technology problems. However, their sales people chose to sell into an environment and IBM has some downstream responsibility.

          I do not accept the argument that the technology vendor has no responsibility during the sales cycle to ensure proper fit to task.
          mkrigsman@...
      • IBM "take responsilibity" = Fictional Utopia

        I don't disagree with your sentiment, but seriously... IBM's business model (and account management incentives) are all about sales, not about customer needs. The rep's are trained to target the highest level of corporate leadership they can engage with, and to leverage those lofty levels to successfully sell the customer the "solution" regardless of need.

        Having been on the recieving end of this too many times (the nightmare that is the Tivoli suite of solutions). Take a recent encounter over a subset of Tivoli v. a competing product. IBM costs 10x the competitor. The competitor's response met exactly what we stated the need was - IBM contending the solution did everything we need, and oh so much more that we don't know that we need, but IBM will assist with... (add to that their tales to our senior management that we are not capable of understanding the true value of what they are offering... so leadership should listen to the wise IBM'ers and dictate the solution for the poor technical staff that just can't comprehend the business needs.

        Result: 1 year into an implementation that still doesn't provide value - an internally developed stop-gap that is providing significant value - and a 3 year support commitment that we are obligated to pay regardless of ever getting the solution running. (add to that 3 rounds of IBM "solution experts" that had no industry experience with the product and finally getting a product developer onsite to at least get the solution in a running (but still not productive) state.

        And the reference to IBM's practice of giving variable discounts to vendors (with the supposed reasoning that the vendor that "broke the deal" deserves the bid...) is a practice that our company finds unacceptable as well. We had sever performance issues with a supplier, and it took an act of congress to get IBM to provide our new chosen supplier with similar discounting. Who are they to attempt to govern our business decisions!?!

        Can IBM provide value added solutions - yes. Cynical about IBM's motives - absolutely! Will they be "altruistic" for a price - professional services is where the real money is ... certainly they will.

        (and no, this isn't the first company I have been at that's had similar dealings with IBM)
        washjc
      • ROTFLMBO

        >>>Exactly where does IBM's responsibility begin and end?<<<
        IBM responsibilities begin and end with legally enforceable contracts with its customers. Since the government is not an IBM customer, IBM is not obligated to provide free services. It is possible that the government did not invite IBM and SAP to bid, since governments are often looking to ?save money? on large projects. This happens all the time. Here is how it government works. ? Bemoan the $200-300 per hour experienced consultants. Demand and get lower paid, less experienced project managers and consultants. Take three times as long and pay twice as much to deliver an inferior product. Look stupid, but declare success anyhow, and force end-users to use pathetic solution.


        >>>A major DB2 deployment is not like buying milk at the grocery store. IBM must have had some sense of its customer and the overall environment into which the product was being integrated. They sold the product and I believe should take more responsibility in getting things right.<<<
        If a major deployment isn?t similar to buying milk, then it would seem to me that the government would hire a vendor highly qualified to implement large-scale DB2/SAP solutions, and legally commit the vendor to implement satisfactory solution or money back. Alas, it didn?t. Anyone can do small integration jobs, but large jobs require real expertise and real experience ? obviously, this is something the government wasn?t concerned with until TSHTF.


        >>>IBM had a choice whether or not to make that sale. Now let them help get the damn thing to work properly<<<
        You?ve got it all wrong. The government had a choice in hiring its integration team. It could have easily hired an IBM/SAP team, but instead chose to roll with a low balling integrator. Now that the government has unwisely spent money, it wants a bailout, or if necessary, a bailout via lawsuits. Why else would it take out ads and try to create unfavorable opinions about IBM?

        IBM is running a business. They would like to take care of their investors and employees - not pay for mistakes by others (the government and the integrator). If the government wants IBM or SAP services, they'll have to pay the price.

        The government team implementing this solution is a joke ? and sadly, so are you ? for suggesting that IBM freely perform the work that another integrator was hired to do.

        (Now that we know you support free labor, what else do you support? Free Housing? Free Food? Free Clothing? Free Medical and Dental Plans? Free Love?)




        ShermB
      • A complete misunderstanding of the market...

        but I guess I shouldn't be surprised. Try calling Microsoft sometime for support of software you purchased through an OEM. They'll tell you to go back to the OEM or pay Microsoft for support. Same goes for IBM. Same goes for SAP. Same goes for every other software company in the world who sells product through an OEM channel. The whole reason companies go through OEM channels is to reduce the number of sales and support staff they need to keep on the payroll. If companies like IBM, Microsoft, SAP and others wanted to be on the hook for implementation and support every time they sold a software package, they'd go back to selling everything directly and dump the OEM channel completely. Considering every single DB2 implementation hasn't failed as miserably as this one indicates to me that the problem isn't with DB2, the problem is with the people hired to implement it. Having been through a large number of ERP implementations I can assure you that this scenario gets played out every single day, sometimes successfully and other times not. The fault for the "nots" is almost always the fault of the implementor or the company buying the software. It very rarely is the fault of the software company that wrote what is being implemented. When that happens, it rarely happens more than once or twice because nobody else will buy a product that can't be implemented correctly.
        jasonp@...
      • Questronix Sold DB2 to GSIS

        There is no Software or Maintenance Agreement for DB2 between IBM and GSIS. Questronix, who is a Partner of IBM, sold DB2 to GSIS. IBM never guarantees that DB2 or any of its software product is free of bugs. If IBM has a Maintenance Agreement with a customer who bought DB2 directly from IBM, and if there is a Maintenance Agreement between the two parties for DB2, IBM will attempt to fix any bugs that come up with DB2. In a very rare (I do not know of one in my 35 years using IBM software) instance where the product cannot be used because IBM cannot fix the bug, the customer may request for its money back. IBM has a Disclaimer in its contracts that it cannot be sued for consequential or any other damages other than for the original cost of the product.
        bertop
  • RE: IBM's Devil's Triangle: An enterprise software soap opera

    right, ibm is not to blame, it is the architect of the system that should be responsible in making it work.
    I think, the IT head of GSIS relied heavily on it's suppliers for solutions, not knowing itself what to do or how to solve their problems. Just goes to show you how incompetent people are in the philippine government.
    steveloso
  • RE: IBM's Devil's Triangle: An enterprise software soap opera

    I agree GSIS has a major hand in the success or failure of the project. I think the implementation of the DB2 upgrade went straight to Production without passing rigid quality assurance and/or load testing process. The problem should have been caught on QA and not on Production. IBM does not have control on the quality assurance process adopted by GSIS. It appears to me, the project manager of the DB2 upgrade project is at fault here.
    roelt
  • No it would not be the IT company or contractor

    It's always the company that's the problem, they are the computer experts, and how can we expect the likes of IBM to know about stuff, like DB2 and so on.

    Thats right it's never the fault of the OEM, or IT experts.

    The customer should have been all Phd's in computer science.. after all are they to expect the likes of IBM to know about computers and systems... !!!!! Oh Wait !
    Aussie_Troll
    • That was a dumb response...

      If you read the article, the government contracted with a systems integrator, who bought SAP and IBM DB2.

      For a large integration project, the integrator should have had the common sense to bring in support from IBM and SAP to minimize risk. This is standard procedure for large projects!

      Since the project manager / systems integrator didn't do that... they assume the risk for the project.

      After all... it wasn't IBM or SAP that made the claims to live up to spec, and then implemented it... were they?
      dominigan
  • conceptually talking.

    if i am unable to do a quick search using:
    select * from table1
    union
    select * from table2
    union
    ...
    select * from table "n"

    (where table "n" is thousand of gb) then i will put the blame on the database.

    :-P
    magallanes
    • You must not know much about databases...

      ...because table design will make or break database performance. We haven't heard anything about table design (why such large data transaction size in a single table?) or indexing schemes.
      dominigan
  • RE: IBM's Devil's Triangle: An enterprise software soap opera

    Gigabytes/Terabytes?

    Exactly how many government employees current and former could the Phillipines have?

    mcp123
  • RE: IBM's Devil's Triangle: An enterprise software soap opera

    Amen and Amen on IBM living up to the sale expectations.
    Ed Tech Girl
  • Devil's Triangle: Wrong metaphor

    By implication I take it you mean that a "classic devil's triangle" means a clash of three entities. Actually, the Devil's Triangle is a geographic area in the Atlantic Ocean.
    Pokeyman2