Must Read: Sorry, but bringing back the Start menu won't help Windows 8

Topic: Data Management

JP Morgan Chase's IT failure: An apology and some informed speculation

Summary: Customers whacked by JP Morgan Chase's information system debacle---the bank's online banking site was down for three days---finally got an apology, but it's still a little fuzzy about what exactly went wrong.

Larry Dignan

By for Between the Lines |

Customers whacked by JP Morgan Chase's information system debacle---the bank's online banking site was down for three days---finally got an apology, but it's still a little fuzzy about what exactly went wrong.

JP Morgan Chase's online site crashed Monday night and stayed offline through Wednesday. On Thursday, the bank said:

We are sorry for the difficulties that recently affected Chase.com, and we apologize for not communicating better with you during this issue. Giving you 24-hour access to your banking is of the utmost importance to us. This was not the level of service we know you expect, and we will work hard to serve you better in the future and to communicate with you better if a situation like this should arise again.

Online Bill Payments scheduled for September 13, 14 or 15 were processed by Wednesday night, September 15. It is not necessary to reschedule these payments. If you scheduled a payment during those dates, but do not see it reflected in your payment activity by September 16, please contact us.

We will refund any late fees that you may have incurred as a result of our delay in processing your payment.

Thank you for your patience and for the opportunity to work harder to serve you in the future.

The big question here is what exactly went wrong. Computerworld notes that JP Morgan Chase blamed a third party database company's software. That third party is Oracle. Curt Monash, principle of Monash Research, offered some informed speculation in a blog post. Citing tips from readers of his DBMS 2 blog, Monash said:

  • A corruption in the Oracle database led to the outage;
  • The Oracle database stored user profiles;
  • Portals, ACH transactions, loan applications and trading portfolio access for private clients (ie the wealthy types) were affected.

The overall conclusion from Monash is that JP Morgan Chase wasn't to blame. More details are likely to emerge. Let's hope JP Morgan Chase does a thorough post mortem. These days banks are their online sites.

Topics: Data Management, Banking, Data Centers, Enterprise Software, Software

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

15 comments
Log in or register to join the discussion

  • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

    They must have been using linux.
    Loverock Davidson
    Reply Vote

  • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

    How could a corruption to the Oracle Database led to the outage? You were still able to access your account via phone. Is it not the same database?
    boneorchard@...
    Reply Vote

    • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

      @boneorchard@...

      The database that stores profiles and login information for the online banking site is almost certainly not the same database that contains your account information. When you go over the phone, they don't need to access your online banking account.
      gotamd@...
      Reply Vote

  • Maybe Chase was too busy playing w/ their iPhones to bother...

    with coordinating to get their online banking back up.

    "UGH!I'm F*king busy trying to synch my iPhone w/ our email! These people are just going to have to wait to pay their bills!"
    SonofaSailor
    Reply Vote

  • It was *NOT* Oracle.

    Where was their hot backup site? Pleeze!
    Smart_Neuron
    Reply Vote

    • It's possible they couldn't switchover

      @Smart_Neuron <br><br>If they switched over, it's possible they could have lost a time period of transactions, account maintenance, etc.<br><br>But you bring up a good point. About a year ago, a [i]major[/i] Bank's ACH functionality went down due to a power loss in the skyscraper that data ctr was located in...the bad thing was the contingency site was located a few floors up in the same skyscraper!
      SonofaSailor
      Reply Vote

  • Chase's continuing debacle

    Before deciding on a bank, especially one you intend to do online biz with
    * Call their customer service phone number to see how impossible they make it for you to speak to a human being who will take time to listen and understand a basic question before they answer from a script.
    Should their online service hiccup, you'll need these folks to be on your side.
    * Walk into a brick & mortar branch (yes, a novel idea, and only works with banks that have branches)...
    Talk to a teller. Can they answer the most basic question? (If not, don't blame him/her -- blame their training or lack thereof.)

    Anyone who has done this test with Chase should already know what they are getting into before they make "the Chase choice."

    I heard that during this 3-day outage, you couldn't even get past their phone menu apology to do any phone banking either -- unless you dialed their hearing impaired number.

    Out of the box thinking is something you'll have to do on your own if you go with Chase.

    Remember, these are the guys who are still putting people out of their homes while raking in huge profits by doing so.
    rroberto18
    Reply Vote

  • Or was it the Crappy Microsoft Web Technologies that tie it together

    Chase is one of the few major banks using the POS known as ASPX and IIS for their web site.....
    itguy08
    Reply Vote

    • Yeah, I don't imagine too many others are running IIS on Solaris...

      @itguy08

      http://toolbar.netcraft.com/site_report?url=chase.com
      SonofaSailor
      Reply Vote

    • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

      @itguy08

      Fail
      surfasb
      Reply Vote

    • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

      @itguy08 <br><br>ASPX and IIS stayed strong... Maybe they should use Microsoft SQL Server as a backend instead of the 6 times more expensive oracle db junk... You lose...
      mikes2nd
      Reply Vote

  • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

    grades - Chase = collusive failure for 3 business days down and
    = major failure not to have hot backup site and
    = biggest failure not informing their customers!
    web84403@...
    Reply Vote

  • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

    I know of another bank - an Australian major - who uses block-replicated data to two other distinct, geographically separate replicated sites. I've been through their architecture and they're /covered/. That said, hardware is only half the picture. Nothing to keep some loon from applying a shaped charge in the form of a bit of SQL code to shatter the organisation of their data. I'm a DBA of some 12 years (and an IT professional for about 40). Nobody intended it to fail, I suspect, but please believe -- these are each machines with several billion moving parts, and they can still out-clever us.
    NefariousWheel
    Reply Vote

  • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

    This submit was rather adequately printed, and in addition, it carries plenty of advantageous information. I appreciated your professed strategy of manufacturing this <a href="http://www.nfl-wholesaler.com/" style="text-decoration: none; color: black;">nfl wholesale</a> submit.
    jackson1984-24316069205748857739440257893812
    Reply 1 Vote

  • RE: JP Morgan Chase's IT failure: An apology and some informed speculation

    Great intriguing subject material. It should give instructional <a href="http://www.nfl-eshopping.com" style="text-decoration: none; color: black;">2012 nfl jerseys</a> insights to the web site internet site people like me.
    tomlin21-24319035676893835085146735905770
    Reply Vote
CNET Join | Log In | Privacy | Cookies Close