RBS Bank joins the IT failures 'Hall of Shame'

RBS Bank joins the IT failures 'Hall of Shame'

Summary: An IT failure hits customers in the worst way possible.

SHARE:
TOPICS: Banking, CXO
21

Most IT failures go unnoticed by the public - backend systems break, projects go over-budget, or perhaps a IT misses an internal deadline. Such was not the case with this week's IT failure at the UK-based Royal Bank of Scotland (RBS), which, incidentally, has a balance sheet the size of the UK economy.

Also Read:
Key questions on the massive RBS / NatWest IT failure
RBS gives more detail on IT failure train wreck

Image credit: iStockPhoto

Image credit: iStockPhoto

According to Reuters, a bad software upgrade made RBS' systems unable to process payments for both individual and business customers. The upgrade failure also hit systems from British bank NatWest and Ireland's Ulster Bank. The BBC describes what happened and explains the ripple effect that occurred when the RBS systems went down:

The bank says there was a failure in the computer system that carries out the overnight transfer of money between accounts.

So, even though payments had been made - such as a business paying wages to staff - this did not show up on their account balances. In turn this meant many customers could not make payments themselves, such as paying rent to their landlord.

The failure effectively caused a traffic jam in the system. It created a huge backlog in updating account balances, which the bank has been trying to clear for some days.

Customers of other banks have also been affected because some payments from RBS, NatWest and Ulster Bank customers have not come through

In what must be an IT failures understatement of the year, RBS chief executive, Stephen Hester, commented:

It shouldn't have happened and we are very sorry

The BBC's Business Editor speculated that outsourcing arrangements interfered with the bank's ability to isolate and solve the problem:

As I understand it, one reason why RBS has not given much detailed information about why its services have been so badly disrupted is that so much of the operational responsibility for IT is outsourced - so there is a sensitive issue of where to attribute blame.

In my conversations with RBS bankers, there is an implication that outsourcing contributed to the problems - though they won't say whether this is an issue of basic competence or of the complexities of coordinating a rescue when a variety of parties are involved.

Adding to the confusion, Information Age reports that the problems were entirely self-made within RBS:

An RBS spokesperson said this morning that the technical fault, to her knowledge, had taken place on RBS' own systems, and not those of a supplier or outsourcer.

Although the problem upgrade occurred on Tuesday night and was fixed by Friday, downstream effects will continue through this week. This IT failure continues to have dramatic negative impact on bank customers, many of whom do not have access to their own money.

Also read: ZDNet UK: Natwest, RBS customers hit by balance glitch BusinessWeek: RBS Systems Failure Unlikely to Be Resolved Until Monday London Evening Standard: NatWest chaos heads into weekend Irish Times: Angry customers seek answers BBC: Ulster Bank 'needs week to clear IT failure backlog' The Telegraph: NatWest computer glitch 'fixed but backlog remains'? NatWest: Helpful Banking

A comment thread, containing thousands of messages, on the bank's online customer service forum describes travelers stranded abroad, home purchase closings that did not go through, and similar stories of difficulty due to customers not having access to funds. Ireland's Minister for Social Protection said that up to 30,000 people did not receive social welfare payments, even though funds had been withdrawn from government accounts. To assist customers, RBS opened 1200 bank branches on Sunday.

A SHAMEFUL PERFORMANCE

Although IT failures happen, this impact of this one is huge by any standard and several important questions remain unanswered:

  • What actually happened? Was this an outsource problem, an in-house problem, or maybe the RBS technology organization is so complex that it's impossible to isolate exact cause.
  • How can one of the world's largest banks roll out an upgrade without sufficient testing? No matter how you look at it, the upgrade created unexpected issues upon deployment. That's why we test, test, and test again, even though it appears RBS did not test enough.
  • Why the long and difficult recovery? Presumably, a bank this size has worked out its business continuity and recovery plans with a level of efficiency only possible for a huge organization. Or maybe not.
  • What accountability does the government hold over banks that evince such apparently poor IT processes as this? IT failures do not just happen randomly or in a vacuum. I suspect process and technology issues will emerge as warning signs that management and technical workers ignored.

THE BOTTOM LINE

This IT failure is sad for RBS and far worse for its customers. I urge the UK government to conduct a detailed review of RBS management, business, and technical processes with respect to all aspects of IT. It appears that RBS did not follow important standard practices in areas such as testing and business continuity planning.

Government regulators should hold RBS to account with stiff fines and other punishments as appropriate. Regulators should treat this IT failure with the same severity and determination they would bring to operational failures in any other part of a major bank.

Topics: Banking, CXO

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

21 comments
Log in or register to join the discussion
  • Who are the outsourcers involved?

    I remember seeing an HP outsourcing presentation listing RBS as their "reference." Bet they are busy trying to delete those slides ...
    terry flores
  • Maybe now...

    ...banks will start hiring properly - like basing salaries on skills and not on salary history? Like understanding that some things should always be kept in-house? Like the true cost of outsourcing should include liabilities in case of failures? Like you can't choose who to lay off based on how much you'll save on salary and benefits? Like if you can't commit enough to hire someone full-time instead of contract you can't ask them to commit to their work?
    Doubt it - it'll take at least dozen of failures like this i'm afraid.
    vgrig
  • Taxpayer picks up tab

    The UK government owns 84% of RBS following their meltdown in the subprime crisis.
    It's a good job we can afford to pay the high salaries needed to attract world class managers. I hate to think what could have happened. (Irony ends)
    Dax1792
    • Hester is still at RBS

      despite not getting his bonus.

      Surely he should have been off like a flash to foreign shores, or have we called the bankers' bluff?
      jorwell
      • Or play the long game?

        With his recent record is he unlikely to see CEO at another major company, play the long game and wait for the government to sell its share and collect massive bonuses then. For what its worth the entire cabinet of the last government voted for his package which included this bonus for targets which he hit.

        I think the lesson here is that if you offshore jobs to make a buck thats fine if you can maintain quality, however at the salaries they were offering in India (low even by Indias standards) you get what you pay for
        the.nameless.drifter
      • Outsourcing

        @the.nameless.drifter

        Couldn't someone in India do Hester's job for a fraction of the price?
        jorwell
      • You're right

        But management never outsources itself and if you pay very low wages you'll attract low talent staff.

        With wage inflation running at what it is India and China are becoming less attractive (it'll be Brazil and Chile next) but still viable, I don't like outsourcing but see why businesses do it. Whats unspeakably dumb is paying so little that you don't get reasonably talented staff in those countries and thats where you get the horror stories like RBS.
        the.nameless.drifter
  • For a fuller explanation

    http://www.guardian.co.uk/technology/2012/jun/25/how-natwest-it-meltdown

    What puzzles me is why it took three days to restore the corrupted scheduler files.

    The other lesson learned is that it is not enough to test the software, you have to test the installation too. You can get the software working in a test environment, but if you don't test the steps to get it onto production then you end up with the wrong software or configuration on the live system.
    jorwell
    • If you pay bottom rate you get poor people

      Outsourcing to India can be good if you pay the market rate in India however if you pay massively below the market rate (which it looks like they were) then you end up with people who are fine so long as nothing goes wrong and are useless in a crisis.

      Batch skills on z/OS are not exactly a popular skill nowadays so finding cheap people in India who will be able to manage roll backs and know how to restore corrupted files from backups on a TSO session won't be easy to come by

      Agree completely on testing and always recommend to people that they have acceptance testing on a identical system before deployment to production
      the.nameless.drifter
    • Thanks for the link

      I will probably write an update based on the article you mentioned. I'm talking with a business reporter shortly who is also following this story up.
      mkrigsman@...
      • What about disaster recover and backups

        In the US following 9/11, we have to follow certain disaster recover policies. Many companies have redundant sites. Does this take place in the UK and why so many days of downtime if you have backups?
        Jazzy Kat
  • software-RBS failure

    Could you please outline the operating system and types of transaction software used in this failure?
    The Management consultant
  • RBS SW failure

    It always happens that there is a Technical Manager having all the high level access giving all instructions for tasks and that person is usually partof the core team in IT services for that company. Only routine work with minimal impact is given to offshore.

    I don't see how offshore is to be blamed. All the testing and rollout decisions are taken at the corporate level.
    srinivas_107@...
  • Tip of the ice berg

    When I left university 15 years ago and joined a large telco in the UK all software was undertaken in house and if items of work were off shored only stuff that were deemed low risk were sent. We had real skills in house and people really did know what they were doing. Since the late 1990s the entire software profession has been destroyed by the bean counters where they see people in the UK as just extra cost and do not value their skills. When I left university, IT and the Software industry was seen as a real career path but sadly nowadays hardly any people are going into the software and engineering industry in the UK and I dare say the US. Where I work most of the skilled people have either retired, have left or have been forced out by constant performance management down grading by HR to meet senior management bonus targets. Our entire infrastructure is now developed and tested off shore and in most cases deployed (by people on intra company Visa's from India). On the UK side the programme managers manage everything and to be honest all they do is tick boxes and don't really have a technical clue what's going on as long as the meet their delivery time scales. We really are running on borrowed time. The senior management are just not bothered as long as they meet their objectives. I think this is an industry problem and the case of RBS, Sony and Blackberry (RIM) will just the norm in the future until we start investing in people in the UK and US instead of sending / trusting the crown Jewells off shore where there is massive staff turnover every six months and their rates keep on increasing once they know your skill base has gone. I know what the answer will be.... bring on more project managers to manage them but manage who and what as the skills are not there either. But I suppose as long as the senior managers keep on getting their pay rises and bonuses nothing will change,
    rrwfatr
  • Why Blame India / Outsourcing ?

    Why Blame India / Outsourcing ?
    JamesBond001
  • It's all too easy for errors to slip through

    My first job was with a large company. My first real "production change" was a simple code change. All of the company procedures (which over time I've learned were "world class") were followed. The code was tested, walked through by an analyst, the change was approved by the right people, the change was implemented on the weekend and tested again. All good.

    By Monday lunch time clients started calling in questions. By 3 PM we took the system off line. The final problem, when the program was compiled for production one uncompressed database definition was used in error. So the program was writing to the wrong places in the DB. Oops.

    The problem was caused because I was a newbie who didn't know that "quirk" in the system, and my analyst was away. His replacement also didn't know that quirk.

    Yes, management had the "one computer programmer is the same as another" frame of mind.

    Same company, years later, a long term contractor was let go. No big deal. Except that she had over 10 years experience in our company. She was an excelent, knowlegable analyst. Why was she let go? Personality conflicts with her new project manager!

    As long as we as consumers and shareholders allow management (and boredes of directors> to manage for quarterly profits and annual BONUSes they are paid we will continue to have this sort of mismanagement for short term profit at the expense of long term growth and stability.

    Same company was on a "down sizing" jag. They cut a bunch of people. One "20-year" man who was pensioned off was a 1-man department. Not very visible, not significant to the day to day operations of the front line. He managed disk space usage on the mainframe. Space that we rented from an outsourcer.

    Within a few months, we "ran out" of unused space and had to pay more than his salary to expand the storage space we were renting. Simply because Sr. management that was "optimizing" employees didn't know exactly what he did and that no one was trained to replace him. The significant amount of disk of space that he "reclaimed" in his daily work, by forcing people to delete unneeded files, no longer was freed up.
    Ron_007
    • con't

      I forgot to say, the system I broke: our Payroll system! oops.
      Ron_007
  • Outsource management

    As long as profits and losses rule the day, IT management will never understand a IT technicians true worth. That is why many companies almost always promote someone from the sales team to executive positions rather than the guy on the ground floor. They are seen as rainmakers and grunts work for them. I had a manager tell me for the money they pay us he would rather have a new computer at his desk at the ready when the old run is not working. My reply, okay then how will you configure this new computer for the companies security and software settings?
    Jazzy Kat
  • Red Bottoms Shoes

    This post has been somewhat of a revelation to me.I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you.
    http://www.discountchristianlouboutinblog.com
    http://www.christianlouboutincheapc.com
    feitian01