40 IT failures caused by software bugs

40 IT failures caused by software bugs

Summary: Rick Hower, who runs the Software QA Test Resource Center has compiled a lengthy listing of "major computer system failures caused by software bugs" Here are several entries from that list:A September 2006 news report indicated problems with software utilized in a state government's primary election, resulting in periodic unexpected rebooting of voter check-in machines, which were separate from the electronic voting machines, and resulted in confusion and delays at voting sites.

SHARE:

Rick Hower, who runs the Software QA Test Resource Center has compiled a lengthy listing of "major computer system failures caused by software bugs" Here are several entries from that list:

  1. A September 2006 news report indicated problems with software utilized in a state government's primary election, resulting in periodic unexpected rebooting of voter check-in machines, which were separate from the electronic voting machines, and resulted in confusion and delays at voting sites. The problem was reportedly due to insufficient testing.
  2. In August of 2006 a U.S. government student loan service erroneously made public the personal data of as many as 21,000 borrowers on it's web site, due to a software error. The bug was fixed and the government department subsequently offered to arrange for free credit monitoring services for those affected.
  3. A software error reportedly resulted in over-billing of up to several thousand dollars to each of 11,000 customers of a major telecommunications company in June of 2006. It was reported that the software bug was fixed within days, but that correcting the billing errors would take much longer.
  4. News reports in May of 2006 described a multi-million dollar lawsuit settlement paid by a healthcare software vendor to one of its customers. It was reported that the customer claimed there were problems with the software they had contracted for, including poor integration of software modules, and problems that resulted in missing or incorrect data used by medical personnel.
  5. In early 2006 problems in a government's financial monitoring software resulted in incorrect election candidate financial reports being made available to the public. The government's election finance reporting web site had to be shut down until the software was repaired.
  6. Trading on a major Asian stock exchange was brought to a halt in November of 2005, reportedly due to an error in a system software upgrade. The problem was rectified and trading resumed later the same day.
  7. A May 2005 newspaper article reported that a major hybrid car manufacturer had to install a software fix on 20,000 vehicles due to problems with invalid engine warning lights and occasional stalling. In the article, an automotive software specialist indicated that the automobile industry spends $2 billion to $3 billion per year fixing software problems.
  8. Media reports in January of 2005 detailed severe problems with a $170 million high-profile U.S. government IT systems project. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. In March of 2005 it was decided to scrap the entire project.

While the list doesn't have links or references out to original sources, it does look well-researched. Here's a link to the entire list.

Update 10/2/07: I had taken this list from another website, which had incorporated it without attribution to the Software QA Test Resource Center. The reference and links have now been updated to reflect the true source of the list. My apologies to Rick Hower for the incorrect source attribution in the original posting.

Topics: Software, CXO, Government, Government US, IT Employment

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

15 comments
Log in or register to join the discussion
  • Dangerous logical bugs in widely used products

    A while back the Life Insurance company debited my account twice in the same month (luckily I had enough money on the account cover it).

    A large US supermarket chain debited a large number of customers debit and credit card transactions more than once.

    What these problems have in common is the same transactions being duplicated.

    I suspect that the route of these problems lies in SQL generating duplicate rows.

    Generating duplicate rows is a logical bug in SQL. According to relational theory, data should only be represented as relations, and as a relation is a set of tuples (rows, roughly in SQL speak) there should be no duplicates (as we all know a set cannot have duplicates).

    We should call on vendors and the SQL standards committee to fix this bug as soon as possible.
    jorwell
    • duplicate rows

      Last I checked primary keys and unique indexes worked fine in most SQL databases. It's just that many developers choose not to use them because "they cause errors" and "make it hard" to manipulate data.
      Erik Engbrecht
      • I'm not talking about tables

        but about queries, which should also not contain duplicates; primary keys and unique indexes do not protect you against this.

        The result of any query on a relational database should be a relation. In SQL this isn't the case and the consequences in practical terms are serious and a major cause of bugs.

        I suspect the reason duplicates exist is for performance reasons (duplicate elimination is apparently relatively expensive). However we all know premature optimisation well known source of bugs.

        I refer those who dismiss the dispute about duplicates as "a religious issue" to the Deutsche Telekom customer who received the same letter several hundred times after moving house.
        jorwell
        • Set Theory Vs. Engineering

          I wouldn't dismiss the issue. If we are going to be so precise as to distinguish
          between tables and relations (a data structure and a mathematical concept), I
          would point out that all useful rdbms support the "SELECT DISTINCT" construct,
          which means either the administrator has overlooked this or they don't trust the
          underlying data integrity and would err on the side of not dropping versus
          duplicating,

          As to SQL not generating relations, it does intermediate between Dr. Codd's
          application of relations and set theory to data persistence and information
          retrieval and the practicalities of solving problems which are np or O(n!) if done
          the obvious way. Indexes are not necessary under the relational model, because
          every row is supposed to be unique. Yet indexes are part of practice because it
          allows for n log n sorting and log n retrieval.

          While some of the problems above may be laid at the feet of incorrect
          normalization, the ages old conflict between analysis and engineering, and
          accumulated data cruft, some of it may be attributable to bad algorithms,
          misunderstood concurrent accesses, and my supposition that software writers
          provide for the easy things and users, when faced with real world complexities,
          bash round pegs into square holes and somewhere a dragon emerges from an
          egg.
          DannyO_0x98
          • Bad Algorithms...

            are the cause of easily 90% of the bugs I encounter on a daily basis. One thing that amazes me is that even professional software companies don't utilize code sharing to anything close to its potential. I've looked inside the same application and found three or four different methods used for retrieving data when a well architected application would utilize a single data manager type class through which all communication to the backend database would flow. This kind of code currently exists in countless Tier-2 ERP systems, mainly because they hire just about anyone who can type a line of VB code just to hae a warm body. If you try to build a house on a garbage foundation, your house is going to crumble. The same can be said of building an application. It's not called architecture for nothing...
            jasonp@...
          • I suspect the reason is historical

            Maybe on 1970s hardware, IBM engineers couldn't make duplicate elimination run fast enough and so kludged SQL so it returned bags instead of sets; it is unfortunate that we still have to live with the kludge.

            On the other hand if were to write

            SELECT gender from PERSON

            against a 10 million row table you would get 10 million rows back from SQL and 2 rows back from a relational DBMS, so what you "gain" in the database can bring the network to its knees.

            As for distinct, well yes, it should probably be standard practice to put distinct in every SQL statement; hands up all that do this? On the other hand why not implement the DBMS properly and get rid of duplicates altogether?
            jorwell
  • These are custom ...

    developed apps. But what is just as bad are the off the shelf apps.

    The software industry, including OS's, has gotten in the habit of letting the consumer debug their products. Microsoft and Intuit come to mind. They should be held financially responsible for the extra time and costs the consumer has to shell out from the time the software is released until they get it right. Unfortunately, about the time the software becomes what it should have been when released, we get a new version and start the maddness all over again.
    bjbrock
    • Don't buy their products

      This is why I really think large software acquisitions should be leases and not buys. Front-loading the payments creates all the wrong incentives. The funny thing is most vendors are relatively happy to offer a leasing arrangement, it's buyers that reject it.

      Consumers should just be educated and stop buying the products.
      Erik Engbrecht
      • When you have moved to Quick Books...

        nd use it for two years, even if the next version they literally force you to buy is buggy, it's hard to think about switching a second time.
        bjbrock
      • How is this different from SaaS?

        In effect, aren't you describing the software as a service model?
        mkrigsman@...
        • Not really

          You can lease licenses for "traditional" software as well. What usually scares people off is the threat of being left without the software. If you buy perpetual licenses then you can always discontinue maintenance but keep on using the software. If you lease licenses (sometimes called term licenses) then at some point you have to stop.

          But when you stop you're data is still all there within your firewall. You don't have to worry about obtaining it from a company that probably doesn't want to talk to you anymore.

          I personally see SaaS as "back-loading" your risk instead of "front-loading" it. You gain rapid implementation, but you lose a significant amount of control, reliability, and potential for deep data mining, and potential for customization.
          Erik Engbrecht
  • Yes, software does have bugs.. BUT

    could the bugs be related to flawed logic requested by the buyers?
    Been_Done_Before
    • Where does the developer's responsibility stop?

      If a developer is asked to do something he knows is problematic, shouldn't he raise the issue with the buyer?
      mkrigsman@...
      • Indemnification...

        You encounter this kind of thing all the time as a programmer-for-hire. Normally if you show the customer where the flaw in their design is and how it will manifest in real life situations, they fix the design. I've run into a few occasions where a customer was unflinching in their design, explaining that they would change another process to account for the flaw. In these cases, my concerns were documented along with a signed piece of paper from the required project leader stating that any problems resulting from this design flaw were not my responsibility. It's my firm opinion that when people are paying the kind of money they pay me to come in and work, they want someone who can think. Of course, if the design flaw is truly a flaw by design and could be used for any illegal activity, all bets are off. Surprisingly, I've only been approached once for something like that. I wasn't surprised to see the guy arrested for some payroll scam a couple years later.
        jasonp@...
  • RE: 40 IT failures caused by software bugs

    Not to mention the ongoing problems with the payroll system in the Los Angeles (CA) Unified School District, blamed on problems with new payroll software. Thousands of teachers are being underpaid, unpaid, and even some overpaid, and the District has no estimate when the problems might be resolved.
    jtlogan9694@...