The spreadsheet love affair

The spreadsheet love affair

Summary: I never thought I'd go on a tear about Excel in these pages, but Josh Greenbaum's correct assertion that Excel is pretty much everywhere and is probably the software industry's most successful product provides the perfect foil. Josh concludes:So, like the floppy disk icon that never dies, the Excel spreadsheet lives on and on, despite advances in technology that should have buried it a long time ago.

SHARE:
TOPICS: Software, CXO
14

I never thought I'd go on a tear about Excel in these pages, but Josh Greenbaum's correct assertion that Excel is pretty much everywhere and is probably the software industry's most successful product provides the perfect foil. Josh concludes:

So, like the floppy disk icon that never dies, the Excel spreadsheet lives on and on, despite advances in technology that should have buried it a long time ago. This ubiquity and staying power says volumes about what users want from enterprise software, and their continued votes in favor of a 20-plus year old user experience should give everyone who believes that the best technology deserves to win a deserved pause. Excel works well-enough for millions of users all day long, and learning to live with it is a strategy that everyone, from CEOs to managers to software developers, needs to keep in mind.

I'm not going to disagree. As Josh alludes, there's very little point trying to roll rocks uphill. What Josh doesn't expose though are the risks that go with spreadsheet use.

Each year I pen a lament to spreadsheet use for my accounting colleagues, usually prefacing with some tale of spreadsheet woe. I have a bag full of them. Everything from the mortgage provider that overpaid some $270 million for a debt book, through to energy futures overpaid by $9 billion down to the $2 million a month interest calculation error. Heck, there's even an annual European spreadsheet risk conference. Ray Panko of the University of Hawaii has been tracking the issues for many years. On its website, the University notes:

Initially, spreadsheet research focused on errors, including typing errors, pointing errors, logic errors, and omission errors. More recently, regulatory compliance pressures have focused a great deal of attention on how corporations are doing financial reporting and other critical corporate processes. What they are finding is lots of spreadsheets, often hundreds arranged in manually-operated webs. Suddenly, spreadsheet error and security research is no longer "just academic."

It never has been 'just academic' but companies, CFOs and users seem so enamored of spreadsheets that their inherent dangers are often simply ignored.

I've always held the view that the spreadsheet was never designed for the sophisticated uses to which companies continue to put it. At best it is a development envronment that is rarely documented because users are not trained as developers. The net result is that when things go wrong, errors are notoriously difficult to find. What's more, there seems to be a fundamental lack of awareness around the extent of spreadsheet error. Some studies I've seen suggest it is as high as 95%. Panko's latest research asserts:

In general, errors seem to occur in a few percent of all cells, meaning that for large spreadsheets, the issue is how many errors there are, not whether an error exists. These error rates, although troubling, are in line with those in programming and other human cognitive domains. In programming, we have learned to follow strict development disciplines to eliminate most errors. Surveys of spreadsheet developers indicate that spreadsheet creation, in contrast, is informal, and few organizations have comprehensive policies for spreadsheet development. Although prescriptive articles have focused on such disciplines as modularization and having assumptions sections, these may be far less important than other innovations, especially cell-by-cell code inspection after the development phase.

The broader question then is why business continues to use this most elementary of tools instead of the wholesale embracing of specialist analysis tools and products? The only conclusion I can come to is that the spreadsheet is seen as convenient in a way that other applications are not and that the learning curve is sufficiently shallow for anyone to pick up the basics and do something useful. It's also cheap, often pre-installed on user machines at low cost in bulk deals.

In the meantime, the software industry continues to find ways of patching up an old horse that should, in my opinion, have been put out to grass a long time ago. And it doesn't go unnnoticed that the most popular topic of conversation on the UK's community site for accounting professionals is: the spreadsheet.

Topics: Software, CXO

Dennis Howlett

About Dennis Howlett

Dennis Howlett is a 40 year veteran in enterprise IT, working with companies large and small across many industries. He endeavors to inform buyers in a no-nonsense manner and spares no vendor that comes under his microscope.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

14 comments
Log in or register to join the discussion
  • That's obvious

    becuase it's not the "most elementary of tools", it's far, far more versatile then you want to give it credit for.
    AllKnowingAllSeeing
  • Santayana

    For those who weren't around at the beginning of the personal computer era, it's worth remembering that it all began because a PC was cheap enough that it could be purchased without going through Corporate channels. In particular, the unit management could get VisiCalc or Lotus 1-2-3 to run a budget without negotiating for six months (or longer) with Corporate Information Systems.

    Guess what?
    Yagotta B. Kidding
    • People were buying the Apple II to run Visicalc

      The history of spreadsheet deserves more exposure.

      Dan Bricklin's Visicalc had a huge disruptive influence on the
      future of PCs. People were buying Apple IIs to run this
      application.

      Other's copied it, and after many generations of copying (and
      a little withholding API information) MS's Excel became to be
      the dominant spreadsheet application today.
      Richard Flude
  • A spreadsheet is a swiss army knife

    You can get different programs to do individual tasks better, but whenever I find myself with a data gathering and/or analytical task that I've never done before or do infrequently, a spreadsheet is always the quickest, easiest, and cheapest solution. And it always works.
    Michael Kelly
  • General versus too specialised

    As other respondents have said, the answer does seem to be obvious: the specialised tools are too rigid in the functionality to support every need. Paying for specialisation to suit current needs is exorbitantly expensive, and the result will almost certainly be out of date by the time it's delivered.

    Why should a corporation tie itself to an inevitably expensive system which will only do a subset of what it needs?

    The humble spreadsheet can be used to do provide "good enough" functionality for whatever needs arise, and can be modified over time (or replaced with new when requirements changed). If accuracy and accountability are essential, they can be applied to the usage of a spreadsheet; there's no reason why testing cannot ensure the required level of robustness.
    Jason Etheridge
    • Testing in quality

      [i]If accuracy and accountability are essential, they can be applied to the usage of a spreadsheet;[/i]

      By turning your spreadsheet into a corporate development project, you've negated its advantages without buying those of more structured tools. C++, Java, Ruby, etc. all have features which aid developers in achieving robustness and reliability ("programming in the large").

      [i]there's no reason why testing cannot ensure the required level of robustness.[/i]

      Pardon me, but there speaks someone who has not only not been responsible for any kind of complex-system quality but has never even been exposed to the mathematics.

      There's a reason "testing in quality" is a conversation-killer.
      Yagotta B. Kidding
      • You missed the point

        "By turning your spreadsheet into a corporate development project, you've negated its advantages without buying those of more structured tools. C++, Java, Ruby, etc. all have features which aid developers in achieving robustness and reliability ("programming in the large")."

        We're talking about functions that can be performed by spreadsheets (number crunching, analysis, data representation), not arbitrarily complex distributed systems. We pick the right tool for the job; if Excel can do it, then why not?

        "Pardon me, but there speaks someone who has not only not been responsible for any kind of complex-system quality but has never even been exposed to the mathematics."

        Who mentioned complex systems? As someone who (successfully) has built and deployed several large-scale systems that are in use today, I have some experience in this area. Of course, that doesn't bear on the kind of limited-scale functionality that can be accomplished through Excel.

        "There's a reason "testing in quality" is a conversation-killer."

        You brought up quality in this context, not me. I was specifically talking about verifying the integrity of whatever was built using Excel by some level of testing, with the usual benefits that accrue from having a mechanism for ensuring that changes didn't break anything.
        Jason Etheridge
      • Spreadsheets ARE programming

        Spreadsheets, even without marcos, are really a form of computer programming. The spreadsheet was the world's first domain specific language and continues to be by far its most successful.
        Erik Engbrecht
        • And That's the Problem

          People who do not have the training, or even the mind set, of programmers are "programming" spreadsheets. Then decisions are being made or actions taken on based on the numbers they see. Sure, Excel is calculating accurately. But did anyone make sure that it's doing the RIGHT calculations? If the spreadsheet started as a quick, cheap way to solve the probelm, probably not.
          MichP
          • Just like any programmed tool

            the end result is only as good as the person creating the thing. But if you know that going in you prepare for it. Someone has got to LOOK at the end result before declaring it accurate.
            Michael Kelly
        • Functional programming

          Yep, spreadsheet use is the most widespread form of computer programming there is. The grid (which takes expressions only) is actually a functional programming interface - so the functional programming people win, there are more people using FP than any other programming paradigm...
          voidspace
  • I'm lovin' it ...

    "The broader question then is why business continues to use this most elementary of tools instead of the wholesale embracing of specialist analysis tools and products? The only conclusion I can come to is that:

    1. the spreadsheet is seen as convenient in a way that other applications are not and
    2. that the learning curve is sufficiently shallow for anyone to pick up the basics and do something useful.
    3. It???s also cheap, often pre-installed on user machines at low cost in bulk deals."

    These are tremendously powerful reasons for many users and, as in my reply to JG, I'd say M$ would do well to heed them. I'd lump VBA in the same category. Swiss Army knife - I liked that analogy - I used 'the 5-iron of programs'. Despite the danger you correctly outline there are times when you want to get something done and haven't the luxury of waiting 3 months for your project to be programmed by the expert in IT development. Not all jobs are complex.

    If I were M$ I'd be building in more checking facilities and making VBA 'parallel'. Indeed I'd have a graphic environment for analysing the structure of programs and spreadsheets so that users could DIY (as EXCEL analyses its calculation precedence at present) in VS2008.

    Note also that some industries try to mix and match e.g. the Finance community have EXCEL front ends and major calculations offloaded to XLL's.

    Keep it simple I say - or you'll get things like VISTA.
    jacksonjohn
  • A proper programming model for spreadsheets? Resolver One

    I work for a startup in London. We're creating a product aimed at solving the 'problem of spreadsheets' by bringing a structured programming model to them - yet retaining the familiar (and damn useful) spreadsheet interface.

    We've just released version 1.1:

    http://www.resolversystems.com
    http://www.resolverhacks.net

    It is aimed at technical users and for creating business applications - which is what spreadsheets actually get used for...

    Michael Foord
    voidspace
  • It is a great tool, but

    it is not always easy to fix a helicopter with a shovel (forgive my allegory). My favorite example is perennially popular use of spreadsheet as a corporate reporting tool - I recall hours of bickering over the "correct" numbers at the sales forecasting meeting. People have started with downloading data from corporate SFA data base into Excel, then applied their modifications and edit to their models, and subsequently could not understand why their projections did not make any sense to anybody else. Sometimes common denominator and uniformity of reporting is very critical, and spreadsheet use can be very harmful. www.evolutionofbpr.com. I also left a comment on Joshua's post on this topic, which addresses "root" technologies issues as they affect enterprise software adoption management.
    GregY