How do we know when IT gets it wrong?

How do we know when IT gets it wrong?

Summary: Victoria's recently announced review of speed cameras had me think about trust. We need technology to make society manageable given the high number of people living on the planet. But how do we know it's doing its job right?


Victoria's recently announced review of speed cameras had me think about trust. We need technology to make society manageable given the high number of people living on the planet. But how do we know it's doing its job right?

Were we really speeding? Or was some mistake put in by a coder ten years ago hitting us with a fine even if we were trundling along granny-style?

Did that government really win the election? Or did the e-voting system get the counting wrong, or accidentally leave out all votes from people whose surnames started with T?

Was the list being used to filter the internet from child pornography accidentally replaced with another longer list because of an innocent (or perhaps not so innocent) file switch?

Did the bank's systems really calculate the interest on my savings account correctly? Or did some glitch mean that it only really gave me 3.75 per cent instead of 4?

Is the aeroplane really flying at that angle to the horizon or was the reading just an anomaly?

We rely on computers to do our work and, sometimes, our thinking for us. We don't have the time or the manpower to do all of these calculations ourselves. Imagine if every speeding fine required a policeman to tail you, clocking your speed with his speedo. It was once so, but there are so many more cars on the road now.

Some modern aircraft can't even fly without computational aid. No human can act quickly enough to keep the inherently unstable designs flying on an even keel.

But computers don't have "hey, wait a second, that doesn't sound right" moments. There's no common sense built into the chips.

The computer doesn't say, "30,000 feet, 30,000 feet, 30,000 feet. Whoa, 40,000 feet! But we haven't actually moved. Don't be stupid! Let's ignore that reading. 30,000 feet". It says, "Oh god. I'd better sink my altitude quick smart by 10,000 feet". Or "Mayday! Turn off the autopilot".

The worst thing is that many humans have been conditioned to always trust the computer.

There's a scene in a science-fiction book called Freedom by Daniel Suarez (sequel to Daemon) where a character orders food at McDonald's. The cashier tells him the price, but the character knows it's wrong. So he tells the cashier he's not paying that. The cashier flips the screen around and shows the character the number, saying "the computer says it, see?"

The character spends a long time trying to convince the cashier that if he uses simple arithmetic, he'll realise, looking at the prices on the board behind him, that the computer's wrong. But the cashier won't believe him. He's too convinced that the all-knowing computer must be right.

This is a concern, knowing that it is, after all, humans which have set the rules by which these computers live; and humans err. Think about NASA losing a Mars orbiter because it was using metric units while partner Lockheed Martin was using imperial units.

So I'm pleased that the Victorian Government is doing a review of its speed cameras. Because, yes, we definitely need technology, but we need to keep our eyes open or it could hang us by rubber-stamping our mistakes.

Topics: Censorship, Government, Government AU, Travel Tech

Suzanne Tindal

About Suzanne Tindal

Suzanne Tindal cut her teeth at as the site's telecommunications reporter, a role that saw her break some of the biggest stories associated with the National Broadband Network process. She then turned her attention to all matters in government and corporate ICT circles. Now she's taking on the whole gamut as news editor for the site.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Which is why you ought to have two or more different measuring systems (for the plane altitude example) for it to use. (eg radar and barometric?)

    For an example of IT chaos, take a look at what happens in a supermarket when the scanning system goes down. I would have thought there'd be failover systems, but apparently not.
  • You're absolutely right!

    Basically, I think it comes down to the following:

    "whilst having the glories of technology assisting, entertaining, and monitoring our daily lives is great, it is vital we keep our brains alert and concentrating on the here and now to keep things in check."

    Too many people have become lazy in their reliance on technology, and as such, sometimes the intended simplifying of our lives through the implementation of technologies has the complete opposite effect, causing accidents of all kinds.

    I love my gadgets but I am also vigilant about being aware of my surroundings; like how fast I'm going or whether the light is green, orange or red.

    Technology doesn't always get it right, and when this ends up costing, either financially, emotionally, or with our lives, it is a clear example of why we should never become complacent.
  • GIGO - remember - my version of that old acronym is "Garbage IN, Gospel OUT"!
    (Not what it used to be.) Now, once upon a time Australia became a signatory to the "Common Criteria" software / hardware evaluation scheme (International Standard IS 15408. Critical software systems. like those in speed cameras, need to be certified by an independent body to test that they do what they claim to do, no more, no less. Governments even used to mandate such testing/evaluation for their ICT products BUT today ?
    Well, yes, we should not become complacent - and that includes, in particular - our governments at all levels. For trust purposes we need an independent evaluation of those products - just like we have for pharmaceuticals, etc. Why not ICT products and systems? The standards have been around for - wait for it - almost 30 years but like most other security/reliability areas cheap, commodity based products have become the norm - and no-one really cares!
    Simple - for such products as those cameras - software evaluation under agreed standards MUST BE MANDATORY, particularly for governments "in whom we trust."
  • 1410 - I think the word you are looking for is "calibration", meaning "checking against a known standard". Those of us involved in measurement on a daily basis (for my part, I am a land surveyor) it is an intrinsic part of what we do. Unfortunately professional measurers are few in number, and the bulk of the population has little idea of the necessity of checking/verifying every measurement to ensure its validity. Neither do many have much understanding of the inherent errors in ALL measuring systems, no matter how technologically advanced. No matter how a measurement is achieved, by complex electronics and software (as in speed cameras) or by the simplest of devices (a carpenter's tape), there must be a means of calibrating the device, checking the measurements, and a statement of the errors which might be present. Evaluation of the software used to operate a system is not usually necessary unless a actual problem has been identified - it is the end result which needs to be precise, reliable and consistent with the standards. And, of course, in life-threatening situations (eg aircraft altitude measurements) there must be at least one backup system which is equally as reliable as the primary device. Operator/installer training is also a critical issue in the reliability of many systems. Too many people who have knowledge in one area of technology think they can move across to an allied area and become competent in using the software and equipment without much training or experience.
    I am sure that there are standards for calibrating speed cameras and radar guns (I came across a baseline setup in the backyard of a country police station once), but with these standards are applied rigorously and consistently is imposible to determine. The law here in Queensland apparently says that the measuring device is correct regardless - someone needs to challenge this!