How do we know when IT gets it wrong?

Summary:Victoria's recently announced review of speed cameras had me think about trust. We need technology to make society manageable given the high number of people living on the planet. But how do we know it's doing its job right?

Victoria's recently announced review of speed cameras had me think about trust. We need technology to make society manageable given the high number of people living on the planet. But how do we know it's doing its job right?

Were we really speeding? Or was some mistake put in by a coder ten years ago hitting us with a fine even if we were trundling along granny-style?

Did that government really win the election? Or did the e-voting system get the counting wrong, or accidentally leave out all votes from people whose surnames started with T?

Was the list being used to filter the internet from child pornography accidentally replaced with another longer list because of an innocent (or perhaps not so innocent) file switch?

Did the bank's systems really calculate the interest on my savings account correctly? Or did some glitch mean that it only really gave me 3.75 per cent instead of 4?

Is the aeroplane really flying at that angle to the horizon or was the reading just an anomaly?

We rely on computers to do our work and, sometimes, our thinking for us. We don't have the time or the manpower to do all of these calculations ourselves. Imagine if every speeding fine required a policeman to tail you, clocking your speed with his speedo. It was once so, but there are so many more cars on the road now.

Some modern aircraft can't even fly without computational aid. No human can act quickly enough to keep the inherently unstable designs flying on an even keel.

But computers don't have "hey, wait a second, that doesn't sound right" moments. There's no common sense built into the chips.

The computer doesn't say, "30,000 feet, 30,000 feet, 30,000 feet. Whoa, 40,000 feet! But we haven't actually moved. Don't be stupid! Let's ignore that reading. 30,000 feet". It says, "Oh god. I'd better sink my altitude quick smart by 10,000 feet". Or "Mayday! Turn off the autopilot".

The worst thing is that many humans have been conditioned to always trust the computer.

There's a scene in a science-fiction book called Freedom by Daniel Suarez (sequel to Daemon) where a character orders food at McDonald's. The cashier tells him the price, but the character knows it's wrong. So he tells the cashier he's not paying that. The cashier flips the screen around and shows the character the number, saying "the computer says it, see?"

The character spends a long time trying to convince the cashier that if he uses simple arithmetic, he'll realise, looking at the prices on the board behind him, that the computer's wrong. But the cashier won't believe him. He's too convinced that the all-knowing computer must be right.

This is a concern, knowing that it is, after all, humans which have set the rules by which these computers live; and humans err. Think about NASA losing a Mars orbiter because it was using metric units while partner Lockheed Martin was using imperial units.

So I'm pleased that the Victorian Government is doing a review of its speed cameras. Because, yes, we definitely need technology, but we need to keep our eyes open or it could hang us by rubber-stamping our mistakes.

Topics: Censorship, Government, Government : AU, Travel Tech

About

Suzanne Tindal cut her teeth at ZDNet.com.au as the site's telecommunications reporter, a role that saw her break some of the biggest stories associated with the National Broadband Network process. She then turned her attention to all matters in government and corporate ICT circles. Now she's taking on the whole gamut as news editor for t... Full Bio

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.