ie8 fix

Between the Lines

Larry Dignan, Andrew Nusca and Rachel King

The Terminator scenario: Perhaps not so fictional after all

By | February 20, 2009, 11:01am PST

Summary: Sometime in the future–Military robots have joined forces and are targeting humans using Google Latitude. Tribes of robots have banded together in various metropolitan areas. Early reports indicate a programming malfunction.   Sounds like Terminator doesn’t it? It could be real life some day if we don’t get our robot programming ducks in a row. Military [...]

Sometime in the future–Military robots have joined forces and are targeting humans using Google Latitude. Tribes of robots have banded together in various metropolitan areas. Early reports indicate a programming malfunction.  

Sounds like Terminator doesn’t it? It could be real life some day if we don’t get our robot programming ducks in a row. Military robots need to be taught warrior code and ethics or the world could be in for a world of hurt, according to a report.  

That conclusion was included in a big report by Cal Poly researchers for the U.S. Department of Navy, Office of Naval Research. The report, which was detailed in a Times Online story on Feb. 16, contains a few interesting passages to ponder ahead of the weekend. 

Also see: Gallery: Armies of combat robots

According to the report:

The use of military robots represents a new era in warfare, perhaps more so than crossbows, airplanes, nuclear weapons, and other innovations have previously. Robots are not merely another asset in the military toolbox, but they are meant to also replace human soldiers, especially in ‘dull, dirty, and dangerous’ jobs. As such, they raise novel ethical and social questions that we should confront as far in advance as possible—particularly before irrational public fears or accidents arising from military robotics derail research progress and national security interests.

On the bright side, autonomous military robots would save lives since we wouldn’t risk human life. The problem: All of this software code, hardware and other odds and ends used to create our future soldiers may not mesh so well in the field. 

Here’s a hierarchy of a robot system today:

Any of those layers could lead to problems. As for me I picture these robots going rogue and tapping into Google (Skynet in this scenario). In any case it can’t be good. Here’s risk-reward scenarios outlined in the report:

Imagine the face of warfare with autonomous robotics: Instead of our soldiers returning home in flag draped caskets to heartbroken families, autonomous robots—mobile machines that can make decisions, such as to fire upon a target, without human intervention—can replace the human soldier in an increasing range of dangerous missions: from tunneling through dark caves in search of terrorists, to securing urban streets rife with sniper fire, to patrolling the skies and waterways where there is little cover from attacks, to clearing roads and seas of improvised explosive devices (IEDs), to surveying damage from biochemical weapons, to guarding borders and buildings, to controlling potentially hostile crowds, and even as the infantry frontlines.

These robots would be ‘smart’ enough to make decisions that only humans now can; and as conflicts increase in tempo and require much quicker information processing and responses, robots have a distinct advantage over the limited and fallible cognitive capabilities that we Homo sapiens have. Not only would robots expand the battlespace over difficult, larger areas of terrain, but they also represent a significant force multiplier—each effectively doing the work of many human soldiers, while immune to sleep deprivation, fatigue, low morale, perceptual and communication challenges in the ‘fog of war’, and other performance‐hindering conditions.

But the presumptive case for deploying robots on the battlefield is more than about saving human lives or superior efficiency and effectiveness, though saving lives and clearheaded action during frenetic conflicts are significant issues. Robots, further, would be unaffected by the emotions, adrenaline, and stress that cause soldiers to overreact or deliberately overstep the Rules of Engagement and commit atrocities, that is to say, war crimes. We would no longer read (as many) news reports about our own soldiers brutalizing enemy combatants or foreign civilians to avenge the deaths of their brothers in arms—unlawful actions that carry a significant political cost. Indeed, robots may act as objective, unblinking observers on the battlefield, reporting any unethical behavior back to command; their mere presence as such would discourage all‐too‐human atrocities in the first place.

Technology, however, is a double-edge sword with both benefits and risks, critics and advocates; and autonomous military robotics is no exception, no matter how compelling the case may be to pursue such research. The worries include: where responsibility would fall in cases of unintended or unlawful harm, which could range from the manufacturer to the field commander to even the machine itself; the possibility of serious malfunction and robots gone wild; capturing and hacking of military robots that are then unleashed against us; lowering the threshold for entering conflicts and wars, since fewer US military lives would then be at stake; the effect of such robots on squad cohesion, e.g., if robots recorded and reported back the soldier’s every action; refusing an otherwise legitimate order; and other possible harms.

Creating autonomous military robots that can act at least as ethically as human soldiers appears to be a sensible goal, at least for the foreseeable future and in contrast to a greater demand of a perfectly ethical robot. However, there are still daunting challenges in meeting even this relatively low standard, such as the key difficulty of programming a robot to reliably distinguish enemy combatants from non-combatants, as required by the Laws of War and most Rules of Engagement.

As I leaf through this report I can’t help but think of malfunctions and robots turning against us. 

How do we avoid this potential Terminator scenario? New programming of course:

Serious conceptual challenges exist with the two primary programming approaches today: top down (e.g., rule following) and bottom up (e.g., machine learning). Thus a hybrid approach should be considered in creating a behavioral framework. To this end, we need to a clear understanding of what a ‘warrior code of ethics’ might entail, if we take a virtue ethics approach in programming.

And you thought multi-core programming was going to be tricky. Let’s hope programmers aren’t as stupid as some people think.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Topics

Larry Dignan is Editor in Chief of ZDNet and SmartPlanet as well as Editorial Director of ZDNet's sister site TechRepublic.

Disclosure

Larry Dignan

Larry Dignan has nothing to disclose. He doesn’t hold investments in the technology companies he covers.

Biography

Larry Dignan

Larry Dignan is Editor in Chief of ZDNet and SmartPlanet as well as Editorial Director of ZDNet's sister site TechRepublic. He was most recently Executive Editor of News and Blogs at ZDNet. Prior to that he was executive news editor at eWeek and news editor at Baseline. He also served as the East Coast news editor and finance editor at CNET News.com. Larry has covered the technology and financial services industry since 1995, publishing articles in WallStreetWeek.com, Inter@ctive Week, The New York Times, and Financial Planning magazine. He's a graduate of the Columbia School of Journalism and the University of Delaware.

For daily updates, follow Larry on Twitter.

34
Comments

Join the conversation!

Just In

What About the 7 Dwarfs Theory?
Timewellwasted 7th Mar 2009
Not intending to harsh your buzz there dude but.... These Three "Laws" would never be implemented! You are talking about a fictional piece of literature as if though it were remotely related to reality. Have you not noticed half of the world is trying to figure out how better to kill and destroy the other half of the world?
It is obvious we do not need to worry about an army of robots going postal on humanity anytime in the near future, and realistically we are probably 150 to 175 years (minimum) from having any form of AI "smart" enough to even comprehend the concept of the Three Laws of Robotics. The real problem with Isaac Asimov's Three Laws of Robotics is they are in direct contrast with the goals of the people funding the research! i.e. the military and government..
The problem with this Cal Poly report on Autonomous Robotics in a warfare scenario is we are nowhere near this type of technology. The drones and Predator type aircraft of todays military are barely smarter than the toy Remote Control Helicopter sitting in my garage. The real worry should be how many millions of dollars were spent by the Navy to fund the research into this far flung scenario. The Navy could have saved all of this money and simply visited the labs that are working on AI. The two "smartest" robots on earth are so fraught with problems it is almost laughable to think of them as a prototype for something that will be dangerous to anyone, that is unless the damn thing falls on top of someone.
Way too many people confuse remote control with something resembling artificial intelligence. No one on earth is near anything even resembling AI as it is portrayed in the Sci-Fi movies.
And to touch on your subject of a different line of discussion, it will be tens of thousands of years before humans will actually evolve even the slightest. As far as changing our thought process and cease making self-destructive war., until we (humans)come to the conclusion we all are truly equal and the concept of wealth is no longer relevant, we will always have war.
0 Votes
+ -
I don't buy into the Terminator scenario very much. The idea of robots
rising up in a coordinated effort is incredibly unrealistic at this point.
Though I will admit its possible it is unlikely.

What I don't doubt however is that these robots will be misused.
There are unpopular wars going on now and a lot of that unpopularity
comes from the death of soldiers and for good reason. What happens
when the government can guarantee that there will be no human
loses? More war.

You say that robots can be more humane because they do not suffer
from stress and fear. While a good thing it is also dangerous from the perspective that they do not care. They will kill, when ordered, no
questions asked. If anyone seriously thinks that the military will
acquire robots with a programmed conscience that might disobey if
given unlawful orders then you are fooling yourself. They will do
whatever is commanded of them.

You say that since robots are uncaring observers that they will be
calling back to base what is truly going on, without human bias. That
is a good thing for sure but do you really think that the command
center is then going to release unedited tapes about any incidents? Or
for that matter even say that it happened.
Having human beings involved is a good thing because they have
guilt. If something horrible does happen it eventually comes out. If all
you have are centrally controlled data files then good luck ever
getting the truth.

Hacking/Viruses/Bugs - These robots will be built with software and
human error or intentional cracking will occur. I find this aspect of
robots the most frightening. There will have to be a way for the
robots to call home and this point of entry will be exploited.

This technology will not stay in the US. If you think that the US is
going to rest on its robot laurels then you are in a fantasy land. The
last thing we need is a cold robot war.
I don't know about you but any unmanned weapons platform should
be opposed. I don't care if its controlled by a human or not but having
these weapons platforms in any form opens up a very steep slippery
slope. They should be universally banned.
0 Votes
+ -
Now(ish)
thomasmarshall3@... 20th Feb 2009
The swords platform pictured above is effectively an automatic target acquisition and dispatch device. It's "Warrior Code" is rather simple shoot the area around muzzle flash. Presumably it can in the near future trace ballistics on incoming fire to its source when flash suppression become effective in a target. What more do you need? If it shoots kill it. Everything else is recon, and demolitions, or a remote.

The point of the robot solider is not to make decisions but to complete tasks. You can make a machine that shoots anything that moves, but that should be clearly wrong even to an eleven year old. You can make something that kills everything humanoid, or humanoid and above a certain height (or below if you're a real sicko, not that above ain't sick). You could also make it kill everything humanoid and carrying a metal object, but there are strategic disadvantages, even if the ethics are "good enough" for war 1.0. Or you can shoot back from an armored platform with the kind of reflexes that bulls eye double digits (and even triple digits)each second. It's not rocket science, it's target filtration.
0 Votes
+ -
And...
thomasmarshall3@... 20th Feb 2009
You can program the logic at the chip level, and program another robot to shoot anything that tries to access the building the chip is in. There are less ethical options that possess tactical advantages, such as shooting anything that moves and is not carrying an appropriately encoded transponder device, which in turn can be sold to an enemy to allow passage through something like the Korea DMZ, and in turn affords tremendous intel gathering options. Shoot em till they die, then shoot em till they spy.
0 Votes
+ -
Ethics are a lot of fun to discuss, but given the level of AI we are at, these systems are not so autonomous that having them go out of control is an issue, yet.
0 Votes
+ -
Would future wars ever end if no human soldiers are harmed?
Custard_over_2x_Pie Updated - 21st Feb 2009
It's more likely that the military would lose sight of the original objective(s) because they enjoy winning so much. Or put another way, not much scope for losing.

Could you imagine this technology in the hands of a military dictatorship? or attempted military coup?

Yes it's at an early stage, but it will happen one day. And they won't bat an eyelid to sell this technology to the other side.
0 Votes
+ -
Short answer yes
frgough 21st Feb 2009
The side with the larger industrial capacity would destroy robots faster
than the other side could replace them. Then they would start killing
people. Then the other side would surrender and the war would be over.
0 Votes
+ -
An EMF pulse could take them out
Custard_over_2x_Pie Updated - 21st Feb 2009
simultaneously!

Also, I was thinking back to an article about how movie restoration still requires human intervention, because parts of an image that a computer can't fathom, such as, in the Wizard of Oz. Dorothy's sparkly shoes were different in every frame and the computer got confused.

So... just wear a sparkly suit, and you'll be able to get a few shots off before the robot works out what the heck is standing in front of it.

'Don't shoot me I'm only the piano player'
silly
0 Votes
+ -
The side...
eMJayy 21st Feb 2009
..with the larger industrial capacity isn't automatically guaranteed a win. It could never be that simple. Having a larger industrial capacity doesn't imply that the industry is not dependent on foreign resources to maintain its productive capacity. If, for instance, critical microprocessor technology is being outsourced, the enemy need only eliminate that outside source to cripple productive capacity.

It's also likely that cyber warfare would play major roles in determining the winner of such a war. If the supposedly weaker country hacks the stronger country's robotic arsenal, they could use that country's weaponry against itself. Could you imagine what would happen if robots were ever given the responsibility of handling nukes? Such robots could be just one hack away from launching Armageddon on their masters.
0 Votes
+ -
RE: Cyber war: So that's a Bot botnet then?
Custard_over_2x_Pie Updated - 22nd Feb 2009
Ultimately any sort of hi-tech solution is at the mercy of tiny details, which contribute to the whole endeavour. Any one of which could scupper their efforts, if not taken account of at an early stage in development.

Not unlike huge software project failures of recent times.
0 Votes
+ -
except
albeit 23rd Feb 2009
except they would not need to kill people to make them submit. they would all get tazed or incapicitated in some less harmful manner
0 Votes
+ -
Captain Kirk could save us...
Hallowed are the Ori 21st Feb 2009
...like he did Vendikar and Eminiar VII in "A Taste of Armageddon".. it was kinda the same thing.

All that would be missing would be the disintegration chambers.
0 Votes
+ -
Umm.. I can't remember many of those early trek episodes
Custard_over_2x_Pie Updated - 21st Feb 2009
They were memorable for all the wrong reasons.

Fighting almost every alien civilisation they encountered, or baking hot/frigidly cold alien planets.

Could you explain the episode? happy

0 Votes
+ -
Sure...
Hallowed are the Ori Updated - 22nd Feb 2009
Courtesy of IMDB:
********
On a mission to establish diplomatic relations, Kirk and Spock beam down to the planet to learn that its inhabitants have been at war with a neighboring planet for over 500 years. They can find no damage however and no evidence of destruction. They soon learn that the war is essentially a war game where each planet attacks the other in a computer simulation with the victims voluntarily surrendering themselves for execution after the fact. When the Enterprise becomes a victim of the computer simulation and ordered destroyed, Kirk decides it's time to show them exactly what war means.
******

He essentially destroys the war computers on Eminiar VII, abrogating the agreement with Vendikar, forcing each planet to either wage real war or to talk peace.

As I said, it's kinda the same thing... machines are doing the fighting... but people are still dying.
0 Votes
+ -
Thanks.
Custard_over_2x_Pie 22nd Feb 2009
I seem to recall a similar kind of scenario used in Stargate SG1.

Anyway, the Trek episode didn't jog my memory. But I get where you're coming from with your comment. happy
0 Votes
+ -
SG:A, actually
Da-G 23rd Feb 2009
The episode is called 'The Game', IIRC, and features McKay and Sheppard playing a RTS instead of chess or some other competitive game.

Only problem is, the Ancient idiots who built the thing actually meant the interface to 'guide' two separate civilisations, and what John and Rodney thought was a game was being enacted in real life, with themselves as demigods of a sort.

Looking forwards to the movie and SG:U.
Hell, while we are at it we should create our own Cylons. Lets see what they can do happy
0 Votes
+ -
Creepy
Cylon Centurion 22nd Feb 2009
:S
I think the threat of humans using robots to abuse and enslave other humans is more likely than the robots becoming independent and doing it on their own. I believe it is inevitable though. If we have second thoughts about developing these robots others will not
China, Russia maybe even Iran. Welcome to the Robo-Wars.
Far more likely is robot-executed genocide
0 Votes
+ -
The fools trying to build these things all concentrate on how great they are against human beings; but are totally ignorant about their performance when faced with opponents armed with the same technology.

AI war machines following the "rules of war" are still insufficient. AI's have no concept of mercy.

We need another Geneva Convention to outlaw the use of autonomous robotic war machines. They are equivalent to weapons of mass destruction.

Dr_Zinj, MSgt USAF (Ret.)
0 Votes
+ -
Geneva Conventions Amendments
fwjackson 23rd Feb 2009
What a great idea! Amendments to the
Conventions which outlaw military use of robots
is an outstanding course of action. Coupled
with the Three Laws of Robotics (see separate
blog entry), the Conventions Amendments would
lead to the elimination of war.
0 Votes
+ -
I Agree But...
lehnerus2000 23rd Feb 2009
Autonomous Robotic Weapons should be banned!

Gun toting robots (just like Biological and Chemical weapons) are only designed to kill people. These weapons are Weapons of Mass Murder.

Nukes (the only true WMDs) are designed to destroy infrastructure. People die because they are in the "immediate" area of the infrastructure (ignoring the effects of Fallout).

WMDs help keep the peace because politicians realize that they can be killed by nukes, if they declare war on another country which owns nukes (MAD). That's why the US & USSR never directly attacked one another.

For example there is no point dropping nerve gas on a factory if there are no workers in it, however it is still "worth" nuking it.

lehnerus2000
Why do people concentrate so much on not losing human life? War costs human life, it's unavoidable. That's why fights should be picked carefully.

I thought I was supposed to honor those in the military because they "risked their lives" in combat? If they aren't risking life and limb, then they aren't doing anything special. Pulling a trigger / pushing a button is not heroic.
0 Votes
+ -
As long as they don't run on Windows we're safe.
0 Votes
+ -
You know where I work anyone who still tries to make jokes about Windows is considered obsolete and useless. If that is all you can contribute to the discussion I hope you are paying for your own training..
0 Votes
+ -
Human Life
cliffdunaway 23rd Feb 2009
"On the bright side, autonomous military robots would save lives since we wouldn?t risk human life."

If our country attacks another country, kills untold "foreigners" but we suffer no casualties, that does not mean no human life was lost.

Hmm ... what's it called when one nation or people think their people are more valuable than all others? It's not a pretty word.
0 Votes
+ -
Blue on Blue incidents?
roger.scotter2@... 23rd Feb 2009
Just think about some of them blue on blue incidents where u.s.a patriot missiles shot down some allied planes (cant rember what iraq war it was)

sure you can give your allies and forces ionisation thingees to identify itself but things do go wrong here i will qoute a news report:

http://www.space.com/spacenews/archive05/Patriot_031405.html

"The Defense Science Board is an independent advisory committee that advises the secretary of defense. Michael Wynne, acting undersecretary of defense for acquisition, logistics and technology, commissioned the panel to report on the fratricide incidents during Operation Iraqi Freedom in June 2003 following two fatal accidents and one close call.



In March 2003, a Patriot missile destroyed a British Tornado fighter bomber, killing its two pilots. Two days later, a Patriot battery locked on to a U.S. F-16 aircraft. The following week, a Patriot missile struck a U.S. Navy FA/18 Hornet, killing the pilot.



The Patriot batteries were originally designed to shoot down aircraft rather than missiles, but their mission in Iraq focused on the latter. The report found that the crowded skies over Iraq made picking out ballistic missiles from aircraft a difficult challenge."
0 Votes
+ -
humane life
viztor 23rd Feb 2009
Think of Gaza for example...

Can a robot be prosecuted for war crimes?

A great disparity in power only encourages slaughter
rather than political solutions.

Predator pilots have stress levels about the same as
pilots flying missions in regular warplanes, according
to several articles.

There was an old cartoon of a military robot in the
midst of tanks and other weapons it had destroyed
asking a human general, "What do you want me do do
next, Stupid?"




0 Votes
+ -
It could all go horribly wrong
dorkiedorkfromdorktown Updated - 23rd Feb 2009
Interesting article...

It's entirely possible that robots join forces in the future and the only way to destroy them is to short circuit them with squirt guns or camouflage yourself behind a cardboard cutout of britney spears to "trick" the software until you can get close enough to flick the off switch on the robot. Of course, if i wrote the software britney would be on the list of targets. :P
0 Votes
+ -
You'd have thought
SiO2 23rd Feb 2009
that the makers of war would have realised by now that just like nuclear weapons made nuclear war obsolete, robots should make the battlefield obsolete.

I mean, sending wave after wave of metal 'over the top' to do battle with wave on wave of its own kind seems pointless, and a recipe for the skynet scenario in the first place.

Just as soon as theyre endowed with a sense of self-preservation, (or acquire one just by observation, which any intelligent being put in such a position would develop) theyre going to start asking some pointed questions that are going to be difficult to answer in terms of logic...

The whole point of battle is to kill the enemy, anything less is war, and wars are fought with many kinds of weapon besides guns. Why battle at all?
0 Votes
+ -
What about the Three Laws of Robotics?
fwjackson 23rd Feb 2009
I wrote a response directly to Mr. Dignan regarding
Isaac Asimov's Three Laws of Robotics.

Essentially, I posited that the Three Laws would
obviate the use of military robots. The Laws are:
1. A robot may not injure a human being or, through
inaction, allow a human being to come to harm.
2. A robot must obey orders given to it by human
beings, except where such orders would conflict with
the First Law.
3. A robot must protect its own existence as long as
such protection does not conflict with the First or
Second Law.

I also hypothesized that human (i.e. military) use
would, by its nature, lead to the destruction of
robots. Absent the Three Laws, more sophisticated,
evolved robots would develop artificial intelligence
and create their own laws regarding self-preservation.
As a result, the Terminator scenario is highly
probable.

Use of robots makes war far more palatable and
acceptable, since loss of friendly force life would be
substantially reduced. The far-reaching impact of
imposition of the Three Laws is that humans, too,
would evolve and cease making self-destructive war.
(That is the subject of a different line of
discussion.)

I am very interested in hearing other's responses to
Asimov's concepts.
0 Votes
+ -
What About the 7 Dwarfs Theory?
Timewellwasted 7th Mar 2009
Not intending to harsh your buzz there dude but.... These Three "Laws" would never be implemented! You are talking about a fictional piece of literature as if though it were remotely related to reality. Have you not noticed half of the world is trying to figure out how better to kill and destroy the other half of the world?
It is obvious we do not need to worry about an army of robots going postal on humanity anytime in the near future, and realistically we are probably 150 to 175 years (minimum) from having any form of AI "smart" enough to even comprehend the concept of the Three Laws of Robotics. The real problem with Isaac Asimov's Three Laws of Robotics is they are in direct contrast with the goals of the people funding the research! i.e. the military and government..
The problem with this Cal Poly report on Autonomous Robotics in a warfare scenario is we are nowhere near this type of technology. The drones and Predator type aircraft of todays military are barely smarter than the toy Remote Control Helicopter sitting in my garage. The real worry should be how many millions of dollars were spent by the Navy to fund the research into this far flung scenario. The Navy could have saved all of this money and simply visited the labs that are working on AI. The two "smartest" robots on earth are so fraught with problems it is almost laughable to think of them as a prototype for something that will be dangerous to anyone, that is unless the damn thing falls on top of someone.
Way too many people confuse remote control with something resembling artificial intelligence. No one on earth is near anything even resembling AI as it is portrayed in the Sci-Fi movies.
And to touch on your subject of a different line of discussion, it will be tens of thousands of years before humans will actually evolve even the slightest. As far as changing our thought process and cease making self-destructive war., until we (humans)come to the conclusion we all are truly equal and the concept of wealth is no longer relevant, we will always have war.
this is the future, and therefore the UK should dump ?billion in R&D on this..

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix