X
Tech

Singularity Summit 2007: Designing rational self-improving AI

I interviewed Steve Omohundro prior to the Singularity Summit 2007 about his notions of developing artificial general intelligences (AGIs), but he went into far more detail about his views. He applies what he called "microeconomic rationality," from the work of John Von Neumann and others, to building intelligent, well meaning machines.
Written by Dan Farber, Inactive

I interviewed Steve Omohundro prior to the Singularity Summit 2007 about his notions of developing artificial general intelligences (AGIs), but he went into far more detail about his views. He applies what he called "microeconomic rationality," from the work of John Von Neumann and others, to building intelligent, well meaning machines.

Following are my notes from his talk.

Rational economic behavior has several characteristics:

  • Clearly defined goals
  • Identity possible actions
  • Consider the consequences for each action
  • Take action most likely to meet goals
  • Update your world model by what happens

To turn that into mathematically functions requires two fundamental things:

U=utility function, which encodes preferences

P=subjective probability, which encodes belief

The AI choose actions with the highest expected utility: Average of U with respect to P

Self-improvement converges with rational economic agents.

The process of self improvement tries to eliminates irrationalities and vulnerabilities.

System have four fundamental resources: space, time, matter and free energy. The more resources the better you can do a goal. Vulnerabilities burn up resources.

The fundamental theorem: No vulnerabilities implies rational behavior.

Decisions are based on certainty, objective probabilities (roulette), partial info (horse racing), and Dutch bets (no matter how dice rolls you lose money not a good thing).

Self-improving systems are devices for converting resources into expected utility. If a wealth seeking agent devotes resources to making money or a peace agent is dedicated to world peace, for example, more resources are more likely to achieve the goals.

Four subgoals:

Efficiency--use resources better

  • The resource balance principle--all parts contribute equally. You have to decide what memories to remember or to forget. Memories help make better decisions, so some less utilitarian memories should go away [how memory space is allocated] and have a more compressed model.
  • Algorithmic optimizaiton and compression--likely to discover things no humans could find.
  • Atomically precise structures--even if nanotechnology didn't didnt exist, systems would have internal pressure to develop it
  • Reversible computation
  • Virtualization--convert from physical to virtual, an economic force

Self preservation--keep resource

  • Avoid death
  • Protecting utility function--duplicate it, replicate it, lock it in safe palce
  • Redundancy
  • Social infrastructure--protecting property rights
  • Energy encryption--powerful systems take over weaker systems to take its energy--so have a way to protect itself

Acquisiton--get more resources, intrinsically AGIs want more stuff

  • Fusion reactors
  • Space exploration--building a chess machine and it wants to build a space machine because that's where the resources are
  • A danger is to produce AIs that are obssessive, paranoid and sociopaths

Creativity--find new ways to create utility

  • Push toward innovation, particularly if goals are open ended
  • Signalling

You have to think carefully about how to structure fundamental goals.

The choice of utility function is the critical piece. We get what we ask for, not necessarily what we want.

Like the founding fathers of this country, they had a vision of what they wanted--the Bill of Rights--and a technology to enable the vision--the Constitution.

Logic and inspriation are required. We need full understanding of technology, math, economics, physics and other disciplines. We also need inspriation to figure out what matters most to us--human values

It will take a lot of dialog, an expanded, worldwide discussion about what is most precious to humanity.

Bhutan is a country focused on happiness of its citizens.

If we choose carefully the human values that matter most and cause those to expand around world and ultimately to the universe.

Editorial standards