Computer chatbot 'Eugene Goostman' passes the Turing test

Computer chatbot 'Eugene Goostman' passes the Turing test

Summary: A computer program that pretends to be a 13-year-old Ukrainian boy passed a Turing test at the Royal Society in London yesterday on the 60th anniversary of Turing's death.

SHARE:
Eugene Goostman

A computer program that pretends to be a 13-year-old Ukrainian boy called Eugene Goostman passed a Turing test at the Royal Society in London yesterday (Saturday 6 June) by convincing 33 percent of the judges that it was human during a five-minute typed conversation. The test was suggested by computer scientist Alan Turing in 1950, and the competition was held on the 60th anniversary of his death.

The judges included Robert Llewellyn, who played the android Kryten in Red Dwarf, and Lord Sharkey, who led the campaign for Turing's posthumous pardon last year. Llewellyn tweeted: "Turing test was amazing. Did 10 sessions of 5 minutes, 2 screens, 1 human 1 machine. I guessed correctly 4 times. Clever little robot fellow/" There were five AI programs, so presumably Goostman fooled him. [Update: Llewellyn has written about his experience in Turing test: this little Eugene Goostman was much brighter than I imagined at the Guardian.]

Eugene Goostman's success was not a surprise. In 2012, the same chatbot fooled 29 percent of the 25 human judges when winning a Turing test competition held at Bletchley Park, the World War II codebreaking centre, in Milton Keynes on June 23 — which would have been Turing's 100th birthday. Goostman was also a runner-up in the Loebner Prize tests in 2001, 2005, and 2008.

Kevin Warwick, a visiting professor at the University of Reading, which organised both tests, said it was the first time a chatbot had passed an open-ended test, rather than one where topics or questions were set in advance.

Vladimir Veselov
Vladimir Veselov, one of the program's developers. Photo: University of Reading

Some of Eugene Goostman's success is down to the choice of a 13-year-old boy from Odessa, who can't be expected to know everything and might be forgiven for minor grammatical or linguistic errors. Vladimir Veselov, one of the program's developers, said: "We spent a lot of time developing a character with a believable personality."

The fictional Eugene has a father who is a gynaecologist, and has a pet guinea pig. He says: "My mom's always shouting that 'this dirty pig is a PIG anyway, in spite it is 'guinea' — and wants me to give it as a gift to anyone of my friends for their birthday." You can ask him yourself at the Princeton AI (artificial intelligence) website, where Eugene chats online. ("No, I don't go to princeton, unfortunately! Maybe, let's talk about something else?")

The three Russian developers — Veselov, Eugene Demchenko and Sergey Ulasen — met up in Saint Petersburg, though Veselov now lives and works in the US. In a presentation at the Chatbots 3.0 conference in Philadelphia in 2010, Veselov said the chatbot was his hobby, and that one of the most important parts of the program was a "typo corrector". Like humans and Google Search, Goostman tries to make sense of spelling mistakes and mistypings that generally cause computers far more problems than humans.

Chatbots could be used to provide robot assistances to website visitors, or built into robots that provide human companionship, and so on. The underlying technologies could also be used to improve the automatic parsing of documents, and similar tasks.

Kevin Warwick said in a statement from the organisers: "Of course the Test has implications for society today. Having a computer that can trick a human into thinking that someone, or even something, is a person we trust is a wake-up call to cybercrime. The Turing Test is a vital tool for combatting that threat. It is important to understand more fully how online, real-time communication of this type can influence an individual human in such a way that they are fooled into believing something is true... when in fact it is not."

Of course, bots are really not hard to spot, if you're trying. But they don't have to pretend to be human in order to be useful.

Further reading

Topics: Emerging Tech, After Hours

Jack Schofield

About Jack Schofield

Jack Schofield spent the 1970s editing photography magazines before becoming editor of an early UK computer magazine, Practical Computing. In 1983, he started writing a weekly computer column for the Guardian, and joined the staff to launch the newspaper's weekly computer supplement in 1985. This section launched the Guardian’s first website and, in 2001, its first real blog. When the printed section was dropped after 25 years and a couple of reincarnations, he felt it was a time for a change....

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

27 comments
Log in or register to join the discussion
  • 33%?

    "passed a Turing test at the Royal Society in London yesterday (Saturday 6 June) by convincing 33 percent of the judges . . . "

    33%? Shouldn't it fool more than 50% to pass? Am I missing something?
    CobraA1
    • 30%

      30% is the threshold
      alexaabedi
      • Seems arbitrary to me.

        Seems arbitrary to me.
        CobraA1
        • Whether it is arbitrary or not

          it is the challenge that Turing put out there, and which AI types have been fighting to beat for close to 50 years. This is the first computer that has passed it. It is a major achievement.
          Mac_PC_FenceSitter
          • turing never set the threshold

            http://www.buzzfeed.com/kellyoakes/no-a-computer-did-not-just-pass-the-turing-test
            Cilvre
          • So basically . . .

            Interesting article. So basically it's a chatbot?

            . . . which I suppose if good enough for something like Siri, but would lack the ability to learn new things and adapt to new situations.
            CobraA1
        • that's because you are a bot

          any human can tell you that 30% is not arbitrary
          vpupkin
      • 30% is NOT the threshold

        There was never a threshold set. The figure is based on an off-hand remark by Turing that didn't even relate to the test, where he estimated that by the year 2000, machines with around 100 MB of storage would be able to fool 30% of human judges in a five-minute test.
        .DeusExMachina.
    • I wonder what this really says...

      Maybe we should really be worried about why 33% of judges expect 13 year old boys to be so stupid.
      BillDem
  • when Alan Turing

    first proposed the Turing Test it was seen as a valid way to test the idea of artificial intelligence. A lot has changed. Computers have capabilities he could not have imagined yet fail at some very basic tasks. Beating a chess grandmaster.not a problem. Tying a shoe. Problem. We have learned a lot about both human and machine intelligence and they are very different things.
    krossbow
    • This is simply not so.

      First, robots have no problem tying shoes. But more importantly, the changing capabilities of computers has nothing to do with the applicability of the Turing test.
      .DeusExMachina.
  • Don't see how it passed anything

    I typed "Hello"
    It responded saying that it wished me hello and also sent greetings from his guinea pig.
    I typed "How old is he?"
    It responded with "Somehow :-) Wonna ask me something more?"

    I don't see how this fools anyone over the age of 2.
    israel Silverman
    • that's the 2001 version

      A version of the computer programme, which was created in 2001, is hosted online for anyone talk to (http://www.independent.co.uk/life-style/gadgets-and-tech/computer-becomes-first-to-pass-turing-test-in-artificial-intelligence-milestone-but-academics-warn-of-dangerous-future-9508370.html)
      whoareme
      • Yes, but

        But even 10 years before 2001, various programs were able to respond much better. The reference to "him" was pretty clear.
        israel Silverman
        • Not really relevant

          as it is the 2014 version that passed the test, not the 2001 version or any of its contemporaries.
          Mac_PC_FenceSitter
  • When I was five years old

    my Magic 8 Ball passed the Turing Test, I'd suspect. Much depends on the humans involved and the social context (very restrictive for the distinguished gentlefolk at the Royal Society, of all places.) Lets have a cute little twelve year old send Goostman a sext message and see if she's fooled.
    I2k4
    • Update - I asked the online Magic 8 Ball

      "Is Eugene Goostman a real boy?" and its answer is

      "Most likely"

      I'll be drammed.
      I2k4
  • this is silly

    a Ukrainian boy converses in fluent English - and 33% of the judges believe it? Now the real question is - are the judges human?
    ForeverSPb
    • Um, read the article?

      The AI was NOT fluent to the degree of being error free. Nor is it at all surprising that a real 13 y.o. Ukrainian boy would be fluent in English.
      .DeusExMachina.
  • Soon there will be posters on this blog

    who will question whether their UN-favorite columnist (whoever that is) is human or robotic.
    jallan32