So Cortana IS smarter than Siri -- but Google Assistant is smartest (for now)
Disclaimer: This study was carried out before Apple's Worldwide Developers Conference 2017, which talked about a significant upgrade to Siri in iOS 11.
Earlier this year, I looked at a study from Stone Temple comparing which was the smartest assistant: Google Home or Amazon Alexa.
Stone Temple has again been comparing the five top personal assistants on the market to see which is smartest. In its latest study, it asked Google Assistant, Siri, Cortana, Alexa, and Google Home 5,000 questions. It kept detailed logs on the type of responses it received.
In its March study, it published the largest test to date of the question-answering abilities of Google Home and Amazon Alexa. Google Home won that contest, so the new study also includes Google Assistant, Siri, and Cortana for completeness.
The 5,000 different questions were compiled about everyday factual knowledge that people would naturally ask a personal assistant or search for on Google. It used the same set of queries it used in the Google Home versus Amazon Echo test.
Each assistant was asked the same 5,000 questions, and it noted many different possible categories of answers, including whether the assistant answered verbally or retrieved the answer from a database such as the Knowledge Graph. It also noted if an answer was sourced from a third-party source ("According to Wikipedia ...").
It noted errors such as how often the assistant did not understand the query, or if it tried to respond to the query but simply got it wrong, as well.
The test showed that Google Home seems to be the smartest virtual assistant. It answered most of the questions at 68.1 percent. And 90.6 percent of questions answered were absolutely complete and correct.
Microsoft's Cortana was not far behind -- with 56.5 percent of questions answered and 81.9 percent of questions answered completely and correctly.
Amazon Alexa did get 87 percent of its answers complete and correct. However, it only answered 20.7 percent of questions put to it. Apple's Siri performed poorly, with 62.2 percent of answers complete and correct, but it only answered 21.7 percent of answers.
Google search was also used as a comparison. It answered 74.3 percent of questions, getting 97.4 percent completely correct.
The test results show Google Assistant on Google Home as the winner, with Microsoft's Cortana not far behind. Cortana has dramatically expanded its capability in providing rich answers over the past year.
It will be interesting to watch how the race for the personal assistant develops as all vendors up their game.