Semantic Search Round Table at the Semantic Technology Conference

Wednesday's opening Keynote here in San Jose sees Guidewire's Carla Thompson joined on stage by senior representatives from many of the more interesting players in the Semantic Search space; Tomasz Imielinski from Ask, Peter Norvig from Google, Riza Berkan of Hakia, Scott Provost from Microsoft, William Tunstall-Pedoe of the UK's True Knowledge, and Andrew Tomkins of Yahoo.

Carla asks each panellist to describe the differentiating aspects of their product in 'one or two sentences;'

Tomasz; "we receive about three times as many questions as other search companies. We want to answer questions the best we can from multiple sources... using structured and unstructured data."

Scott; "Bing really focusses on understanding the intent behind queries, and organising the page to help people get to their answer much faster."

Peter; "We focus on being comprehensive, accurate and fast... so we have to keep on innovating in crawling, ranking, systems engineering. One thing that differentiates us... most companies decide whether to focus on marketing or sales. We focus on engineering."

Riza; "We are a complete semantic search engine, from the bottom up. We don't even have an index. We've optimised the entire process for semantic operations. We focus on credible and dynamic content, and offer users a new perspective." Instead of popularity, they focus on credibility.

William; "True Knowledge is a platform that does direct question answering. There's a knowledge base and an inference engine to answer questions we haven't seen before." True Knowledge tries to 'help when it can, and stay quiet when it can't,' as can be seen demonstrated in their recently released Firefox plugin.

Andrew: "Yahoo! is very aggressive about semantic annotation... SearchMonkey is about acquiring semantic information and surfacing it in search results on the page."

Carla mentions Tom Tague's keynote from yesterday, where he suggested that 'semantic search is an answer to a question no one is asking'... so "why do we need to change search?"

Tomasz responds, suggesting that users don't necessarily demand new products that subsequently become successful. eg; no one was asking for the iPod before it launched. "When they see it, they will want it."

Turning to Google and Yahoo!, Carla asks them "why do we need to change search?"

Peter... "as an industry, satisfaction is very high... but that is just because that's what people know [now]... People don't like technology... people like solutions. When we deliver it, people will want it."

Andrew; "Does search need to change? It already is... Today, on any major search engine, if you search for a restaurant, you'll see structured information about that restaurant; reviews, phone number, etc... This has been accelerating over the last 3-4 years... When we put this information up, and trigger it correctly, we see far higher levels of engagement from our users than anything else."

Carla; "it may be a stupid question, but it has to be asked; what is semantic search?"

Scott; "it means a lot of different things. At Powerset we focussed on understanding the meaning in web pages, so we could present them, rank them..."

Carla; "Has Powerset's focus been diluted by the [Microsoft] acquisition?"

Scott; "No."

Carla asks Riza; "Someone from Hakia that I spoke to last year said you were the only one doing 'true semantic search.' Is that true?"

Riza; "No... Semantic Search can enrich search results... Semantic Search can improve precision/disambiguation... Semantic Search can organise results better. In the future, search will move to more conversational systems, and for that you really need semantic technology."

Carla; "How do you measure the 'semanticity' of a search engine?"

Tomasz; "That's my favourite question... We took a sample of 'equivalent' queries from the logs, and ran it to evaluate ranking etc; does the search engine give similar answers to questions like 'Top 10 songs' and 'Top Ten songs,' etc. Should they?"

Andrew; "It's incredibly hard to understand what a user will like... if you mess with the logo, it changes the perception of the results... if you make tiny changes, it can have a big impact on perception... When it comes to understanding semantic contact in search, we should identify the task the user is trying to solve... and have a metric that's aligned to that use case... We can break search queries today into different classes; how do we do when a user is trying to book dinner, or a vacation? Semantic Technology should be judged on its impact based on these task metrics rather than any underlying notions of entity resolution, etc... SearchMonkey, for instance, lets users inject structured data into the process... The information can be incorporated in any way... and change how the results are presented. We have about 15,000 people in our development community, changing the way those results are presented every day."

Tomasz; "I would expect a semantic search engine to deliver equivalent results to queries that would appear similar to a human being; 'Top 10 songs' and 'Top Ten Songs' should deliver the same answer. Today in most mainstream search engines they don't."

Carla; "Search v. Answers. True Knowledge is billed as 'the Internet Answer Engine;' is it necessary to move search to an answer-based format, or has Google trained users to think in keywords?"

William; "We support both keyword search and full-text questions. It's important to answer users' questions."

Peter; "Different types of answers are appropriate for different types of questions; sometimes the answer is a fact, or a page, or a series of results to support a process of study. To say there's going to be one technology or one type of answer doesn't make sense."

Riza; "You could be asking a 'where,' 'why,' 'how' type of question. Questions are important, and the search engine needs to be able to interpret the mode of the question and return results appropriately."

Carla; "You mentioned talking about the credibility of search results. How do you define a 'credible' search result, and how much of a need is there really? I'm not hearing users question the credibility of search results they see today."

Riza; "Practically, credibility is important in 'serious' subjects; medical information, etc. You want to know where the results come from and how credible they are. When it comes to credible content, you can't really do a statistical search or have a 'popularity vote,' because much credible content isn't 'popular.'

Scott; "People's expectations for credibility are different depending upon the query. If you ask an 'instant answer' type query you expect the answer to be credible. If you do a broader search, you expect a mix of results to be returned"

William; "If a system understands structured knowledge, it can understand when different sources contradict one another"

Riza; "A system doesn't need to know what's credible; we can go to a librarian for that. Hakia doesn't decide whether a resource is credible or not; we use librarians for that"

Tomasz; "If you ask for the capital of Japan we expect a single answer. If you ask about taxes, maybe the IRS is the best source but there are others. If you ask 'how to get rid of acne' you expect a lot of results."

Carla; "We've seen three news-making launches in the past month; Wolfram Alpha, Bing, Siri. Is Wolfram the first step towards 2001? How is this engine valuable to those of us who don't need to solve complex maths?"

Scott; "it's not the first step... we've been working on these problems for a long time. There are a lot of questions people want to ask about the types of data that Wolfram aggregates... We see these things as part of full-search services. Powerset has moved along this path as well, pulling structured data in response to full-text queries."

William; "Wolfram is a tremendous effort. An interesting example of question answering with structured data. I think people will find uses for it in particular use cases; I spoke to someone who'd used it to calculate when his visa expired, because it could do date calculation. I think there will be use cases in various scenarios; maths, nutrition information, etc... if you remember that it has that sort of information and remember to go to it... However one thing it doesn't have is a decent back-fill. If it doesn't have the data, or doesn't understand the way you asked the query, it gives you nothing. We try to keep quiet and fail over to standard internet search in that sort of circumstance."

Carla; "Does a semantic search engine know how not to answer a question?"

William; "that's absolutely fundamental. You need the ability to reliably keep quiet when you don't have the answer... and fail over reliably to other search services. [True Knowledge does try to do this...] "That requires very high quality semantics."

Andrew; "One way to characterise the approach of Wolfram Alpha is that it's a centralised approach. The Wolfram Alpha team goes out to find data and bring it in-house to convert to a standard form. A different approach is to have an ecosystem contributing data in the public eye... It's not clear yet how much of a value-add is going to come from this centralised knowledge mapping approach. Yahoo! is focussed on the ecosystem approach, and helping people with knowledge to make it available."

Peter; "Our inclination would be that we don't want a closed walled garden. We want all the information available to combine in different ways. We want the information to be open, and the tool set to be open for mashing up in different ways."

Scott; "If Wolfram Alpha hadn't taken a walled garden approach they might never have launched a product."

Tomasz; "Wolfram Alpha is great, but it's not a search engine"

Carla; "Siri... caused a lot of buzz, uses True Knowledge... what are your thoughts?"

Andrew; "To be counter-cultural... the notion of getting much deeper and assisting a user with a task is spot on. We're going to see much more of that. Search has tended to be stateless. Each query you enter is more or less processed without context. Yahoo! is rolling out more stateful search tools, and other companies will do the same. We expect people to use these tools on lots of devices. Would be expect people to come to the same place for purchase, navigation, etc? Do we expect one interface? There are going to be virtual assistants... I just don't know if they're going to be embedded into a search box."

Scott; "Conversation is the ultimate user interface... but it's not clear that I want to have a conversation with my laptop during the working day. How do I display the results? But there's a huge role for conversation and dialogue in refining search and getting a user to their results faster."

Tomasz; "What is the goal of Siri? If you try to go to broad you become a search engine."

Scott; "When people have a conversational interface, they won't speak in keywords."

Carla; "What are the larger goals for Bing?"

Scott; "Bing is trying to simplify key tasks that people do when they come to a search engine. In travel, health, shopping, we can understand what people are trying to do, and get them to better results faster. The thinking has evolved from ten blue links to the whole page, and organising things to help the user by understanding their tasks."

Carla; "Peter; what did you think of Bing?"

Peter; "I like the idea of innovation in the user interface. There's a lot of room for that. There's been a lot of emphasis on getting the ranking right. You still need to do that, but other things are important too. I'm usually happy with results on my big screen. On a mobile device, I'm usually not happy with the results I get."