Working with Anja Jentzsch of Freie Universitt Berlin, Richard Cyganiak of Galway's Digital Enterprise Research Institute (DERI) has returned to the diagram he first drew in 2007, and brought it right up to date. 203 data sets are represented, and together they comprise around 395 million links between over 25 billion RDF statements. Not bad for something that began life as a small academic exercise, but there's a very long way still to go.

Starting as one of Sir Tim Berners-Lee's Design Issues notes, the idea of Linked Data was given credence by the informal Linking Open Data project. As the project web site notes,

The goal of the W3C SWEO Linking Open Data community project is to extend the Web with a data commons by publishing various open data sets as RDF on the Web and by setting RDF links between data items from different data sources.

From those early days, when researchers 'found' data sitting on the web and set about transforming it into RDF by themselves, the initiative has grown significantly. Governments, media properties from the BBC to the New York Times, and retailers such as Best Buy have been amongst those to set about plugging their own data into the increasingly rich network.

Links between these data sets are established when one draws upon concepts defined in another. GeoNames, for example, is seen by many as a useful resource to draw upon in describing places. Rather than fuzzily talking about 'Paris,' I might make the statement unambiguous by referring to this Paris rather than that one. For a dataset (describing works of art, for example) in which location is not key, it's far easier to let someone else worry about recording where Paris is, what country it's in, how big it is etc rather than do it all myself.

This concept of linking from one resource to another is clearly key, but the relative dearth of connections on the diagram shows that there is still some way to go. DBpedia remains disproportionately important, and whilst it's unlikely that every resource will ever meaningfully connect to every other resource, some more connections would be valuable.

The latest iteration of the cloud diagram is based upon information that the community was invited to record in the CKAN data repository, which should make data collection and update easier and more accurate. In principle, future versions of the diagram could be generated programmatically as the data changes.

This month's revision of the Linked Data cloud is a useful illustration of progress within the community. Much remains to be done, both in terms of clarifying licenses and by collecting even more data. There is also work to do in adequately explaining what people might find. The diagram does link through to descriptions of each resource, but the 'explanations' are all-too-often only meaningful to people knowledgeable enough not to need an explanation in the first place!

This repository contains data from JISC, who fund research and infrastructure in the UK.

What data? About what sort of thing?

Airport data from Our Airports published as RDF

What sort of data? Where in the world?

Siri reappeared on the scene back in February, launching their innovative (US-only) iPhone app after a protracted silence. In March, the company snatched Gummi Hafsteinsson from Google, strengthening their mobile credentials still further and appearing to maintain momentum. Semantic technology enthusiasts watched for Siri to get even smarter, mobile officianados waited with bated breath for the promised Android app, and those of us outside the United States waited for a version of the app that understood our accents, our geographies, and the APIs of the companies offering travel, events, entertainment, restaurants and the rest in our territories. Although Winarsky, VP for Ventures, Licensing and Strategic Programs at the company from which Siri was originally spun out, felt unable to comment on the specifics of the product road map moving forward, I can't help speculating that any Android (or Blackberry) app has probably moved a lot further down the priority list.

Even with billions burning a hole in his pocket, it's unlikely (although not impossible) that Steve Jobs spent more than $200 million just to interfere with the release of an Android app. So what did Apple want?

Firstly, they probably wanted to make sure that someone else didn't pounce and grab the opportunity. Secondly, Siri is smart, innovative and impressive. The application does a great job of appearing simple and intuitive, whilst fulfilling a useful - and potentially complex - role. Given the surprisingly limited voice capabilities of Apple's current mobile devices, they may well be interested in harnessing some of that language interpretation capability for future mobile devices and even their desktop line. I doubt that required Siri, though, so much as the far cheaper poaching of some key staff from Siri or their partner (and fellow SRI spin-out), Nuance.

In its current form, Siri remains relatively limited in scope (finding and booking things on the move, essentially) and geography (the United States), and if Apple is serious about making something of the company they've acquired (rather than just securing its staff and IP) it will need to move forward aggressively on both fronts. Do they want a travel assistant with global reach, or do they want to give every iPhone/iPad/Mac owner their very own virtual secretary?

Winarsky is quick to suggest that the notion of the 'virtual personal assistant' applies in contexts far beyond the mobile niche carved out by Siri. Citing a 'perfect storm' created by the conjunction of bandwidth, compelling user interfaces and on-demand network compute power, he argues that Siri's current incarnation merely represents a taste of what's still to come. SRI continues to build upon the CALO project from which Siri was born, and opportunities are becoming increasingly clear in verticals as diverse as healthcare, retail, and call centre operations. SRI conducts 'more than 2,000 projects per year,' and the next CALO spin-off could emerge from their labs 'in less than six months.' Surely Apple didn't buy the wrong CALO baby?

Far from embarking upon a mission to solve the Turing Test, which Winarsky argues cannot be done, SRI researchers (and the Siri team) are amongst those transforming the tarnished image of Artificial Intelligence. By working within a defined vertical (healthcare, etc), it is entirely feasible to train an agent in such a way that it can reliably infer and act intelligently with respect to context. When you're only dealing with medical complaints (say), it's not impossible to train a machine to excel with idiom, jargon, and even ambiguity. It's also feasible to expect that the machine could learn - and adapt - 'in the wild,' without recourse to the lab every time external factors shift substantially.

As well as pointing to SRI's long heritage, successful track record, and future plans, Winarsky was keen to stress the important role played by Siri's venture capital backers in reaching the current settlement with Apple. Both Menlo Ventures' Shawn Carolan and Morgenthaler's Gary Morgenthaler apparently 'did the heavy lifting' on the deal. Winarsky points out that the hardest part of getting a company like Siri to this stage isn't actually developing the technology, but recognising the real market opportunity and 'navigating amongst the giants' in the space. In all of this, he reiterated, good VCs proved invaluable. And they weren't even listening in on the call!

I never thought I'd see 'metadata' discussed in the UK Parliament, then I did when we adopted Government Metadata Standards a few years back. I never thought I'd see 'RDF' pass the lips of a head of state, then Prime Minister Gordon Brown said it this year. Will Steve Jobs stand up at WWDC'10 and talk about Apple's embracing of/invention of 'the Semantic Web?' We shall see!

Good luck to all at Siri, and I look forward to seeing what Apple does next... and what SRI spawns next.

Siegel's definition of the 'Semantic Web' is broader than that preferred by many, extending far beyond technical standards such as RDF and OWL to embrace anything that is both unambiguously structured and on the Web. He offers a 'Semantic Web acid test,' and relies on this throughout the book in an effort to clarify his understanding of the space.

I spoke with David this week, and recorded our conversation as a podcast that is now available online. We discuss the premise behind the book, and then explore some of the presumptions upon which his argument is built.

Others, such as Greg Boutin, have written their own reviews of the book, and Siegel also recorded a different Technometria podcast with Phil Windley earlier this month.

So what do you think? Have a listen, and see how you see Siegel's concept of 'Pull' relating to the Semantic Web. Broader? Narrower? The same? Different? And will the emotive language and examples used throughout the book succeed in driving break-out growth where so many earlier efforts have had only modest success?

Last week, I spoke with Martin Hepp of Germany's Universitt der Bundeswehr Mnchen and Jamie Taylor of San Francisco startup Metaweb (best known to readers of this blog as the home of Freebase) to learn more. The result has just been released as a podcast.

Within specific organisations and supply chains, of course, Master Data Management is already well understood. Processes and procedures are in place to ensure that products are accurately and unambiguously described, and I touched on some of the semantic technology applications in this 2009 piece for Semantic Universe.

The picture becomes somewhat less clear as data moves out of the enterprise and onto the web. Even on the product website itself, much of that internal structure and richness is inadequately conveyed. For the consumer (or aggregator) wishing to compare and contrast MP3 players from a number of competing providers, it can frequently be difficult to accurately ensure that they really are comparing apples with apples. It is here that GoodRelations comes into its own, offering data providers a consistent way in which to describe key attributes of their business and its products.

Yahoo! already works to intelligently represent GoodRelations data in search results, and Martin Hepp asserts that Google "is doing something" with the data too. GoodRelations-encoded product descriptions, it seems, perform better in the mainstream search engines than their less structured competitors. The rich structure is also available for use in specific applications far beyond the generic search engine, and we touch on some of these possibilities during the conversation.

US-based consumer electronics retailer Best Buy already embeds GoodRelations RDFa in product pages, and reports improvements in findability and use.

Closer to home (for me, at least), UK supermarket giant Tesco has begun to experiment with embedding RDFa in pages. GoodRelations terms aren't used - yet - but it will be interesting to see how quickly that changes, and the applications that third parties might begin to build that leverage all this rich structure.

Have a listen to the podcast, and see what you think. Does GoodRelations have a place on your website?

Today's iPhone app is the first consumer offering from a company that has spent a long time thinking about this space. Much of the core research resulted from the $150million CALO project at SRI, funded by DARPA. Siri itself emerged from SRI to close an $8.5million Series A round with Menlo Ventures and Morgenthaler back in October of 2008.

Explicitly described as complementary to web search, rather than a replacement for it, Siri seeks to move beyond a paradigm based upon keywords and links to embrace one that is personal, task oriented, and conversational in nature. Siri guides the user along a path, making query formulation iterative and relatively painless, and ensuring that the application gets the information it needs. Early use cases are optimised to 'help you get things done.' probably whilst mobile. You might, for example, ask (by speaking to Siri's Nuance-powered speech processing engine) for 'Sushi near work at 7pm.' That simple request is relatively straightforward for a human being to understand, process, and act upon, but requires a significant degree of intelligence on the part of a software agent. Where is 'work'? What is 'sushi,' and what do you actually want to do with it (find a restaurant where you can eat it, presumably)? When is '7pm'? Alternatively you could let Siri conversationally lead you through the 'right' questions to reach the same outcome, as the sequence of screenshots at the end of this post demonstrates.

At this point, it's worth mentioning that Siri (like so many location-powered applications on the Web, the iPhone, Android, or wherever) is currently only really effective in the United States. This is due to any number of factors, including consumer readiness and the easy availability of cheap yet comprehensive data, but the situation will doubtless hopefully improve over time. Siri currently makes the point explicit, refusing to allow registration of a home or work address (or timezone) outside the United States. For the purposes of experimentation, my office has temporarily relocated to 1600 Pennsylvania Avenue, Washington DC, where there seems to be plenty of sushi available for tonight's dinner.

The speech processing works well on the whole, although I'm bemused that Siri interpreted my daughter's 'Where is the nearest Greek island?' as '18 me.' Whilst it's possible that this is an AI's attempt to avoid causing offence by responding 'Greece, you silly girl' it seems more likely that the engine got very confused by her non-American enunciation. Trying to sound like Hannah Montana just made it worse.

Even in the States, data is key to ensuring that apps such as this one deliver a rich and useful experience, as all the AI smarts and user interface polish in the world can't help an app that ignores the Starbucks across the street when you ask it to find you a coffee. Siri has lined up an impressive group of data providers including OpenTable, MovieTickets, TaxiMagic, Citysearch, Yelp, Yahoo Local, Gayot, Rotten Tomatoes, NYTimes.com, WeatherBug, AllMenus, StubHub, LiveKick, Maponics, Nuance and TrueKnowledge. Kittlaus celebrates the recent explosion of accessible APIs from sites such as these, claiming that Siri has acquired 'far more data than we've had time to integrate yet.' In a number of cases, revenue sharing arrangements mean that Siri gets a cut when money changes hands. A selection of test searches focussed on areas of the US to which I travel regularly delivered the sorts of results that I'd expect, and there's clear value in the integration of data from a number of different providers.

The Siri team looks forward to analyzing the logs once users start putting this app to work. Amazon's Elastic Compute Cloud (EC2) will handle the heavy lifting in the early days, allowing computing resources to scale with demand. Once the team has an understanding of real-world loading, Cheyer suggests that they'll pull much of the computing resource back in-house to lower costs. There is a clear expectation that Siri's responses will iterate rapidly as data become available to show how users use the app.

Further out, there's the ever-present need for more data. Kittlaus is also interested in increasing the opportunities for facilitating revenue-generating commercial transactions, and in allowing Siri to 'know you better.' Work, home, and current location is one thing. Why not favourite food, names and contact details for family (so I can have Siri 'tell my wife I'll be late home'), preferred airline, and more? It makes sense not to introduce these features from the outset, as consumers will need to both value and trust Siri before willingly giving up such detail. But you can be sure they'll be included soon.

Voice already plays a role on mobile devices such as the iPhone, perhaps most usefully in Google's search app. It remains to be seen whether consumers will really use two, or look for many of Siri's features to move across and enrich the voice-powered search experience they're already getting from Google, which presumably has many of the same data deals already in place.

Kittlaus stressed several times that Siri will deliver value on other platforms, suggesting a Siri email address (similar to plans@tripit.com, presumably), a destination web site with which users might converse, or a Siri IM buddy that could be drawn into conversations.

By delivering value to users, and by building an ongoing relationship (backed by data) that's difficult to replicate, Siri seeks to offer a compelling and defensible business. Playing with the application from the other side of the Atlantic it shows clear promise, and I look forward to putting it through its paces on my next trip to the States.

And, of course, you can be pretty sure it'll run on the iPad.

This sequence of screenshots illustrates Siri's conversational approach to getting from my vague opening query about 'restaurants' to a reservation for specific people in a specific place at a specific time on a specific day. It would have been quicker to simply say what I wanted up front... but sometimes you just don't know until prompted.

[gallery]

More importantly, perhaps, direct integration with an Enterprise product such as Oracle's database says much about how far the semantic technology community has come in being able to offer solutions capable of scaling - robustly - to meet Enterprise-scale demands.

Xavier Lopez, Oracle's Director for Spatial and Semantic Technologies, is quoted in Thomson Reuters' press release;

"This interoperability lets users quickly process documents in different formats (such as Microsoft Word and Adobe PDF), to extract semantic metadata that can be used for more semantically complete searches in Oracle11g."

At the heart of many of today's initiatives lie programmes to surface Government data for use and re-use by third parties. The 'open' in 'Open Data' is, of course, a very loaded term, and I've looked before at some of the ways in which data might become 'open' whilst remaining effectively useless. Nevertheless, Governments' current enthusiasm for being seen to embrace transparency should certainly be both welcomed and encouraged, and there are real opportunities to work with Government in ensuring that today's transparency fervour continues undiminished, whether by omission or commission.

Given the complex and varied nature of the data involved, and the obvious linkages between the entities (you and I, our communities, our schools, our hospitals) described in numerous different databases, there's a clear opportunity for technologies and approaches from the Semantic Web community to play a significant role in simplifying the whole process of moving these legacy databases online.

Already interested in Open Government from previous roles, and (obviously!) committed to encouraging real-world adoption of semantic technologies, I've spent some time recently talking to a number of those involved. A number of those conversations are now available as podcasts, and I'll continue to seek out fresh examples and perspectives to share.

My most recent podcast conversation, released today, is with Professor Jim Hendler and Dr Li Ding of the Tetherless World Constellation at Rensselaer Polytechnic Institute in Troy, NY. The team at Rensselaer have been working with some of the US Federal Government's data sets on Data.gov, and so far they've converted sixteen data sets from their original form, resulting in 2,927,398,352 freely available RDF triples and a number of demonstration applications.

Other conversations already released in the series include;

• David Eaves, talking about Vancouver's commitment to Open Data
• John Sheridan, Head of e-Services at the UK Government's Office of Public Sector Information, talking about his Department's efforts to get Government data online
• Mark Birbeck, talking about work with the UK Government's Central Office of Information to embed lightweight RDFa into workflows and web pages

Each offers an example of ways in which 'open data' contributes to Government transparency, or to increasing the value of the massive sunk investment in collecting, managing and curating the data upon which Governments depend. The Semantic Web's notion of Linked Data (whether actually in RDF or not! :-) ) offers a means to increase the utility of the data we have, without a massive programme of reengineering the systems used to manage it. The examples we see today, and the work of the individuals and teams with whom I have been speaking, will teach us a lot about how to make this work at Government scale.

There are, of course, different ways of managing data, and it's not always necessary to store everything in one massive store... but for those concerned about scale today's news from UK-based Garlik may well put their minds at rest.

The company has taken their internally developed (and massively scalable) RDF triple store and released it to the world under an Open Source license as 4store.

I spoke with the company's CEO and Head of Architecture just ahead of the launch, to learn more about the system and their motivation behind sharing it.

The result has just been released as a podcast.

Audio and video of the session is now available, with Gang members and conference organiser Tony Shaw engaging in a discussion of the event's highlights and the underlying trends at work.

"The first semantic search system for The New Times was released in 1913 and was available bound in either paper ($6) or cloth ($8). In the 96 years since the advent of The Historical Index to The New York Times, semantic technology has become central to The New York Times' daily operations and the focus of much internal research and development. In our keynote, Rob Larson, VP of Digital Production, and Evan Sandhaus, Semantic Technologist, will review the long history of semantic technology at The New York Times; discuss the application of this technology in our operations; and review an innovative initiative to enlist the global community in solving some of our toughest challenges."

Sandhaus and Larson begin by referring back to the Times' long history, and the early importance of the paper's emphasis on building - and selling - a comprehensive abstracting and indexing service to stories in the paper. This, they suggest, was important in leading to the paper being considered as the paper of record, ahead of its numerous competitors.

Building upon the paper's nine-month old 'Annotated Corpus' and its associated APIs, Larson closed the session by announcing that the Times' thesaurus is to be made available using a license and APIs that will see it available to play a part in the wider Linked Data cloud.

Wolfram Research, of course, is the company behind the recently launched Wolfram Alpha; a 'computational knowledge engine' that attracted a wave of attention that reached into the mainstream media.

"Putting all of the world's computable knowledge; it sounds impossible... or over-confident, maybe. What is computable knowledge?"

"It's 'systematic knowledge.' It can be compared, contrasted, correlated, computed on. It's not a movie review. Examples are classical physics, financial data and models, weather data and models... It's not the latest opinion on who Britney Spears is dating. We don't have a model to do anything with that in our system."

Nova asks if it's the difference between objective and subjective... Alpha deals with objective information. 'Facts,' almost?

Nova asks about sources, pointing to the example of Tibet; is it a 'fact' that Tibet that is part of China, or not... ?

"In the case of geo-political things, and religious things, we have to make choices... and allow the community to let us know whether they agree or not..." Couldn't the system represent multiple views, tied to the diverse sources? Could we not show the different opinions, and allow the user to make informed decisions themselves?

Nova; "is the world's computable knowledge infinite?"

Russell; "the foundation of computable knowledge is likely to be finite... The amount of knowledge that can be computed and generated from that is infinite..."

Nova; "I can see that maths could be finite. But geopolitics, health, etc... that's much, much larger..."

Russell; "The instances seem very complex... Huge, but finite... I don't want anyone to think we'll have this done in ten years... It's a long term thing."

Nova; "Stephen [Wolfram] reckons it could be done in three years...?"

Nova; "Looking at the back end, the ontology seems to be implicit. I didn't see any classes, just a lot of instances... a set of facts. As the team grows, how do you prevent people adding facts in different ways?"

Russell; "There are a set of stored facts; things you know about a city. But then there are computed facts that you couldn't store in a traditional ontology." Huh?

Nova; "Can you make a statement about what percent of the world's computable knowledge is there today?"

Russell; "I can't make a statement..."

Nova; "The syntax is quite interesting... but enigmatic. It wasn't necessarily that the knowledge wasn't there, but that I'd asked for it in the wrong way. Can't you make a manual? ... Stephen [Wolfram] said it would be an impossible task to write the manual... or to make a generic natural language on top."

Nova; "In some cases a naive query will get you the answer, but maybe there's a need for a layer that helps you when you don't get what you want..."

Russell; "I think we're getting close... we're going to put an API out in the next few weeks, and hopefully someone will build the application using that to parse natural language and translate it for Alpha... Do we spend our time doing that, or putting more data and more models into the system... I reckon our time is best spent adding more data..."

Nova; "Is there a set of schemas or ontologies to link all of this stuff together?"

Russell; "There isn't an ontology over the whole system... but within a domain there is structure... Is there some grand scheme that we have internally? Not really. The company has been doing this stuff for 23 years, so there's a bit of a shared understanding internally."

Examples keep coming back to mathematics... To succeed, Alpha has to offer compelling examples that are far broader...

Nova; "What about reasoning. You've said that you can derive additional knowledge. What kinds of reasoning is the system capable of?"

Russell; "I'd call it very simple reasoning. For example changing the currency based upon your geo-location... Is there any weak or strong AI in here? Not really. Could you build something like that? Probably. Will we? I don't know..."

Nova; "Alpha seems to be a subset of Mathematica capabilities... Would you expand that, and bring a full Mathematica to the Web"

Russell; "It is, and there are plans to extend the capabilities. I don't know if we'd go to a full-blown Mathematica on the web."

Russell's mentioning a subscription service for people working with more data, or needing more compute time. The public web site tends to time out a query in 4-8 (or 48?) seconds... The professional subscription version will have a monthly subscription version that will allow you to compute bigger questions. There will also be a pay-per-use API... and 'primitive' advertising. More advanced advertising, based on transactions, to be launched soon.

Nova; "Alpha's really cool, but I want to do this on my own knowledge... inside an enterprise, inside a government agency..."

Russell; "We can roll out a custom Wolfram Alpha for those who want it behind the firewall. We will also let people upload their own data sets. We need to find a sensible way to let people do this..."

Nova; "There was a lot of hype - possibly my fault - around Alpha being a Google Killer. Obviously it's not that. It's something quite different. Who is the user, and what are they using it for?"

Russell; "Use will evolve, and it already has. There's an obvious use by students, but the school year has just ended.

Nova; "Wolfram Alpha; now even Ph.D's can cheat on their homework." :-)

Nova; "Are consumers using it? Obviously they're having a play, but are they coming back and using it?"

Russell moves off to talk about academic use... Dodging the question?

Nova; "Are the financial capabilities in Alpha differentiated from the capabilities banks and investors already have in their vertical?"

Russell; "more sophisticated than a general finance web site, but probably less sophisticated than you'd find on a terminal in a bank."

Nova; "Do I really need to know how long it would take an ant to get from San Francisco to Cairo?"

Russell; "Because of the way the system is engineered, it just keeps computing until it runs out of time. With simple queries you'll get a lot of data. It just keeps computing."

Nova; "What's the big challenge, moving forward?"

Russell; "Setting priorities."

Nova; "So let's talk about Google. They made some aggressive marketing moves during the Alpha roll-out, and they're continuing to roll products out to chip away... Do you think that what you've built is defensible, just because it's hard... or can you defend it in other ways?"

Russell; "There are significant barriers to what we're doing. Someone else could build this... but would they want to? That's an open question."

Nova; "Do you hope to work with other companies? Perhaps revenue share with them?"

Russell; "Obviously."

Nova; "There's been a lot of interest in how Alpha might connect with open standards and the Semantic Web..."

Russell; "If you want the platform to be used, we'll have to do some of this stuff... RDF, OWL, etc could play a huge role."

Nova; "Timeframe?"

Russell; "It'll depend on pick-up of the API... which is due out in a few weeks."

Nova; "So what's the implication for education? It makes it possible to do some things without even thinking..."

Russell; "It'll be a heated debate for a while... Some things are positive, some negative. There's going to be a reorientation... It has to happen."

Nova; "The danger is that if you delegate thinking [inside education] to a computation service... you may not actually understand enough to know if the answer that comes back is correct."

Russell; "That's a valid concern."

Q&A

"You rely more on your computational engine than natural language... but you lay a lot of emphasis on the linguistics in your system. So if it's not NLP what is it?"

Russell; "Domain linguistics, mainly; mathematical language, engineering language, etc... We think about how people describe things and search in these domains... and crawl the web looking for examples of how people use language in these domains."

"Stephen is focussed on quality of data, which is important to a lot of people here. There aren't a lot of tools. In addition to making your data store, I wonder if there might be scope to make some of your data curation tools available to the community, to improve the data out there."

Russell; "Great point. Can we make these tools genuinely useful to people, without creating a support nightmare..."

Carla asks each panellist to describe the differentiating aspects of their product in 'one or two sentences;'

Tomasz; "we receive about three times as many questions as other search companies. We want to answer questions the best we can from multiple sources... using structured and unstructured data."

Scott; "Bing really focusses on understanding the intent behind queries, and organising the page to help people get to their answer much faster."

Peter; "We focus on being comprehensive, accurate and fast... so we have to keep on innovating in crawling, ranking, systems engineering. One thing that differentiates us... most companies decide whether to focus on marketing or sales. We focus on engineering."

Riza; "We are a complete semantic search engine, from the bottom up. We don't even have an index. We've optimised the entire process for semantic operations. We focus on credible and dynamic content, and offer users a new perspective." Instead of popularity, they focus on credibility.

William; "True Knowledge is a platform that does direct question answering. There's a knowledge base and an inference engine to answer questions we haven't seen before." True Knowledge tries to 'help when it can, and stay quiet when it can't,' as can be seen demonstrated in their recently released Firefox plugin.

Andrew: "Yahoo! is very aggressive about semantic annotation... SearchMonkey is about acquiring semantic information and surfacing it in search results on the page."

Carla mentions Tom Tague's keynote from yesterday, where he suggested that 'semantic search is an answer to a question no one is asking'... so "why do we need to change search?"

Tomasz responds, suggesting that users don't necessarily demand new products that subsequently become successful. eg; no one was asking for the iPod before it launched. "When they see it, they will want it."

Turning to Google and Yahoo!, Carla asks them "why do we need to change search?"

Peter... "as an industry, satisfaction is very high... but that is just because that's what people know [now]... People don't like technology... people like solutions. When we deliver it, people will want it."

Andrew; "Does search need to change? It already is... Today, on any major search engine, if you search for a restaurant, you'll see structured information about that restaurant; reviews, phone number, etc... This has been accelerating over the last 3-4 years... When we put this information up, and trigger it correctly, we see far higher levels of engagement from our users than anything else."

Carla; "it may be a stupid question, but it has to be asked; what is semantic search?"

Scott; "it means a lot of different things. At Powerset we focussed on understanding the meaning in web pages, so we could present them, rank them..."

Carla; "Has Powerset's focus been diluted by the [Microsoft] acquisition?"

Scott; "No."

Carla asks Riza; "Someone from Hakia that I spoke to last year said you were the only one doing 'true semantic search.' Is that true?"

Riza; "No... Semantic Search can enrich search results... Semantic Search can improve precision/disambiguation... Semantic Search can organise results better. In the future, search will move to more conversational systems, and for that you really need semantic technology."

Carla; "How do you measure the 'semanticity' of a search engine?"

Tomasz; "That's my favourite question... We took a sample of 'equivalent' queries from the logs, and ran it to evaluate ranking etc; does the search engine give similar answers to questions like 'Top 10 songs' and 'Top Ten songs,' etc. Should they?"

Andrew; "It's incredibly hard to understand what a user will like... if you mess with the logo, it changes the perception of the results... if you make tiny changes, it can have a big impact on perception... When it comes to understanding semantic contact in search, we should identify the task the user is trying to solve... and have a metric that's aligned to that use case... We can break search queries today into different classes; how do we do when a user is trying to book dinner, or a vacation? Semantic Technology should be judged on its impact based on these task metrics rather than any underlying notions of entity resolution, etc... SearchMonkey, for instance, lets users inject structured data into the process... The information can be incorporated in any way... and change how the results are presented. We have about 15,000 people in our development community, changing the way those results are presented every day."

Tomasz; "I would expect a semantic search engine to deliver equivalent results to queries that would appear similar to a human being; 'Top 10 songs' and 'Top Ten Songs' should deliver the same answer. Today in most mainstream search engines they don't."

Carla; "Search v. Answers. True Knowledge is billed as 'the Internet Answer Engine;' is it necessary to move search to an answer-based format, or has Google trained users to think in keywords?"

William; "We support both keyword search and full-text questions. It's important to answer users' questions."

Peter; "Different types of answers are appropriate for different types of questions; sometimes the answer is a fact, or a page, or a series of results to support a process of study. To say there's going to be one technology or one type of answer doesn't make sense."

Riza; "You could be asking a 'where,' 'why,' 'how' type of question. Questions are important, and the search engine needs to be able to interpret the mode of the question and return results appropriately."

Carla; "You mentioned talking about the credibility of search results. How do you define a 'credible' search result, and how much of a need is there really? I'm not hearing users question the credibility of search results they see today."

Riza; "Practically, credibility is important in 'serious' subjects; medical information, etc. You want to know where the results come from and how credible they are. When it comes to credible content, you can't really do a statistical search or have a 'popularity vote,' because much credible content isn't 'popular.'

Scott; "People's expectations for credibility are different depending upon the query. If you ask an 'instant answer' type query you expect the answer to be credible. If you do a broader search, you expect a mix of results to be returned"

William; "If a system understands structured knowledge, it can understand when different sources contradict one another"

Riza; "A system doesn't need to know what's credible; we can go to a librarian for that. Hakia doesn't decide whether a resource is credible or not; we use librarians for that"

Tomasz; "If you ask for the capital of Japan we expect a single answer. If you ask about taxes, maybe the IRS is the best source but there are others. If you ask 'how to get rid of acne' you expect a lot of results."

Carla; "We've seen three news-making launches in the past month; Wolfram Alpha, Bing, Siri. Is Wolfram the first step towards 2001? How is this engine valuable to those of us who don't need to solve complex maths?"

Scott; "it's not the first step... we've been working on these problems for a long time. There are a lot of questions people want to ask about the types of data that Wolfram aggregates... We see these things as part of full-search services. Powerset has moved along this path as well, pulling structured data in response to full-text queries."

William; "Wolfram is a tremendous effort. An interesting example of question answering with structured data. I think people will find uses for it in particular use cases; I spoke to someone who'd used it to calculate when his visa expired, because it could do date calculation. I think there will be use cases in various scenarios; maths, nutrition information, etc... if you remember that it has that sort of information and remember to go to it... However one thing it doesn't have is a decent back-fill. If it doesn't have the data, or doesn't understand the way you asked the query, it gives you nothing. We try to keep quiet and fail over to standard internet search in that sort of circumstance."

Carla; "Does a semantic search engine know how not to answer a question?"

William; "that's absolutely fundamental. You need the ability to reliably keep quiet when you don't have the answer... and fail over reliably to other search services. [True Knowledge does try to do this...] "That requires very high quality semantics."

Andrew; "One way to characterise the approach of Wolfram Alpha is that it's a centralised approach. The Wolfram Alpha team goes out to find data and bring it in-house to convert to a standard form. A different approach is to have an ecosystem contributing data in the public eye... It's not clear yet how much of a value-add is going to come from this centralised knowledge mapping approach. Yahoo! is focussed on the ecosystem approach, and helping people with knowledge to make it available."

Peter; "Our inclination would be that we don't want a closed walled garden. We want all the information available to combine in different ways. We want the information to be open, and the tool set to be open for mashing up in different ways."

Scott; "If Wolfram Alpha hadn't taken a walled garden approach they might never have launched a product."

Tomasz; "Wolfram Alpha is great, but it's not a search engine"

Carla; "Siri... caused a lot of buzz, uses True Knowledge... what are your thoughts?"

Andrew; "To be counter-cultural... the notion of getting much deeper and assisting a user with a task is spot on. We're going to see much more of that. Search has tended to be stateless. Each query you enter is more or less processed without context. Yahoo! is rolling out more stateful search tools, and other companies will do the same. We expect people to use these tools on lots of devices. Would be expect people to come to the same place for purchase, navigation, etc? Do we expect one interface? There are going to be virtual assistants... I just don't know if they're going to be embedded into a search box."

Scott; "Conversation is the ultimate user interface... but it's not clear that I want to have a conversation with my laptop during the working day. How do I display the results? But there's a huge role for conversation and dialogue in refining search and getting a user to their results faster."

Tomasz; "What is the goal of Siri? If you try to go to broad you become a search engine."

Scott; "When people have a conversational interface, they won't speak in keywords."

Carla; "What are the larger goals for Bing?"

Scott; "Bing is trying to simplify key tasks that people do when they come to a search engine. In travel, health, shopping, we can understand what people are trying to do, and get them to better results faster. The thinking has evolved from ten blue links to the whole page, and organising things to help the user by understanding their tasks."

Carla; "Peter; what did you think of Bing?"

Peter; "I like the idea of innovation in the user interface. There's a lot of room for that. There's been a lot of emphasis on getting the ranking right. You still need to do that, but other things are important too. I'm usually happy with results on my big screen. On a mobile device, I'm usually not happy with the results I get."

Despite the wider economic situation, attendance for this fifth year of the event feels a little up on last year, and there's clearly real enthusiasm in the buzzing Halls.

Tague's Open Calais has been one of the success stories for useful and easy application of semantic technologies beyond a core community of enthusiasts and adopters, and has been covered here and on Cloud of Data a number of times since it launched. Just today, they announced a new set of partners and a postal service that should remove one more perceived barrier for another set of potential adopters.

Speaking to the theme of 'Web 3.0 - the Web of Me,' Tague's abstract suggests;

"The mainstream adoption of Web 2.0 technologies – from RSS feeds to social networks – is hastening the demise of the portal. With each new face on Facebook, and each new Twitter account, our once routine habits and traffic patterns shift. This wave of change in the way we consume, transact and interact on the Web is dis-intermediating 'destination' sites of all kinds. Our once centralized content has been atomized.

And yet our fundamental problem persists. We're overwhelmed with input, yet still can't find the one thing we need... now.

Semantic technologies – and the content interoperability and Linked Data connections they beget – offer new hope. That is not to say the answer lies in building new search engines, and few would argue for another news aggregator. Rather, our point of inflection lies at the point of consumption. Our task is to simultaneously refine and enrich our digital experience of everything from content and community to commerce."

Early on, Tague made a 'non-apologetic statement;'

"People need to start deriving financial benefits from semantic technology. It's time"

Absolutely!

Tague looks back at the move from 'Web 1.0,' described as 'the last Web we agreed on,' to 'Web 2.0,' which he sees as largely defined by the 'addition of social.' Today, he reckons, we are 'extraordinarily content-rich', 'extraordinarily information-poor' and 'experientially deficient.' Despite a wealth of content, we are failing to make the most of it.

'We're at the inflection point' where 'innovation is exploding' as we move from developing and inventing toward mainstream adoption of technologies in the semantic technology space. Lots of things will be tried; 90% will fail, but that's ok.

'Everyone needs plumbing,' and that's what Calais is; semantic plumbing. 13 version releases in 18 months; about 100 presentations, 13,000 registered Open Calais developers, a million great ideas.

Tague reckons the various efforts he comes in contact with fall into six broad buckets;

Tools; Social; Advertising; Search; Publishing; Interface.

First, Enabling Tools. Data Management, Data generation, Databases, Integration and workflow. 'A big yes.' 'We need tools.' Everyone needs tools, especially as you move from early adopters toward the mainstream. Tools build the bridges that cross the chasm to enterprise adoption.

Enterprise adoption will not happen because it's cool. Enterprise adoption will not be talked about on Twitter. Enterprise adoption will happen because it's cheaper/faster/better than what they have just now.

'Tool vendors need to simplify their story; it's not about more functionality.' 'If I can't understand your story, then Enterprise IT certainly can't'

Second, 'let's put some frosting on top of social.' 'Wouldn't it be cool if we could...' Some of it might be cool, but there's a challenge in monetising social. Adding frosting to the top of an industry that hasn't worked out its own monetisation is fraught with risk.

'I haven't seen a compelling story yet.'

Next, advertising. Almost a dirty word in the semantic technology domain last year. But advertising is fuel, and semantic technologies have a clear role to play in enhancing advertising (see my podcast with Scott Brinker from last year...).

Semantic search; 'the semantic industry's brilliant yet under-achieving child.' The answer to a question no one is asking? General, consumer-facing semantic search... directly competing with Google et al? Not viable.

But vertical search in specific domains... a huge growth opportunity, and people are willing to invest the time, effort and money to make it happen. Room for a handful of players in each domain?

Search; 'a bifurcated marketplace.'

Publishing; content producers, editorial/aggregation, 'robotic publishing.'

'Classic publishers can get enormous value from this technology... not all of the value is in the user experience.' Much of the value is being found in the back office, making existing data and investments work harder.

Little value in 'robotic publishing,' because the content isn't that readable. Aggregation services like Huffington Post and Daily Me present 'enormous opportunities.'

Interface; gaming a huge and growing market. $57bn industry. A 'seamless, interactive and responsive experience,' it's 'graphically engaging and fun.'

Zemanta, AdaptiveBlue, Feedly, Apture et al 'trying to make the consumption experience different' [better?]. Not suggesting that these are like a game, but many of the drivers may be similar?

"People are on their mobile devices and in the browser; go where the people are." Which links well to the next keynote... :-)

"Do you care about semantics or about user value?"

"Don't fund/buy semantic infrastructure beyond what you need; use infrastructure built by others where possible."

"Think very hard about the user experience; make it compelling and exciting."

Following Tague's presentation, Tom Gruber took to the stage to talk about Siri; a company building a Virtual Personal Assistant (with an interesting iPhone app to start things off) that we discussed during a podcast last week. As Gruber's says;

"We are beginning to see a new interaction paradigm for the web: the Virtual Personal Assistant (VPA). A VPA is task focused: it helps you get things done. You interact with it in natural language, in a conversation. It gets to know you, acts on your behalf, and gets better with time. The VPA paradigm builds on the information and services of the web, with new technical challenges of semantic intent understanding, context awareness, service delegation, and mass personalization.

Siri is a virtual personal assistant for the mobile Internet. Although just in its infancy, Siri can help with some common tasks that human assistants do, such as booking a restaurant, getting tickets to a show, and inviting a friend. We will describe the technology underlying Siri and how it fits in the larger ecosystem of services and data providers. And we will offer a vision of where assistants like Siri are going."

Tom starts off by showing the Knowledge Navigator video from Apple... which dates all the way back to 1987. Many of the ideas are now coming to fruition; touch screens, a global network, awareness of temporal and social context, speech in and out, a 'conversational interface,' 'delegation of work' to the machine, and trusted use of personal data.

Is the Knowledge Navigator possible today? 'No, but we're getting there.'

Siri is pretty close... in certain well understood contexts, as Gruber shows in a video demo of the evolving iPhone application.

What is a Virtual Personal Assistant? It does things for you; it's task-oriented. It understands your intent via a conversational metaphor. It gets to know you; it's not the same for everybody, unlike a search engine.

'Service delegation [like Siri]; the mother of all mashups'

'Context is king' in communicating with a VPA; where am I, what time is it, who am I, etc.

"This really is the beginning of the age of the start of Virtual Assistants."

Need to solve authorisation/ authentication. If we reach a 'data commons' there will be more, better, information to drive choices and decisions.

Tom Tague is a regular member of the Semantic Web Gang podcast, which I moderate. Tom Gruber was the latest guest in my Executive Briefing podcast series.

It should be remembered, of course, that broadly similar approaches are already taken to managing and navigating data inside the data centres of large corporations where Autonomy, FAST, Endeca and their peers provide powerful capabilities.

I recorded a podcast with Endeca Chief Scientist Daniel Tunkelang in January and, by chance, spoke with Robin Johnson yesterday. Robin is CEO of FT Search, part of the Financial Times Group, and responsible for a new vertical search tool called Newssift. Newssift combines components from various technology companies (including Endeca, Nstein, Lexalytics and ReelTwo) to offer a useful means of learning more about businesses and the external factors affecting them.

This month's episode of the Semantic Web Gang takes a look at both stories, and Gang members share their impressions on the news and what it might mean moving forward.

Next month's Semantic Web Gang will be coming live from the Semantic Technology Conference in San Jose.

Drawing upon years of experience in the development and deployment of Semantic Web solutions, Eric has spent the past few years considering the ways in which semantic technologies could bring structure and value to the increasingly visible online conversations around products and brands.

Have a listen, and share your views on the ways in which this might impact your brand, or your interaction with those of others.

A report from the island by Yahoo's Peter Mika begins this month's episode of the Semantic Web Gang podcast, and leads to a wide-ranging discussion of the role that vocabularies and ontologies continue to play within the Semantic Web. The very nature of these ad hoc VoCamps says much, though, about the way in which attitudes have shifted away from expectations that massive all-encompassing ontologies are the best way to help machines to reason about the world around them.

The conversation has just been released as a podcast which looks at the rationale behind the company's offer and the specific licensing choices that beneficiaries are asked to make.

Have a listen, and see if the Connected Commons might help your next project.

Disclaimer: Talis is my former employer

As the company's press release notes,

"True Knowledge offers two distinct API services for developers: the 'Direct Answer API' and the 'Query API.' The Direct Answer API allows developers to leverage True Knowledge’s natural language question answering technology, giving any search site or application the ability to provide a single direct answer for questions asked on any subject in plain English. This is especially well suited to mobile applications where providing a lengthy list of search results may be impractical.

The Query API allows developers to bypass True Knowledge’s natural language translation system and directly query True Knowledge’s knowledge base using a simple query language. This allows automated systems such as web and mobile applications to tap into True Knowledge’s vast machine-understandable knowledge of the world, making them behave more intelligently."

The company was founded in August 2005 and is based in the British city of Cambridge, at the heart of 'Silicon Fen.' A $4m Series A investment round was closed in July 2008, led by Octopus Ventures.

I spoke with CEO William Tunstall-Pedoe ahead of today's announcement to see how the core knowledge base continues to improve, and to discuss the company's plans.

For those who haven't tried it, True Knowledge offers an interesting slant on attempting to answer your question rather than simply return hundreds or thousands of documents that might contain the answer as traditional search engines tend to. Tunstall-Pedoe quoted Google's Larry Page during our conversation, noting that Page has asserted that

"the perfect search engine would understand exactly what you mean and give back exactly what you want."

It is this that True Knowledge attempts with their 'Internet Answer Engine,' and core to their solution are a comprehensive (137 million facts, and growing) knowledge base, a proprietary system for understanding a query and a powerful inference capability that enables the system to answer questions more reliably. Part of that reliability, as Tunstall-Pedoe frequently stresses, lies in the system's ability to know when it doesn't know the answer. Along with a success rate of less than 50% for providing answers to questions, this may seem little more than an academic curiosity, but an ability to reliably know when to fall back to less structured approaches (such as passing the query to Google) is far better than 'guessing' or delivering wholly inappropriate responses... especially once the Answer Engine's capabilities are embedded in some third party site.

True Knowledge's process of inference also allows the system to cope with ambiguity, and even with contradictory 'facts.' During our conversation, we told the system that President Obama was born in Cambridge. It allowed us to make this assertion, but subsequent analysis of the overwhelmingly contradictory data drawn from elsewhere in the knowledge base means that it was deemed to be untrue and flagged as such.

A different query, in which I ask 'How far is San Jose from SFO?,' shows both how the system copes with ambiguity and the manner in which supporting facts are drawn from sites such as Metaweb's Freebase.

The current True Knowledge home page is not going to draw huge numbers of users away from their search engine of choice, but that isn't really the point. As Tunstall-Pedoe pointed out, the site is intended to showcase the company's capabilities and facilitate the addition of new knowledge (as well as the millions of facts drawn from Wikipedia, Freebase and a growing body of licensed commercial content, over 120,000 facts have already been added by individuals in the beta programme.) The real utility of True Knowledge will lie in licensing the underlying system for use in vertical and horizontal third party applications, and public availability of the True Knowledge API begins that process. There's a long way to go in further extending the knowledge base, suggesting that vertical search applications may be the first to sign up; it's much easier to approach comprehensiveness within a bounded domain than across all areas of knowledge.

The market for semantically enhanced search is growing crowded, and stalwarts of the search industry have been hard at work too, with Google and others getting increasingly good at returning actual answers to factual questions.

Tunstall-Pedoe used a slide to demonstrate the differentiation the company sees between itself and 'obvious' competitors such as Wikipedia, Freebase, and hotly anticipated Wolfram Alpha. Key differentiators in the diagram included True Knowledge's ability to infer (something Wolfram Alpha also claims), its language independence (although currently only available in English, the concept extraction techniques used by True Knowledge should work equally well in other languages), and the system's reliance upon an internal ontology comprising 20,000 classes (plus biological species, product information, etc). True Knowledge (unsurprisingly) scored far better than the competition, but in a market that also includes the likes of Hakia and Powerset (neither of which could usefully answer my question about San Jose and SFO) the true picture is a lot more complex.

True Knowledge is certainly interesting, and frequently impressive. It remains to be seen whether a Platform proposition will set them firmly on the road to riches, or if they'll end up finding more success following the same route as Powerset and getting acquired by an existing (enterprise?) search provider.