How social data is changing the way we do business

It's a familiar litany now, that the openness and transparency of social media will unleash a wide range of compelling outcomes for our organizations, if we'll only embrace it. While there is little doubt that social media is one of the great phenomenons of our age, there are certainly those that think the hype surrounding it is also a bit over-egged.

Yet a growing set of compelling examples is showing us unique and vitally useful outcomes that are only possible in social media.  For those just catching up with the story, part of the uniqueness and power of social media is that it generaly makes the information that's shared -- using social networks, blogs, and other forms -- public by default.

This is in contrast to earlier forms of analog and digital communication, where we were required to have perfect foreknowledge of who should be involved in a conversation, such as their phone number(s), e-mail address(es), or other contact  info. Everyone else was automatically excluded from the process and the contents of the discourse itself was largely invisible and left little behind. 

In sharp contrast to this model, as we've learned through a decade of global online experimentation, is that it's often best when we don't overly call-out the precise identities of those we wish to converse with. Instead, we've discovered that it's better for us to let those who find value in what we're saying to find us (typically via search or recommendation by friend or technology.) This enables us both embrace and enable serendipity, emergence, and open innovation and this -- as MIT's Andrew McAfee recently pointed out for the business world --- lets us bring enough of the right people together dynamically to create the most interesting and useful results possible. 

The result is a rich tapestry of conversations and information that everyone can join in on if they desire, or benefit from later on.

Related: Social media taps more directly into the power laws of digital networks

It's also turned out that the default public mode of social media, though it can also cause no end of headaches for companies in industries where information tends to be very private and controlled, allows us to realize certain very significant new scenarios that older communications technologies simply couldn't (or sometimes, just wouldn't.) What scenarios you ask? The most important and broadest one is letting us perceive virtually all global conversation across social media in real-time and then analyze it. While observing such conversations has been possible for a while, the timeliness hasn't been easy to achieve. The same for effective and meaningful analysis.

Scale is the first part of the problem with such timely analysis of social media, especially when it must be thorough and accurate. As I pointed out in my last post, that there are hundreds of large social networks, millions of blogs, and countless online communities and forums to look at and engage with. Important and business relevant conversations can be found in far-flung corners that must be examined, or we potentially pay the consequences. As Douglas Merrill observed last week in the Harvard Business Review blogs, adding new signals is one of the single most valuable ways to improve data analysis.

Thus, forging an integrated picture that truly conveys the constant and endless stream activities and insights within social media conversation is a relatively new phenomenon, enabled by a growing raft of so-called "big data" technologies like Hadoop and Mahout.

Related: 'Big Data' or 'corporate spying'?

Isolating relevant meaning from the noise is the second part of the problem, because the form and content of social media is relatively unstructured and informal. It's also filled with rich media, particularly pictures, as well as video and audio. Given the vast diversity inherent in the nearly 1.5 billion worldwide participants, including their language, cultures, customs, and idioms and it's clear that even if we can actualy process all of this conversation quickly enough, making sense of it all is as big -- and probably much more substantial -- a challenge.

Yet perhaps it really doesn't have to be all that hard, depending on what we're trying to accomplish. In collecting compelling examples of what we can do by simply by listening to and analyzing social media, it quickly becomes pretty obvious that we don't necessarily have to boil the entire ocean of scale and semantics when it comes to making social media strategically useful. In fact, some of the best examples often take relatively simple data sets and infer the meaning by combining them together with straightforward geographic visualizations, for instance. That said, the more signal and analysis we apply, the more accurate and useful the insights will be.

Related: How social media and big data will unleash what we know

While it's clear that we're still in the early, formative stages of social media analysis, it's equally clear that there is a great deal of business and civic utility in applying its techniques to our lives and work today. I've collected 10 good instances of what's possible, with thanks to Tom Raftery of Greenmonk for identifying some of the stellar examples in this gallery.

Gallery: Ten examples of big data extracting value from social media

Note: The stream graph in the visual above was generating using Twitter StreamGraphs. A stream graph is a common technique employed to visualize the shape of online conversations over time. This graph shows the most recent 1,000 tweets involving ZDNet on the afternoon of November 7th. It shows that conversations about 'Microsoft', 'Skype', and 'mobile' were the most interesting -- or at least the largest -- public conversations of that time period about ZDNet.