How social media and big data will unleash what we know

How social media and big data will unleash what we know

Summary: Today social media generates more information in a short period of time than was previously available in the entire world a few generations ago. Making sense of it and understanding what it means for your business will require all new technologies and techniques, including the emerging field of big data.


With this development -- as the world continues to become more and more social -- competitive advantage will come to those who understand what's happening better than their peers and can directly connect it to their business outcomes and other useful pursuits. Social networks and enterprise social software has long been driven by two things: The connections between the people that use them and the information they share.

Just as Facebook uses the insights gleaned through its analytics on how people behave to enable personalization and better user experiences, the same phenomenon has been happening on the Enterprise 2.0 side, most recently exemplified by last month's acquisition of Proximal Labs by Jive Software.

While gleaning insight and contextualizing interaction in social environments is nothing new, the challenge in doing so has been pushing the boundaries of available technology for some years now. Organizations across the social business spectrum -- the consumer and workplace both -- are only now beginning to understand the vast intelligence that can be derived by looking at millions of conversations taking place, mostly out in the open, between those engaging in social media. While there are certainly interesting privacy, legal, and regulatory issue with doing so, even with internal social networks, it's not likely to delay the growing adoption of such capabilities, given their potential value.

In the shorter term, the ability to analyze and mine data in scale within social networks is enabling a range of intriguing and useful applications that can plug into social media networks and make use of the knowledge inside them. Doing it well, however, has proven to be more than non-trivial, such as making making analytic sense of content types that are very large and opaque, such as high definition video, or sensing the connections between the thousands of unstructured natural language messages. Each of these require technologies that can handle the enormous scale, complexity, speed, and computation requirements in a way that remains cost effective inside a rapidly-rising exponential window. From this you can begin to see the challenge of traditional approaches to large data, which tend to break down fairly soon under large geometric growth.

The Intersection of Social Media and Big Data

Perhaps more to the point, and where the discussion of big data comes in, is that the key to social media interactions between people is that it leaves knowledge behind for others to find and reuse. This can be the original content that started the conversation or the subsequent comments, discussion, ratings, ranking, retweets, etc. These conversations will remain on the network afterwards, usually for a digital eternity, and forms an invaluable history and knowledge repository of society, culture, and business that can be discovered, brought back to life, shared, learned from, ad infinitum. Of course, some of this isn't inherently valuable by itself (much has been made about the signal-to-noise ratio of social media). Also, finding what one is looking for in the vast sea of a million or billion human conversations is a difficult task. Thus, separating the wheat from the chaff is where big data, and the analytics it makes possible, go hand-in-hand.

Related: How an accidental IT future is becoming reality.

But the deeper issue is not just finding the valuable nuggets in a galactic sea of social media. Rather it's knowing what you are able to know, or even what its possible to know as social media becomes a dominant form of communication (probably now the dominant form.) Last year at Defrag, I spoke of this as the difference between being able to find a specific needle in a haystack versus having the ability to discern the shape of the haystack itself. Social analytics has arisen precisely to help us make sense in the large of the endless flow of our activity streams and social news feeds.

Social Media: A Big Data Inflection Point

Though social networks may soon contain the visible sum of humanity's communication and interaction, the challenges of deriving what is increasingly called social business intelligence are two-fold. First, big data sets itself apart from previous approaches because it applies new ways of thinking about the capture, storage, and processing of truly vast amounts of data, precisely the kind that emanates from today's social media ecosystems. This includes the supporting technology, often starting with emerging tech such as data mining grids or MapReduce infrastructures (see my exploration of one example, Hadoop, here) as well as software architecture that is often surprisingly non-deterministic and non-linear in design. For a quick example, see this discussion of LinkedIn's challenges and counter intuitive solutions to data scale in social networks. In practice, this means that there is a distinct generational and technical divide between how most organizations are dealing with data today, and the very different things they'll need to do in the future.

Computerworld: As 'big data' grows, IT job roles, technology must change.

The second issue goes back to the old adage that "you manage to what you measure." In the big data world of social media, this means that ones analyzes what you know of to analyze. However, one of the things I hear frequently from business users of social media is that they'd like to "spot trends", to "know what's going to happen before it actually happens", to "get ahead of the conversation and see where it's going." Sentiment analysis, knowledge mining, aggregating conversations into trends, these are all possible when you know what you're looking for (and you have the technology that's capable.) It also helps -- and it's no mean feat -- if the tools you use are smart enough to tell you what there is to know, but that you don't know to ask for.

Right now big data still remains a specialized niche of technologies and techniques that makes fundamentally new assumptions when it comes to tackling and understanding vast amounts of data. While consumer social networks have long used big data techniques in the custom platforms they've built for themselves, enterprise social networks are just now starting down the road to catching up. Big data is now moving into the realm of mainstream IT. With this development -- as the world continues to become more and more social -- competitive advantage will come to those who understand what's happening better than their peers and can directly connect it to their business outcomes and other useful pursuits.

Where social media and big data are headed

For now, however, here's the short list of what the growing intersection of social media and big data will likely result in:

  • Big data-enabled applications that are plugged into consumer and enterprise social networks. As 3rd party software becomes increasingly embedded into social networks, these are analytic applications that can give you the latest social media "weather report", allow you to data mine, ask queries, view top trends, and display insights of various kinds only because big data approaches make it possible to happen quickly enough in the face of the full sum of social media knowledge.
  • The blurring of external and internal big data. The interconnectedness of everything is only continuing as the Internet drives the IT conversation to a greater and greater extent, including a growing blend of the consumer world and the enterprise. Tomorrow's big data solutions, particularly when it comes to business intelligence, will include a fully integrated view of this entire social media landscape.
  • Privacy, governmental, and regulatory concerns will grow. Much of this will look like Big Brother vacuuming up every scrap of people's behavior and knowledge and using in ways that were never intended. This in fact is already a significant issue in the social media space but the increasingly mechanization and maturity of big data, social media, and analytics will only make it more pronounced. Those engaging in crawling, analyzing, and sharing data garnered from social media, whether it is from consumers or employees, will have to think carefully about the implications. Firms should carefully play the role of steward as issues from intellectual property to ownership of data are resolved in the industry.
  • Analytics that finds you. Knowing what to ask for is an essential skill when it comes to extracting value from today's social media landscape. But the confluence of social media and big data will allow vital new intelligence to find you, before you know you need it. The field of predictive analytics, for example, requires very smart software combined with the ability to quickly run enormous amounts of speculative queries in a timely fashion. Big data will make that possible, at least until we reach the era of Really Big Data.
  • Cloud big data analytics emerges. The sheer data and compute volumes, combined with the highly decentralized nature of social media, is ideal for a cloud computing approach. Look for products that combine big data, analytics, and social media for on-demand, cost-effective solutions. Companies like IBM and HP are likely to lead the way here.

For another good indicator that big data represents one of the next big things in IT, I urge you to read Mark Gomes overview at Seeking Alpha of recent M&A activity in the big data space. In the meantime, the strategic roadmaps of those engaging in social media for business should now include big data and social business intelligence as a key component for reaping full ROI.

Topics: Social Enterprise, Collaboration

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • RE: How social media and big data will unleash what we know

    Very interesting article and right on the money. Couldn't have been written better! Brings together the various forces that leads to the conclusion that social media and big data will lead to revolution. Unfortunately, currently, the users are not in this revenue path since they have no control over who users (or abuses) their data. Users need to be in control so that they can get part of the pie!

    Privachi (, a privacy-centric social network, gives users the benefits of a social network while letting them own their social information. On Privachi, messages that a user posts are ?locked? in such a way that even Privachi servers can?t unlock them, only the user?s friends can. In addition, user updates, photos, and videos are stored in locations that the user chooses (even locations outside Privachi servers, if the users chooses them) ?to prevent any one service from knowing everything about the user. Photos, videos, or comments deleted by a user are truly erased since they are stored in the user specified location such as the user?s or dropbox account. We hope Privachi provides a fun social network for users while helping them protect themselves from social profiling and putting them back in control of their social data.
    • RE: How social media and big data will unleash what we know

      @privachi <br><br>I've not heard of privachi before...<br>--------------
      <a href="">Film-izle</a><br><a href="" title="Sinema izle">Sinema izle</a><br><a href="" title="Sinema Filmi izle">Sinema Filmi izle</a><br><a href="">Vizyondaki Filmler</a><br><a href="">hd film-izle</a><br><a href="">online film-izle</a><br><a href="">Recep ivedik 3 izle</a><br><a href="">Avatar izle</a><br><a href="">kolpacino 2 izle</a><br><a href="">incir receli izle</a><br><a href="">Ask Tesadufleri Sever izle</a><br><a href="">HIZLI ve Ofkeli 5 izle</a><br><a href="">Transformers 3 izle</a><br><a href="">Harry Potter Olum YadigarlarI 2 izle</a><br><a href="">Kung Fu Panda 2 izle</a>
      • RE: How social media and big data will unleash what we know

        @sinemafilmizle cok guzel filmler var evet sitenizde ama arabesk rap tarznda mzik iin lider tek isim ise ( dinleyin.Tavsiye ederim.
  • RE: How social media and big data will unleash what we know

    I agree ! Awesome article ! Nothing more topical ; in professionnal field, "social" data mining is perceived (by enterprise social networks users & their managers) as a supposedly easy target, I'd enjoy it... This article and related links bring interesting information for those (like me) working in social software deployment field and often confronted to such requests.
    May I also add, in everyday life it's not always easy to explain to friends fearing the big brother era, what are really the technical means (and human resources as well) required to use big data, especially if the stake is to find what is valuable to know, rather than "simply" find what is looked for.
    • RE: How social media and big data will unleash what we know

      i think the same
  • RE: How social media and big data will unleash what we know

    From a news and information sharing point of view, one of the big issues is the opacity of Facebook data. No appears to have cracked the aggregation/trending of material shared on Facebook (Likes, Recommends etc). Is this because Facebook is refusing access to this data at an aggregate level?
  • RE: How social media and big data will unleash what we know

    Great article! I agree that with social data, sensor data and even mobile data (i.e. location based social interactions) the need to address BigData problems is going to become more mainstream in data management for Enterprises. One of the advantages BigData and its approaches like MapReduce offer is the ability to find optimal solutions which will play in further supporting personalization and engagement through Gamification. Often in a social network, finding the best path to a solution could require looking at a multitude of paths, each with edge functions determined by a varying set of attributes. Physical and logical partitioning of data will allow for support of massive parallel processing to support analyzing smaller chunks of data in parallel against a set of analytical queries, gaining efficiencies in processing.

    • RE: How social media and big data will unleash what we know

      @YoshiNav I agree with you <a href="" title="wallpaper narutoshippuden">wallpaper narutoshipuden</a>
      <a href="" title="android cellphones">Android Cellphones</a>
      <a href="" title="wallpaper lucu">wallpaper lucu</a>
  • Big Data, Social Media and Enterprise Value

    Thanks for an insightful post, Dion. At Jive, we think there's no doubt about the value of the coming intersection of Big Data and Social Media for enterprises. As you point out, the combination of deep mining at very large scale over both structured and unstructured information can produce insights and intelligence previously hidden.

    For us, the value to companies is central to guiding how we?ve shaped the Jive platform. And while the Social CRM use case you concentrate on here is certainly an obvious place where our web-scale machine learning and data mining can make a difference for users, we think there are some additional possibilities as well.

    Take improving collaboration within or between enterprises for example. Yes, Big Data analytics can power dashboards of activity and help you monitor and even predict what?s happening. But what if you could use the same capabilities to personalize recommendations, intelligently filter huge streams of information, and draw the right people into the right discussions? What if you could mine not just Twitter, but combine the entire social web with your current and legacy enterprise systems (CRM, email, CMS, etc.)? What if you could use those techniques to learn what those people cared about most, and improved over time?

    The combination of Big Data and social media analytics provides entirely new ways of making information useful. Getting the right people working together towards common goals means creating social capital and trust in communities, and allows for new, higher quality conversations to take place and be made available in the future. It means being able to swarm the right resources ? which could be people, documents, conversations, or other artifacts ? around emerging community needs. And it means creating higher-performing teams that can deliver results that CFOs can measure.

    The combination of Big Data and social analytics will go far beyond "eye-in-the-sky" dashboards and sparkline indicators. The real emerging promise of Big Data and social analytics is creating a fundamentally new kind of technology that learns to adapt itself to you in order to surround you with the resources you need to be effective in a work context. That work context could be inside your organization, or collaborating with a partner organization, or working directly with external customers. Any way you look at it, Big Data techniques in social media will create entirely new opportunities to create meaningful relationships that can add value for enterprises.

    And that is an incredibly exciting future.

    David Gutelius
    Chief Social Scientist
    Jive Software
  • RE: How social media and big data will unleash what we know

    You point out an interesting trend, Dion, that will be crucial for companies that want to maintain a competitive edge. Users are generating an enormous amount of data on social media sites, posting information through status updates, links, videos and comments. While many companies have taken the first step to monitoring social media sites, as you say, the complexity of the task means that there needs to be multiple iterations of the technologies before we reach the utopia of social monitoring. It's not an either/or thing. The data that's being pulled in from social networks now is making us smarter about how a brand is perceived, or customer service issues, than we were even 3-5 years ago. Now, marry that information with the data we already have from other touch points like email, direct mail, POS, etc. to gain a complete 360-degree view of customers, and we're smarter still.

    By opening up access to the entire corporate ecosystem and incorporating information from social media sites, emails and call centers, businesses can achieve a supercharged view of the customer, creating a baseline, or framework, for expanding the frontiers of social media marketing as a channel for engagement. Those who are able to cleanse, organize and socialize the data at hand will be able to make smarter business decisions and gain a significant advantage over others. The technology will mature, sure, but there's real value to be had now. There are some real-world examples ( of how a number of social networking and ?social intelligence? companies are harnessing data to provide more value to members which may provide some ideas for other companies on what?s possible.

    Steve Wooledge
    Senior Director, Aster Data Product Marketing
    Teradata Corporation
  • RE: How social media and big data will unleash what we know

    Big data and social media is already beginning to fundamentally alter the online advertising business model. Every $1 of analytics solution value enables roughly $10 of online advertising revenue. Mobile applications and global mobile data traffic will more than double that ratio to $2-$3 per $10 within the next few years. The Insight era is just beginning.

    Jeff Vogel
    Calpont Corporation
  • RE: How social media and big data will unleash what we know

    It is equally important to analyze small metrics which we ignore most of the time. It can also helps in understanding user behavior. After all we do it for a user. <a href="">SEO India</a>
  • RE: How social media and big data will unleash what we know

    its always important to know about this.. you have to keep and eye on the <A href="">free article directory</a> for the info
  • RE: How social media and big data will unleash what we know

    i also found you can get more info by researching <A href="">Resume Writing</a>
  • RE: How social media and big data will unleash what we know

    Social media gets really really bigger each day. It's like avalanche.
    • RE: How social media and big data will unleash what we know

      Mediarun has been providing London SEO services and website optimisation services to clients spread over fashion, retail, travel and financial sectors.
      <a href="">SEO London</a>
  • RE: How social media and big data will unleash what we know

    I really agree with jeff here. Social media gets bigger each day.
  • RE: How social media and big data will unleash what we know

    We will all soon know where they are headed. Let's just hope that it is for the best. Remember effective <a href="">house selling tips</a> and do not forget <a href="">personal injury insurance</a> and <a href="">personal injury attorneys</a>.
  • RE: How social media and big data will unleash what we know

    I agree with the point about the emergence of cloud-based Big Data solutions. Realistically the majority of companies are probably not going to have the resources to process very large data volumes in real time. More importantly the analytical tools necessary to infer patterns and behavious will be beyond the reach of many. If you would like a first glimpse of Microsoft's cloud-based Social Analytics offering you might be interesting in this:
  • RE: How social media and big data will unleash what we know

    <b><a href="" target="_blank" rel="nofollow">arsiz bela</a></b>