How big data will solve your email problem

How big data will solve your email problem

Summary: That deluge in your inbox needs to be handled. A team of Israeli researchers thinks big data has some answers that can help.

SHARE:
iStock_000018635994Small
Image: iStockphoto

Over 150 billion email messages are currently being sent every day from about 3.5 billion email accounts worldwide. So, it’s easy to understand why we miss important messages as we struggle to keep up with the surge. For some, the situation has become so bad that email is no longer a reliable way to get in touch with them since they can’t quickly sort out the important stuff.

A new research project coming out of Israel looks to solve part of the problem by using big data to boil down email messages to their most important information and summarize them to be digested much more quickly, especially on mobile devices.

mark-last-bgu-250px
Dr. Mark Last, pioneer in text summarization

The project is led by Mark Last, an assistant professor at Ben Gurion University in Be’er-Sheva, Israel, and it’s focused on using algorithms to summarize blocks of text into their most important elements. From the standpoint of email, this could have two main benefits: 

1. Create one-sentence email summaries to be used in preview panes so that users can quickly flip through a list of messages and see the main idea of each email without having to open it.

2. Summarize long emails into 100-200 words that highlight the key points.

Last and his team of researchers at BGU are accomplishing this by using the tools of big data. In fact, Last has been working on big data and using it to solve problems since 1996 when he was a PhD student at Tel Aviv University. That was long before it was ever called “big data.” Back then it just data mining with unstructured data -- one of the key elements of big data -- and Last’s PhD sponsor barely understood the web mining and text mining research that he was doing.

Now, Last, who was born in Russia and came to Israel as a kid in 1977, is putting that experience to good use in what has become one of the hottest fields in IT. In 2008, he became a professor of Information Systems Engineering at Ben Gurion University and one of his big projects as been using text mining to find terrorist sites on the web.

There are tens of thousands of terrorist organization sites on the Internet, but they often disguise themselves as news, information, or community sites. Last and his team have used algorithms called “characterization models” to scan the web and pinpoint terrorist sites by identifying words that they use repeatedly, such as “enemy” and “martyr,” and phrases that they try to dance around, such as saying “human bomb” rather than "suicide bomber."

Clearly, this kind of data mining is different from the text summarization that is used in the email research mentioned.

"In data mining, in text mining, we have different methods, different tools," said Last.

Nevertheless, the work supports each other and both are aspects of big data. The text summarization work started as an initiative to help summarize lots of news articles, short books, and documents on the web. This was especially useful to intelligence agencies, who used this technology to comb through thousands of news reports and web documents as quickly as possible. They could look at the 100-200 word summaries of the pages/documents and then decide which ones deserved a further look verses which ones they could avoid wasting time on.

Out of that grew the idea from Last and his team to apply this idea to email, where summarization could be employed to help quickly sort through messages and find the ones that you need to pay attention to versus the ones you can safely ignore.

The work started in English since so much of the web is primarily in English, and there are already very good natural language processing tools in English.  However, the work has now progressed to Hebrew, Arabic, and other languages as well. Ultimately, they have developed a new method of text summarization that is language-independent, and that’s a big part of the magic.

Israel and the next Silicon Valley

How Israel is rewriting the future of cybersecurity and creating the next Silicon Valley

How Israel is rewriting the future of cybersecurity and creating the next Silicon Valley

If you're looking for the rise the next Silicon Valley, the place to watch is the Negev region in Israel. Its new cybersecurity juggernaut is breaking new ground in public-private partnership.

The algorithm scans sentences in the document and first calculates metrics such as number of words and the relation of words in the sentence. The second stage is weighting the sentences to find the most important ones. The algorithm also looks for summaries created by humans (in the case of documents and news articles) and then looks for similar words, phrases, and themes to help the text summarization.

While the technology is improving by leaps and bounds that doesn't necessarily mean it's coming to any commercial email services in the immediate future.

"In general, we’re not here to develop products," said Last. "We're here to develop methodologies."

Last said that his team has applied for a U.S. patent on their text summarization methods. Ben Gurion University is very resourceful about licensing and selling its patents to commercial companies (as discussed in my article on Israel's quest to build the next Silicon Valley), so there's the potential that this will eventually be productized. There's also the possibility that a company like Google is already working on something similar.

However, the email service that could potentially benefit from text summarization first could be Yahoo Mail, since Yahoo bought text summarization pioneer Summly in March. Summly was the brainchild of British teen entrepreneur Nick D'Aloisio. Last said his team exchanged many emails with D'Aloisio on the similarities of their work (before Yahoo bought D'Aloisio's company) and was very complimentary about the work that D'Aloisio was doing and the attention that it received. Last viewed it as validation for the power of text summarization and its rising importance.

I like to think of text summarization as one of the fruits of big data that will have a direct impact on consumers and professionals alike.

Also read

Topics: Big Data, Going Deep on Big Data

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

6 comments
Log in or register to join the discussion
  • words

    Just a small correction - use versus (comparison) in place of verses.
    robb@...
  • It appears that the word "big"...

    ... was the secret word of the day at ZDNet: big email, big customer analysis, big telco data, big brain. Feeling a little Freudian today, are you all?
    Vesicant
    • Well now

      Thats big of you to point that out.
      ammohunt
  • big data?

    I think the author is confusing "big data" with data mining of unstructured data. The given example is incorrect. Just a textual analysis/summary of a paragraph of text is not a "big data" problem. Applying statistical methods to determine which combinations of let's say five words can be reduced to only two words is not a "big data" problem either. Storing billions of billions of billions of emails, indexing them, data mining in them, development of proper statistical analysis algorithms to handle them, finding ways to access them on a harddisk or a database that simply cannon handle these many separate files would constitute a "big data" problem.
    pupkin_z
  • another "solution" from mossad

    great...just a new way of spying on people
    ljenux
  • Data Overload

    Great article Jason!

    It provides great insight into the reality of the many frustrations professionals experience and the crucial role Ci and data management tools play in today’s organization. Data overload is a serious problem, especially if it’s affecting our most effective way to communicate in the business world.

    clearCi recently published a white paper discussing how companies can conquer data overload with greater speed to intelligence. Feel free to check it out here: http://clearci.co/speed2intel
    FatimaZimichi