The emergence of the Internet of Things (IoT) is now playing a central role in intensifying the volume of data that is being collected, according to the seventh EMC Digital Universe study.
The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things study, conducted by IDC, has revealed the digital universe is doubling in size every two years and will multiply by 10-fold between 2013 and 2020 — from 4.4 trillion gigabytes to 44 trillion gigabytes.
EMC Australia and New Zealand chief technology officer Matthew Zwolenski said while consumers are currently the main driver for this "explosion in the digital universe", connected devices, such as smartphones, fitness tracker bands, and scales that are connected to the internet, will drive it further.
"Today the digital universe is mainly being driven by people, and that's one of the highlights out of the analysis. Two thirds of information is generated or captured by consumers who are accessing a website or sending an email, yet 85 percent of that is being stored by a company," he said.
The study said the IoT will influence the massive amounts of "useful data" — data that could be analysed — in the digital universe. In 2013, only 22 per cent of the information in the digital universe was considered useful, but less than 5 percent of the useful data was actually analysed — leaving a large amount of data left untouched in the digital universe. By 2020, more than 35 percent of all data could be considered useful data due to the growth of data collected from the IoT, but it will be up to businesses to put the data to use.
"A lot of companies have transient data. For example, when a customer checks-in to a hotel, the hotel obtains data about the customer, but as soon as the customer checks-out that data is deleted, and this happens quite often," Zwolenski said.
"But what companies are realising is that if they could analyse the data, they can find out more information about their customers, and possibly then offer deals suited to their travel patterns. But to do that you need to keep that data, and that is what's driving this explosion in the digital universe, because every one is starting to keep that data to analyse it."
Zwolenski highlighted this is has been particularly noticeable amongst Australian banks, telcos, and retailers who have started to realise there is "real value in this data".
While data is creating opportunities for businesses, it is also creating particular challenges too. Based on the study, the world's amount of available storage capacity of unused bytes across all media types is growing slower than the digital universe. In 2013, the available storage capacity could hold just 33 percent of the digital universe. By 2020, it will be able to store less than 15 percent.
Zwolenski said to address this, Australian companies are looking to build a Hadoop environment, an open-source cloud software framework that will enable businesses to store their "big data" based on their storage levels, rather than rely on traditional hardware. This format is already being used by global companies, such as Google and Facebook.
He said a major benefit of a cloud-based framework is it reduces turnaround times, as what would usually take a company 8-10 weeks to roll out a new server, can now take them 8-10 seconds.
This means companies, such as local banks — who Zwolenski said are concerned that global companies, like Amazon and Google, could potentially enter the Australian banking market in the future — can use this form of technology to store data, analyse it, and build services quicker to meet the needs of end-customers.
"A study has shown that Gen Y will change banks based on a bank's apps, and that's the number one reason why they change banks. So the question is who will build the next best app, which is why there is a pressure on banks to build mobile apps," Zwolenski said.
"There's now a faster way to build applications and roll it out quickly. This is one thing that is driving this digital explosion of connecting more consumers to the internet."
Another challenge businesses are coming across in the growth of the digital universe is gaining support from their IT staff for the adoption of cloud to their growing data collection.
"There's a war between IT teams and cloud providers," said Zwolenski, who believes IT departments fear they will be held accountable, and not the cloud service provider, if wrong applications are placed in the cloud.
He suggests a key solution to this is education. Currently only 20 percent of IT departments have a good understanding of how the cloud can benefit a business, especially because "they've been doing the same thing for the last 20-30 years and something like this is so new, it can be hard to adapt".
Zwolenski said another resolution that can help overcome this tension is for businesses to take a hybrid cloud strategy approach. This would give IT control over what applications go in the cloud, while managing other important data internally.
"It should be a seamless process whether the information is put here or in the cloud," he said.