Kathy Gibson reports from CeBIT, Hanover – Big data and analytics has reached the peak of expectations and is well on its way into the trough of despond, according to Gartner’s hype cycle – but why has it disappointed?
Dave Goldberg, CEO of Survey Monkey, says the technology has started to slide because it has failed to live up to its promise. But there are ways to make big data work as intended.
Overall, many companies have spent a lot of time and effort on big data projects, Goldberg says, with 54% of German companies, 50% of Us firms and 42% of Chinese companies having invested in the concept.
The problem is that most organisations felt that these investments hadn’t paid off, that the technology hadn’t delivered on its promise.
In fact, only half of companies survey felt that big data projects had solved even some of the problems it was implemented to address. And a whopping half of respondents believed big data had solved nothing at all.
“There is a lot of disillusionment among companies that have tried big data,” Goldberg says.
Part of the reason that people are disillusioned, he adds, are that expectations were too high. “Big data was sold as a cure for every business ailment and the reality is that it’s not meeting expectations.”
Another problem is that there is simply too much data being collected, and companies find it hard to make sense through all the noise. In addition, this data is usually siloed and not connected, making it difficult to use it effectively to make decisions.
“However, now being at the top of the hype cycle is not a bad thing – it can now become a valued technology. Companies can now go beyond the hype and find ways to effectively use it.”
Goldberg explains that there are two types of data – implicit and explicit – and companies have to understand how to effectively gather and use each type.
implicit data is what we generally think of as big data; it’s what is collected by activity – by clicking on a link etc. It’s what happens when people do things.
Explicit data, on the other hand, is what people actually tell someone – it’s what they really think.
“We think of implicit data as big data because you can get it at scale,” says Goldberg. “But it’s what gets people creeped out, and makes them nervous about data privacy. Privacy is a huge deal people don’t like the idea that they are being observed.”
Every major retailer in the world is trying to analyse data to understand what their consumers are doing so they can more efficiently market to them, and serve them. “So it’s not always a bid thing, but it can be disconcerting,” Goldberg says.
It’s also easy to get caught up in the gathering of data and use it to generate an inaccurate result, he warns. “There is a lot of implicit data that is being captured but it doesn’t always lead to better insights.”
What is generally more valuable, Goldberg says, is explicit information that the user has volunteered. “If you really want to know what people like, you really have to ask them.
“Explicit data is what someone has told you, so you are not guessing. And it’s not creepy to people because they know they volunteered the information.”
The reason explicit data has fallen from grace, he says, is that it’s traditionally time-consuming, expensive, non-scalable, it isn’t often available electronically, and can be cumbersome.
“But technology is changing all that,” Goldberg says. “You can now easily gather explicit data at scale. For example, when you review something on Yelp, you are creating explicit big data at scale.”
It’s how Survey Monkey runs its business; and the organisation can boast that it collects 3-million questionnaires every day, with 29-million answered, generating 80Gb of new data.
Another example is where Google picked up a trend that the company was losing more women staffers, and so conducted a survey to try to find out why.
“What they discovered is that they didn’t have a women problem; they had a new mom problem,” Goldberg says. “They realised that new moms didn’t find the 12-week maternity leave was satisfactory.
“On the basis of the findings, the company instituted a five-month maternity leave benefit and dramatically improved employee retention among women.”
