If you don't really understand big data how about tackling medium data first? - Theo Chapman

The lifeblood of the information age is data and the prevailing wisdom is that the companies that can extract insights from data have an advantage over those that don't.

The problem with big data

‘In God we trust. All others bring data.” So said William Edwards Deming, American statistician, professor, author, lecturer and consultant.

The lifeblood of the information age is data and the prevailing wisdom is that the companies that extract insights from it have an advantage over those that don’t, because they can use it to understand their businesses better, drive product innovation and generate new sources of revenue.
Banks, telecommunications providers and retailers have been creating enormous amounts of data for decades and have created systems to drive insights about their customers. But the term “big data” refers to the huge quantities of raw data from outside the organisation that can be commingled with internal data and mined for intelligence.

Innovation stations

Analysis company Gartner says big data is “high-volume, high-velocity and high-variety information assets that require cost-effective, innovative forms of information processing for enhanced insight and decision-making”.

Not necessarily, says Matt Kuperholz, a partner in PwC’s modelling and analytics group. “Big data is simply using different tools and techniques to extract the full value from data.”

It’s not surprising, then, that respondents to a new BOSS and University of Sydney Business School survey showed a fairly good understanding of what big data is. Sixty-four per cent of the responses defined big data as one of – or a combination of – three things: multiple data sets linked together; the use of data to make evidence-based decisions; and large models which use data to explain complex processes.

What do you want to know?

Businesses should tackle big data by asking themselves a business question, says Sally Wood, professor of business analytics at the University of Sydney Business School.
“I always say to businesses, ‘Tell me what you would do with the data if you had all the data in the world. What is the research question you want to answer?’ ” Wood believes it’s the nature of the question that determines whether big data is the solution. So what sorts of questions require a big data solution?

Forty-two per cent of respondents to the survey said their company used big data. Among them is Staples Technology Solutions, which uses internal and external data sources to identify market opportunities.

“When we’re doing propensity analysis [which estimates the relative contributions of factors to outcomes] on customers, we’ll cross-check the information we have with external data,” says business head Karl Sice.

“And we’ll work with our marketing team, or in some cases [external] marketing organisations, to determine not only whether the information is correct, but also to help us understand which opportunities we’re targeting.”

Big data is also useful far beyond the realm of sales. “Some companies want to understand what factors affect leadership qualities,” Wood says. “All these things that were thought of as fluffy and non-rigorous suddenly can become much more evidence based.”

Public data 

At US recruitment firm SwoopTalent, Australian expat Stacy Chapman is harnessing data from multiple sources to find staff for her clients. Using publicly available information from professional sites such as LinkedIn and networking groups such as Meetup, SwoopTalent is able to identify individuals with particular professional skills – from Java programmers to insurance sales people.

“I can show you an interactive Venn diagram of the internet and social footprint of a particular skill set,” Chapman says. “I can show you which Java engineers are on GitHub and Stack Overflow and LinkedIn and SecurityFocus.com and wherever else they might be. People get a bit weirded out by this – it feels like we’re doing some kind of bizarre data stalking. We only get professional and skills stuff but there’s so much of it out there.”

Another option is to marry a client’s personnel records with SwoopTalent’s external sources to identify internal candidates who might have been overlooked.

“People feel uncomfortable about going up to an employee and saying, ‘I see you’re going to Hadoop meet-ups. Do you want to talk about launching [software framework] Hadoop here internally?’ ” Chapman says. “Is that creepy or a really good way to curate that employee’s life with you?”

Complex processes

PwC’s Kuperholz says big data comes into its own when a business has a large number of customers who are serviced via multiple channels with different costs. Throw in serious competition to win those customers away and retain their loyalty and the case to use big data grows stronger.

“Then you have complex supply chains or complex processes that can be optimised,” he says. “How does [global freight and logistics company] UPS get a jump on lower cost of delivery? Because they optimise routes.”

In many US states, drivers can turn right through a red traffic light, so UPS puts more right-hand turns in its drivers’ routes. This has saved hundreds of millions of dollars. “That’s a clever use of analytics,” Kuperholz says.

But do all companies need big data? “I hate the term, big data,” he continues. “A lot of it is marketing spin by vendors who are trying to sell hardware and software and trying to convince companies they’re in a whole new world. Oftentimes the companies are not in need of this advanced hardware or software because they’re not making use of the data they have. I don’t think there’s any organisation listed on the sharemarket that could not benefit from greater use of analytics on the existing data they have.”

Analytics begins at home

Getting more from existing data wasn’t a reason for avoiding external big data among survey respondents. Of the 58 per cent of respondents not using big data, the reasons largely fell into two categories: cost and lack of understanding about its nature and benefits.

In fact, the advice is for businesses to make absolutely sure, before embracing big data, that they have a fantastic handle on the data they already generate. “Most organisations have more than enough data to get started, but more need to know how to use it to drive commercial value,” says Sahil Merchant, head of McKinsey Digital Australia.The challenge for companies is to develop their internal capabilities with their own data. “Rather than focusing on big data, companies can start with medium data and use what they have got.”

Australia’s retailers have been forced by the explosion of online shopping to mine their data for better ways of engaging with customers. Dymocks Australia managing director Steve Cox says the bookseller routinely surveys its customers and gets a large number of responses. When it emails customers with targeted direct offers, it usually sees a spike in sales of 9 per cent, or as much as 50 per cent when it also offers loyalty points or cash.

Cox acknowledges that some customers find this annoying. “There is a line between helpful and creepy. It’s about curation. As long as it’s relevant to the individual, saving them time and you’re making it interesting, that takes the creepiness out,” he said at a recent Retail Council conference in Sydney.

Outgoing Woolworths chief executive Grant O’Brien agrees: “It’s about providing customers with an experience they can benefit from.”

Instincts put to the test

Are we moving into an age when human instinct is redundant? The majority of those using big data still see greater value in the expertise of people (70 per cent rated the team as most important) over data (30 per cent). But that mindset is increasingly insufficient.

“There is a view [in business] that it’s all about gut instinct and ‘I don’t need data to tell me,’ ” Sydney University’s Wood says.

“Then I say, ‘Let’s see how well you can predict the future. Let’s have a data-driven model and let’s pit it against your instinct. Let’s see which one wins.’ Nine times out of 10 it’s the data-driven model [that wins].

“It’s less about having opinions …” Whether a model is based on gut instincts or on what data tells you, the test of it is how well it predicts what will happen.
Kuperholz sees it slightly differently. “The great leaders are data mining,” he says. “When you have a gut feel about something you’re making a decision based on all the information you’ve taken in combined with your expertise, which is essentially the process we’re trying to replicate inside a computer.

"There’s still a role for the person who has the gut feel, the intuition, the biological computing power, but now it’s more redirected to where do I best get the evidence and what approach to extracting value from data assets is there.”
The right corporate environment is crucial, says Staples’s Sice. “The culture I work in encourages people to not necessarily feel pressured to have the right answer,” he says.

“I have no hesitation in saying I don’t have the right information and to ask the business for more information or ask outside for more information. With good data you make good observations and from good observations, hopefully, you can make good decisions.”

Doing it right

Which companies lead the way in using big data? “Google, Facebook, eBay and Amazon and to some extent Apple,” Kuperholz says.
“[Fashion retailer] Zara does this so well,” Wood says. “It observes what people are wearing. It can spot a trend, manufacture it and have it in stores within two weeks. And it is “constantly re-evaluating based on what people buy. Essentially Zara is doing real-time experiments; they’re thinking like scientists.”

What next? “The future’s very, very exciting from where I sit,” Kuperholz says, “because I think analytics and clever use of data provide the win-win scenario where we create value that didn’t exist before. So what you have is economies where everyone is better off and where employees are working fewer hours for greater overall satisfaction [while contributing more] to the organisation and the community.

“It’s a really exciting time when diseases are conquered, food shortages are reduced, and all manner of inconveniences are reduced, because of the use of existing data which then begets the collection of more data and more automation and more real-time stuff which only increases the value again. This whole thing snowballs beautifully.”