8 reasons you'll do big data this year - Andrew Oliver

Over the past 12 months, I’ve been digging in the data trenches. OK, mostly I’ve been sitting next to the smarter people digging through the trenches and oversimplifying what they were doing in reports to management.

Very few IT projects are truly unique -- and the ones that sound unique often fall into relatively predictable buckets. Lucky for you, I've decided to come up for air and share the top eight types of projects I’ve seen over the past 12 months.

1. Exploring the life of a deal

Companies that do e-commerce take for granted that you can hook up a few tools and know the close rate of users coming to the website, from sales to payment. But many companies deal with a lot more data sets than Web-to-close. Mainly those data sets originate from distributors and resellers.

Each distributor or reseller presents a different data set in a different format. Sure, fundamentally, this is a core ETL/data consolidation project with BI/visualization on the front end. But for many companies, truly understanding the life of the deal (from inception to close and ongoing) is more difficult than you think. You need to combine a lot of CRM, Web analytics, and finance to say, “Yes, PPC yielded closings, but 40 percent of those customers defaulted on the first bill, so ...” 

2. Big Brother is watching and would like to offer you a latte

Companies like to know what you're doing (credit for this headline goes to a co-workerwho shall remain anonymous) and sell you an item based on that activity. Your telemetry data, for example, might indicate that you've installed an app and the company knows where you are in the mall. A huge amount of big data horsepower goes into predicting what you might want to buy at any given moment.

3. Measuring the effectiveness of marketing

Marketing people do “things,” and they want to know what those things were and how they relate to their KPIs. This is again fundamentally a BI project and often involves a fair amount of change data capture (CDC) and ETL or data consolidation. The actual KPIs they measure varies a lot. Sometimes these involve cubes in tools like Kylin or Greenplum. Other times these cross over into the next category: social media.

4. Taking the temperature of social media

People are talking ... a lot. Sometimes they talk about you (or, rather, your company), often in a public or semipublic social network. A lot can be learned from this, such as: How do people feel about your brand? Are your marketing efforts working? If the USGS can detect earthquakes and their magnitude from Twitter, then certainly you can learn whether your new ad campaign caused a stir or fell flat. For some industry verticals, this data capture goes well beyond Twitter and Facebook, thanks to specialized social networks.

5. Slogging through logs

Whether you're intent on detecting intruders or you think the auditors are coming, you need to capture logs and make them searchable. Splunk has made a killing here, but there are other, more flexible options in big data.

6. Because I don’t want to buy Teradata

This is not Teradata’s year. Big data has eaten at the edges (including the growth frontier), and now Apache Kylin has made cubes available to everyone. MPP as a distributed system is becoming more significant, thanks to Impala and HAWQ/Greenplum. There's less room for a big expensive item that only does that thing and doesn’t fit in with other data analytics efforts -- and can’t go to the cloud unless it's a single vendor’s weird, proprietary cloud.

7. Plain old extract, tranform, load

ETL is still probably the most common Hadoop workload today -- and I’d venture to say that ETL is the most common nonstreaming workload for Spark. By the way, 100 startups have come out of the woodwork claiming to handle this task.

8. Capture sensor data now, figure out what to do with it later

Whether it's the power grid, manufacturing, water pumps, or people driving around, stuff is telling us things. Those things need to be captured. Some people have even figured out what do with the data, but capturing it in time is the first big step, because many people feel it's technically hard.

Note that I usually push people to start thinking about analytics early in their big data projects. Why? Because sizing and designing a flow is far easier up front than rethinking the whole arrangement when the data is already in a box. But sometimes you need to ingest and hope for the best.

Over the past year, I’ve seen a few other project types, but most use cases fit in one of these eight categories. What do you folks see in the wild?