A Long View on How Big Data and AI Have Transformed Business Culture - Randy Bean

As my colleagues and I attend 3 major data and technology industry events in New York City this coming week – ML Ops, Finovate, and the granddaddy of them all, Strata Data Conference – it is interesting to note how far we have come in a few short decades.

It may be hard to imagine today, but there was a time not too very long ago when data analysts, with a few notable exceptions, were relegated to the hidden recesses of most corporations. Better to toil away in the bowels than to be shown the light of day. For many decades, even as information technology (IT) emerged as a critical business function, data was viewed more as something that firms filed away in vaults for the mandatory seven years to comply with regulators, and not a business asset that could be mined to unlock critical business insights. Data was perceived as the purview of those who were sometimes derisively referred to as data geeks or “propeller heads”. This was long before Silicon Valley, or Wall Street, embraced the term “geek”. As one long-time analytics executive puts it, “we were the math club”.

Times change though. Not too long ago, data analysts were rechristened as data engineers, data scientists, and data architects, reflecting a professionalization of these roles, with new connotations – engineer, architect, scientist – that reflected technical competency, achievement, distinction, and respect. In late 2012, for crowning effect, Professor Tom Davenport proclaimed in the esteemed Harvard Business Review that data scientist was the sexiest job of the 21st century. This change in perception, with the accompanying elevation of the data professional, has occurred within just over a single decade. Today, as data has become a lifeblood of industry leading firms, competing on analytics has become an unquestioned mantra, and firms have anointed Chief Data Officers to occupy the C-Suite.

While these are the obvious manifestations of the recognition and acknowledgement of data as a business asset that is critical to measurement, insight, and differentiation, it is interesting to note the way in which data has gained acceptance at the Board and within the C-Suite. Sometimes considered the “money ball” effect, data as gained acceptance as its complexity has been made understandable and digestible in fundamental business terms. How can we go to market faster? How we can increase revenues or reduce costs? How can we win more games? Once data experts learned to translate and communicate complex technical or scientific jargon in terms of basic outcomes – win, lose, fast, outsell, grow – the business value and benefit of data gained traction and appreciation.


Much credit, in my opinion, goes to the popularizers who first invented the term “Big Data” around 2011, notably the folks at McKinsey in their May 2011 report, Big data: The next frontier for innovation, competition, and productivity. For those old-timers who appreciate the analogy, this was akin to then-music-critic Jon Landau’s declaration, having just seen Bruce Springsteeen play in a Boston club in 1974, “I have seen the future of rock and roll”. The term ‘Big Data’ served to popularize data in a way that decades of data work had never previously achieved – by making it simple, accessible, and Big! It led to my own 2-year stint as a Wall Street Journal guest columnist writing “light and lively” monthly columns on Big Data and its potential. Companies at the highest level – the Board and C-Suite – were now being compelled to pay attention. It was the moment that corporations across the board embraced data as a critical business asset, and recognized that to compete in the 21st century they must have a data strategy and data leadership.

Similarly, organizations are now embracing Artificial Intelligence (AI) with the same zeal. As those in the field know all too well, AI has been around for many decades in various formations – natural language, machine learning, deep learning, robotic process automation. Just a year ago, an esteemed colleague who received his PhD in AI in the early 1980’s remarked to me, “AI will never take off”. What has changed is the coalescing of these various forms of computer assisted activities, along with massive and inexpensive computing power, under a new and unifying vision of AI as a capability that lies at the core of business transformation for the coming decades.

It does not matter that the terms Big Data and AI may be used or misused with great technical imprecision. What matters is that Big Data and AI have managed to capture the imaginations and attention of senior business decision makers at the Board and C-Suite levels, and as a result, organizations have made significant commitments to elevating these activities and giving them business primacy – through Centers of Excellence, Big Data and AI Labs, and Moonshot Initiatives.

Big Data and AI are now widely, and near-universally, accepted in business culture, as evidenced by the most recent results of the C-Executive survey that my firm, NewVantage Partners conducts each year. Note some of the 2019 findings:

Figure 1. Investment in Big Data and AI

Figure 1

 NEWVANTAGE PARTNERS

Figure 2. Urgency to Invest in Big Data and AI

Figure 2

 NEWVANTAGE PARTNERS

Figure 3. Level of Investment in Big Data and AI

Figure 3

 NEWVANTAGE PARTNERS

Today, nearly all leading companies state in their annual reports and somewhere in their business mission statements that data is a critical business asset, that they are striving to become data-driven in their analysis and thinking, that they are deeply engaged in forging a data culture at all levels of their organization, and that they view data as a basis for innovation and competition in the global marketplace. We have come a very long way in a few short decades. Times change.

(Forbes)