The "Big Data" Revolution

During the Industrial Revolution, a key challenge was mining and transporting enough of the abundant coal and iron ore deposits to feed the growing demand. Today, in our information-based economy, data is the new vital raw material and there is certainly no shortage of that raw material.

Every day, every second, huge volumes of data are being created as a by-product of business and individual activity. Companies are tapping seemingly endless streams of data about their customers, suppliers, and operations, while sensors — which increasingly are embedded in devices such as mobile phones, smart energy meters, automobiles, and industrial machines — are also recording and communicating data. Contributing further to this deluge of data is the use of social media sites, smart phones, and other consumer devices (including PCs and laptops), by billions of people around the world.

Until recently, available hardware and software technologies have limited the commercial impact of performing “deep analysis” on big data. However, several technology leaders, including Oracle and IBM, have started to close that gap.

The result of all this activity is known as "big data." A recently released report from the McKinsey Global Institute titled "Big Data: The Next Frontier for Innovation, Competition, and Productivity,"¹ defines big data as "data-sets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze."

McKinsey estimates that, in 2010 alone, "enterprises worldwide stored 7 million terabytes of new data on disk drives" and "consumers stored more than 6 million terabytes on devices including PCs and notebooks." U.S. and EU organizations had a combined online storage capacity at the end of 2010 of roughly 27 million terabytes The volume of new data is expected to continue growing by 40 percent every year, while IT budgets are expected to grow at a 5 percent annual rate.

Over 20 years ago, the Trends editors saw this opportunity on the distant horizon. Working with clients in several industries, they identifie... To read the full article, you must be a Trends Magazine Subscriber. To learn more, click here