• Tweet

  • Post

  • Share

  • Save

  • Get PDF

  • Purchase Copies

  • Print

Artwork: Tamar Cohen, Happy Motoring, 2010, silk screen on vintage road map, 26″ 10 18″

"Yous tin can't manage what you lot don't measure."

In that location'due south much wisdom in that saying, which has been attributed to both W. Edwards Deming and Peter Drucker, and information technology explains why the recent explosion of digital data is so important. Just put, considering of big information, managers tin measure, and hence know, radically more nigh their businesses, and direct translate that cognition into improved decision making and performance.

Consider retailing. Booksellers in physical stores could always track which books sold and which did not. If they had a loyalty program, they could tie some of those purchases to individual customers. And that was about it. Once shopping moved online, though, the understanding of customers increased dramatically. Online retailers could track non only what customers bought, just also what else they looked at; how they navigated through the site; how much they were influenced by promotions, reviews, and page layouts; and similarities beyond individuals and groups. Before long, they developed algorithms to predict what books private customers would like to read next—algorithms that performed ameliorate every fourth dimension the customer responded to or ignored a recommendation. Traditional retailers simply couldn't admission this kind of information, let solitary deed on it in a timely manner. It's no wonder that Amazon has put and then many brick-and-mortar bookstores out of business.

The familiarity of the Amazon story about masks its power. We wait companies that were born digital to reach things that business executives could only dream of a generation agone. But in fact the use of big information has the potential to transform traditional businesses as well. Information technology may offer them fifty-fifty greater opportunities for competitive reward (online businesses take always known that they were competing on how well they understood their data). As we'll talk over in more detail, the big data of this revolution is far more powerful than the analytics that were used in the past. We can measure and therefore manage more precisely than ever before. Nosotros tin make meliorate predictions and smarter decisions. We can target more-effective interventions, and tin can do so in areas that then far have been dominated by gut and intuition rather than by information and rigor.

As the tools and philosophies of large data spread, they volition change long-standing ideas about the value of feel, the nature of expertise, and the practice of management. Smart leaders across industries volition see using big data for what it is: a management revolution. Simply as with whatever other major alter in business, the challenges of becoming a big information–enabled organization can be enormous and crave easily-on—or in some cases hands-off—leadership. Nevertheless, it's a transition that executives need to engage with today.

What's New Here?

Business executives sometimes ask the states, "Isn't 'large data' but another way of saying 'analytics'?" It's true that they're related: The big data motion, like analytics earlier it, seeks to glean intelligence from information and interpret that into business advantage. However, there are iii key differences:

Volume.

Equally of 2012, near 2.5 exabytes of information are created each twenty-four hour period, and that number is doubling every 40 months or so. More than data cross the internet every second than were stored in the entire internet just xx years ago. This gives companies an opportunity to work with many petabyes of data in a single data gear up—and not just from the internet. For instance, it is estimated that Walmart collects more than 2.5 petabytes of data every hour from its customer transactions. A petabyte is one quadrillion bytes, or the equivalent of well-nigh twenty million filing cabinets' worth of text. An exabyte is one,000 times that amount, or one billion gigabytes.

Velocity.

For many applications, the speed of data creation is even more important than the book. Real-time or well-nigh real-time information makes information technology possible for a company to exist much more agile than its competitors. For instance, our colleague Alex "Sandy" Pentland and his group at the MIT Media Lab used location data from mobile phones to infer how many people were in Macy'due south parking lots on Blackness Friday—the showtime of the Christmas shopping flavor in the United States. This fabricated it possible to estimate the retailer'south sales on that critical 24-hour interval even before Macy's itself had recorded those sales. Rapid insights like that tin provide an obvious competitive reward to Wall Street analysts and Main Street managers.

Variety.

Big data takes the form of messages, updates, and images posted to social networks; readings from sensors; GPS signals from cell phones, and more. Many of the nearly important sources of large data are relatively new. The huge amounts of data from social networks, for case, are only as erstwhile as the networks themselves; Facebook was launched in 2004, Twitter in 2006. The same holds for smartphones and the other mobile devices that now provide enormous streams of data tied to people, activities, and locations. Because these devices are ubiquitous, it'southward easy to forget that the iPhone was unveiled only 5 years ago, and the iPad in 2010. Thus the structured databases that stored most corporate information until recently are ill suited to storing and processing big data. At the same fourth dimension, the steadily declining costs of all the elements of computing—storage, memory, processing, bandwidth, then on—mean that previously expensive data-intensive approaches are rapidly becoming economical.

As more and more than business activity is digitized, new sources of information and ever-cheaper equipment combine to bring united states into a new era: ane in which large amounts of digital information exist on virtually any topic of interest to a business. Mobile phones, online shopping, social networks, electronic communication, GPS, and instrumented machinery all produce torrents of data as a past-production of their ordinary operations. Each of united states is now a walking data generator. The data bachelor are often unstructured—non organized in a database—and unwieldy, but there's a huge corporeality of point in the noise, simply waiting to be released. Analytics brought rigorous techniques to decision making; big data is at one time simpler and more than powerful. As Google'southward director of research, Peter Norvig, puts it: "Nosotros don't accept meliorate algorithms. We just have more data."

How Information-Driven Companies Perform

The second question skeptics might pose is this: "Where's the evidence that using big data intelligently will better business performance?" The business press is rife with anecdotes and example studies that supposedly demonstrate the value of being data-driven. Just the truth, we realized recently, is that nobody was tackling that question rigorously. To accost this embarrassing gap, we led a team at the MIT Center for Digital Business, working in partnership with McKinsey's business technology office and with our colleague Lorin Hitt at Wharton and the MIT doctoral student Heekyung Kim. We set out to exam the hypothesis that data-driven companies would be better performers. We conducted structured interviews with executives at 330 public North American companies almost their organizational and engineering management practices, and gathered operation data from their annual reports and independent sources.

Not anybody was embracing information-driven decision making. In fact, we found a broad spectrum of attitudes and approaches in every industry. But across all the analyses we conducted, one relationship stood out: The more companies characterized themselves as data-driven, the meliorate they performed on objective measures of financial and operational results. In particular, companies in the top 3rd of their manufacture in the use of data-driven decision making were, on average, v% more productive and 6% more profitable than their competitors. This performance difference remained robust later accounting for the contributions of labor, capital letter, purchased services, and traditional IT investment. Information technology was statistically significant and economically of import and was reflected in measurable increases in stock market valuations.

Often someone coming from outside an industry can spot a better way to use big information than an insider, but because and so many new, unexpected sources of data are available. 1 of us, Erik, demonstrated this in research he conducted with Lynn Wu, now an assistant professor at Wharton. They used publicly bachelor web search data to predict housing-toll changes in metropolitan areas beyond the United States. They had no special cognition of the housing market when they began their study, but they reasoned that virtually real-time search information would enable good virtually-term forecasts near the housing market place—and they were right. In fact, their prediction proved more authentic than the official ane from the National Clan of Realtors, which had adult a far more complex model but relied on relatively slow-irresolute historical information.

This is hardly the only instance in which elementary models and large data trump more-elaborate analytics approaches. Researchers at the Johns Hopkins Schoolhouse of Medicine, for example, constitute that they could use data from Google Flu Trends (a complimentary, publicly available aggregator of relevant search terms) to predict surges in flu-related emergency room visits a week before warnings came from the Centers for Disease Command. Similarly, Twitter updates were every bit accurate as official reports at tracking the spread of cholera in Republic of haiti after the January 2010 earthquake; they were as well two weeks earlier.

And then how are managers using large data? Permit's wait in particular at two companies that are far from Silicon Valley upstarts. 1 uses big data to create new businesses, the other to drive more sales.

Improved Airline ETAs

Minutes affair in airports. Then does accurate information near flight arrival times: If a airplane lands before the basis staff is ready for it, the passengers and crew are effectively trapped, and if it shows upwardly afterwards than expected, the staff sits idle, driving upwards costs. So when a major U.Southward. airline learned from an internal study that about x% of the flights into its major hub had at least a 10-infinitesimal gap between the estimated time of arrival and the actual arrival time—and 30% had a gap of at least five minutes—it decided to accept action.

At the fourth dimension, the airline was relying on the aviation industry'southward long-continuing exercise of using the ETAs provided by pilots. The pilots made these estimates during their final arroyo to the airport, when they had many other demands on their time and attending. In search of a better solution, the airline turned to PASSUR Aerospace, a provider of determination-back up technologies for the aviation industry. In 2001 PASSUR began offer its ain inflow estimates as a service called RightETA. It calculated these times by combining publicly available data about conditions, flight schedules, and other factors with proprietary data the visitor itself collected, including feeds from a network of passive radar stations it had installed virtually airports to assemble data about every plane in the local heaven.

PASSUR started with but a few of these installations, only by 2012 it had more than 155. Every 4.6 seconds it collects a broad range of data nigh every airplane that it "sees." This yields a huge and constant flood of digital data. What's more, the company keeps all the information it has gathered over time, so it has an immense torso of multidimensional information spanning more than a decade. This allows sophisticated analysis and pattern matching. RightETA substantially works past asking itself "What happened all the previous times a airplane approached this airport under these conditions? When did it actually land?"

After switching to RightETA, the airline near eliminated gaps between estimated and actual arrival times. PASSUR believes that enabling an airline to know when its planes are going to land and plan accordingly is worth several million dollars a year at each airport. It's a simple formula: Using large information leads to better predictions, and improve predictions yield ameliorate decisions.

Speedier, More Personalized Promotions

A couple of years ago, Sears Holdings came to the conclusion that it needed to generate greater value from the huge amounts of customer, product, and promotion information information technology collected from its Sears, Craftsman, and Lands' End brands. Manifestly, information technology would be valuable to combine and make use of all these data to tailor promotions and other offerings to customers, and to personalize the offers to have advantage of local conditions. Valuable, just difficult: Sears required nearly eight weeks to generate personalized promotions, at which betoken many of them were no longer optimal for the company. It took so long mainly because the data required for these large-scale analyses were both voluminous and highly fragmented—housed in many databases and "data warehouses" maintained by the various brands.

In search of a faster, cheaper way to do its analytic piece of work, Sears Holdings turned to the technologies and practices of big information. Every bit one of its first steps, it set up up a Hadoop cluster. This is simply a grouping of inexpensive commodity servers whose activities are coordinated by an emerging software framework called Hadoop (named after a toy elephant in the household of Doug Cutting, one of its developers).

Sears started using the cluster to shop incoming data from all its brands and to concord information from existing data warehouses. Information technology and so conducted analyses on the cluster directly, fugitive the time-consuming complexities of pulling data from various sources and combining them so that they tin can be analyzed. This change immune the company to be much faster and more precise with its promotions. According to the company's CTO, Phil Shelley, the time needed to generate a comprehensive set of promotions dropped from eight weeks to one, and is nonetheless dropping. And these promotions are of higher quality, considering they're more timely, more granular, and more than personalized. Sears's Hadoop cluster stores and processes several petabytes of data at a fraction of the price of a comparable standard information warehouse.

Shelley says he'southward surprised at how easy it has been to transition from one-time to new approaches to information management and high-performance analytics. Considering skills and knowledge related to new data technologies were so rare in 2010, when Sears started the transition, it contracted some of the work to a company called Cloudera. But over time its old guard of Information technology and analytics professionals accept go comfortable with the new tools and approaches.

The PASSUR and Sears Property examples illustrate the power of big data, which allows more-accurate predictions, better decisions, and precise interventions, and tin can enable these things at seemingly limitless scale. We've seen large data used in supply chain management to sympathise why a carmaker's defect rates in the field suddenly increased, in customer service to continually browse and intervene in the health care practices of millions of people, in planning and forecasting to amend anticipate online sales on the basis of a information set of product characteristics, and then on. We've seen similar payoffs in many other industries and functions, from finance to marketing to hotels and gaming, and from human resource management to machine repair.

Our statistical analysis tells usa that what we're seeing is non just a few flashy examples but a more primal transformation of the economy. We've become convinced that nearly no sphere of business activeness will remain untouched past this movement.

A New Culture of Decision Making

The technical challenges of using big data are very existent. But the managerial challenges are even greater—starting with the part of the senior executive team.

Muting the HiPPOs.

One of the most critical aspects of big data is its bear on on how decisions are made and who gets to make them. When data are scarce, expensive to obtain, or not available in digital form, it makes sense to let well-placed people make decisions, which they practice on the basis of experience they've congenital upwards and patterns and relationships they've observed and internalized. "Intuition" is the characterization given to this fashion of inference and decision making. People land their opinions about what the future holds—what's going to happen, how well something volition piece of work, then on—and then plan appropriately. (See "The True Measures of Success," by Michael J. Mauboussin, in this issue.)

Big data's power does not erase the demand for vision or human insight.

For particularly of import decisions, these people are typically loftier up in the system, or they're expensive outsiders brought in considering of their expertise and rails records. Many in the big data community maintain that companies ofttimes make most of their important decisions by relying on "HiPPO"—the highest-paid person's opinion.

To exist sure, a number of senior executives are genuinely information-driven and willing to override their own intuition when the information don't hold with it. But we believe that throughout the business concern world today, people rely too much on feel and intuition and not plenty on data. For our research nosotros constructed a v-signal blended scale that captured the overall extent to which a company was data-driven. Fully 32% of our respondents rated their companies at or below iii on this calibration.

New roles.

Executives interested in leading a big data transition can start with ii simple techniques. First, they tin get in the habit of asking "What do the information say?" when faced with an important conclusion and post-obit up with more-specific questions such as "Where did the data come from?," "What kinds of analyses were conducted?," and "How confident are we in the results?" (People will get the bulletin quickly if executives develop this discipline.) 2d, they can allow themselves to be overruled by the data; few things are more than powerful for changing a conclusion-making culture than seeing a senior executive concede when data have disproved a hunch.

When information technology comes to knowing which problems to tackle, of course, domain expertise remains disquisitional. Traditional domain experts—those securely familiar with an area—are the ones who know where the biggest opportunities and challenges lie. PASSUR, for ane, is trying to hire every bit many people as possible who have extensive knowledge of operations at America'southward major airports. They volition exist invaluable in helping the company figure out what offerings and markets it should go afterward next.

Equally the big data move advances, the role of domain experts will shift. They'll be valued not for their HiPPO-style answers but considering they know what questions to ask. Pablo Picasso might accept been thinking of domain experts when he said, "Computers are useless. They tin can but requite you answers."

Yous don't need to make enormous up-forepart investments in IT to use large information (dissimilar before generations of It-enabled change). Here'southward ane approach to edifice a capability from the ground up.

ane. Pick a business unit to be the testing footing. It should take a quant-friendly leader backed up past a squad of data scientists.

2. Challenge each key function to identify five business opportunities based on large data, each of which could be prototyped inside five weeks by a team of no more than than 5 people.

three. Implement a process for innovation that includes 4 steps: experimentation, measurement, sharing, and replication.

4. Keep in heed Joy'south Law: "Nearly of the smartest people piece of work for someone else." Open up some of your data sets and analytic challenges to interested parties beyond the internet and around the world.

5 Direction Challenges

Companies won't reap the full benefits of a transition to using big data unless they're able to manage change effectively. Five areas are peculiarly of import in that process.

Leadership.

Companies succeed in the big data era not simply because they take more or better data, but because they take leadership teams that set clear goals, define what success looks like, and ask the right questions. Large data's power does not erase the need for vision or human insight. On the contrary, nosotros still must have business organisation leaders who can spot a great opportunity, empathise how a marketplace is developing, think creatively and propose truly novel offerings, articulate a compelling vision, persuade people to embrace information technology and piece of work difficult to realize information technology, and deal finer with customers, employees, stockholders, and other stakeholders. The successful companies of the next decade will be the ones whose leaders can do all that while changing the way their organizations make many decisions.

Talent management.

As information become cheaper, the complements to information become more than valuable. Some of the most crucial of these are information scientists and other professionals skilled at working with large quantities of information. Statistics are important, but many of the key techniques for using big data are rarely taught in traditional statistics courses. Maybe even more important are skills in cleaning and organizing large data sets; the new kinds of data rarely come in structured formats. Visualization tools and techniques are as well increasing in value. Along with the information scientists, a new generation of calculator scientists are bringing to comport techniques for working with very big data sets. Expertise in the design of experiments can help cross the gap between correlation and causation. The all-time data scientists are too comfortable speaking the language of business and helping leaders reformulate their challenges in ways that big data can tackle. Non surprisingly, people with these skills are hard to find and in bully demand. (See "Data Scientist: The Sexiest Job of the 21st Century," by Thomas H. Davenport and D.J. Patil, in this outcome.)

Technology.

The tools available to handle the volume, velocity, and variety of big information have improved greatly in recent years. In general, these technologies are not prohibitively expensive, and much of the software is open source. Hadoop, the most unremarkably used framework, combines commodity hardware with open-source software. It takes incoming streams of data and distributes them onto inexpensive disks; information technology besides provides tools for analyzing the information. However, these technologies do crave a skill fix that is new to most It departments, which will need to work hard to integrate all the relevant internal and external sources of data. Although attending to technology isn't sufficient, it is always a necessary component of a big information strategy.

Conclusion making.

An effective organisation puts information and the relevant determination rights in the same location. In the big data era, data is created and transferred, and expertise is oft not where it used to be. The artful leader volition create an organization flexible enough to minimize the "non invented hither" syndrome and maximize cross-functional cooperation. People who understand the bug need to exist brought together with the right data, merely also with the people who have problem-solving techniques that tin finer exploit them.

Company culture.

The outset question a data-driven organization asks itself is not "What do we think?" but "What do we know?" This requires a move away from acting solely on hunches and instinct. It also requires breaking a bad addiction we've noticed in many organizations: pretending to be more than data-driven than they actually are. Too often, we saw executives who spiced upwardly their reports with lots of information that supported decisions they had already made using the traditional HiPPO approach. Simply afterwards were underlings dispatched to find the numbers that would justify the decision.Without question, many barriers to success remain. There are too few data scientists to go around. The technologies are new and in some cases exotic. It'due south too like shooting fish in a barrel to mistake correlation for causation and to find misleading patterns in the data. The cultural challenges are enormous, and, of course, privacy concerns are just going to become more than significant. But the underlying trends, both in the engineering science and in the business concern payoff, are unmistakable.

The bear witness is clear: Data-driven decisions tend to be ameliorate decisions. Leaders will either comprehend this fact or exist replaced by others who do. In sector subsequently sector, companies that figure out how to combine domain expertise with data science will pull away from their rivals. Nosotros tin can't say that all the winners will be harnessing big information to transform conclusion making. But the data tell us that's the surest bet.

A version of this article appeared in the October 2012 issue of Harvard Business organization Review.