Big Data Commentary

For John, BLUF:  Big Data is a big deal.  Nothing to see here; just move along.

Big Data is one of the hot issues of the day.  Foreign Affairs, the prestige high brow magazine, has an article titled "The Rise of Big Data:  How It's Changing the Way We Think About the World" in its May/June 2013 issue.  The authors are Mr Kenneth Neil Cukier and Mr Viktor Mayer-Schoenberger.  The article can be found here.  Here is the lede and following paragraph.

Everyone knows that the Internet has changed how businesses operate, governments function, and people live.  But a new, less visible technological trend is just as transformative:  “big data.”  Big data starts with the fact that there is a lot more information floating around these days than ever before, and it is being put to extraordinary new uses.  Big data is distinct from the Internet, although the Web makes it much easier to collect and share data.  Big data is about more than just communication:  the idea is that we can learn from a large body of information things that we could not comprehend when we used only smaller amounts.

In the third century BC, the Library of Alexandria was believed to house the sum of human knowledge.  Today, there is enough information in the world to give every person alive 320 times as much of it as historians think was stored in Alexandria’s entire collection an estimated 1,200 exabytes’ worth.  If all this information were placed on CDs and they were stacked up, the CDs would form five separate piles that would all reach to the moon.

Yes, knowledge is exploding, but as the last three paragraphs show, we have not reached the point of the Singularity, made popular in books by Author Ray Kurzweil.

Ultimately, big data marks the moment when the “information society” finally fulfills the promise implied by its name.  The data take center stage.  All those digital bits that have been gathered can now be harnessed in novel ways to serve new purposes and unlock new forms of value.  But this requires a new way of thinking and will challenge institutions and identities.  In a world where data shape decisions more and more, what purpose will remain for people, or for intuition, or for going against the facts?  If everyone appeals to the data and harnesses big-data tools, perhaps what will become the central point of differentiation is unpredictability:  the human element of instinct, risk taking, accidents, and even error. If so, then there will be a special need to carve out a place for the human: to reserve space for intuition, common sense, and serendipity to ensure that they are not crowded out by data and machine-made answers.

This has important implications for the notion of progress in society.  Big data enables us to experiment faster and explore more leads.  These advantages should produce more innovation.  But at times, the spark of invention becomes what the data do not say.  That is something that no amount of data can ever confirm or corroborate, since it has yet to exist.  If Henry Ford had queried big-data algorithms to discover what his customers wanted, they would have come back with “a faster horse,” to recast his famous line.  In a world of big data, it is the most human traits that will need to be fostered—creativity, intuition, and intellectual ambition—since human ingenuity is the source of progress.

Big data is a resource and a tool.  It is meant to inform, rather than explain; it points toward understanding, but it can still lead to misunderstanding, depending on how well it is wielded.  And however dazzling the power of big data appears, its seductive glimmer must never blind us to its inherent imperfections.  Rather, we must adopt this technology with an appreciation not just of its power but also of its limitations.

Things to think about.

Regards  —  Cliff