Knowledge@Wharton staff
Presented here for discussion is a synopsis of a current article published with permission from Knowledge@Wharton, the online research and business analysis journal of the Wharton School of the University of Pennsylvania.
In his book, “Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are,” Seth Stephens-Davidowitz explores in part how Big Data can help brands understand consumers better than consumers understand themselves.
“What people click on, what people purchase, what people search — that’s more valuable than many of the other sources that you might consider,” the former Google data scientist said on the Knowledge@Wharton Show on Sirius XM.
The problem with survey-driven data gleaned from surveys is that it is unreliable. He stated, “You can’t trust what people tell you. With a lot of the traditional data sources, there are incentives for people giving you that data.”
Netflix, for example, initially asked subscribers which videos they wanted to watch in the weekend ahead and documentaries or art films often came back as responses. However, when the weekend comes, subscribers ignore the suggestions and go for the lowbrow comedies or romances they usually watch.
“Netflix just realized they should ignore what people tell them, and instead focus on what they actually do, and let the algorithm tell the story,” stated Mr. Stephens-Davidowitz. “We tend to make horrible predictions about what we’re going to do in the future. Almost all of us are way too over-optimistic. I think data can ground us much better.”
Mr. Stephens-Davidowitz believes more academic research needs to be done around understanding people from their internet behavior and around Big Data’s ultimate ramifications.
“The pessimistic scenario is that companies would use this to take advantage of people, to get them to spend more money that they don’t have, or spend more time on their websites even though they don’t need to be on those websites. The optimistic scenario is that we would have insights into really, really important areas — health, racism, sexuality — and really learn how to improve society.”
But he believes surveys will lose influence as a data source. He stated, “Surveys have been dramatically overvalued, and really are going to play a much smaller role in the future as some of these new internet data sources become more accessible.”