In labor productivity studies, they call it the Hawthorne Effect. In more general terms, it's called the Observer Effect. In both cases, the conclusion is that the mere act of observing something affects the outcome. The Hawthorne Effect has been an important consideration in quantifying the impact of process change for over 85 years. Now The Economist reports that a new analysis of the original data casts doubt on the initial conclusions.
The Hawthorne Effect was derived from observations of workers conducted at a large telephone industrial plant outside Chicago in 1924 known as the Hawthorne Plant. The original goal was to measure the impact of improved lighting, but instead reached the conclusion that just knowing they were being observed affected the way workers performed.
There was never any detailed econometric study of the original data from the Hawthorne Plant. However, because it was so intuitively reasonable, the conclusion was widely accepted, quoted and factored into research methodology. Now two economists, Steven Levitt and John List, at the University of Chicago have discovered the original data and conducted the analysis. It turns out that electricians changed the lighting on Sundays while the plant was closed, so workers arrived to an improved lighting environment on Monday mornings. But looking at other data for periods when the lighting did not change also revealed improved productivity on Mondays. It led the economists to another obvious conclusion: workers are more productive at the beginning of the week when they are fresh. Other factors also impacted the study, so the effect of being observed could not be isolated.
Retailers are constantly trying to extract information from the raw data they receive on labor productivity and customer purchases. This finding reveals how difficult it can be to distinguish between correlation and causation. Few would argue that the conclusions we reach from raw data are often influenced by our own experiences and in some cases this leads to insightful observations. However, in other cases, it almost undoubtedly leads to missteps and the wrong the conclusions.
Discussion Questions: Are workers affected by being observed as postulated from the original Hawthorne project? How should personal experience be factored into conclusions drawn from data analysis? How can we do this without projecting our expectations on the results?
[Author's Commentary] On the first question, I am reminded of a store labor study we did that observed what everyone in the store was doing every 15 minutes. Everyone was asked to work as usual. At one point, when the observers could not find the night crew at around 3:00 AM, they followed the aroma of barbecue steak to the basement where the subjects were happily preparing their lunch while enjoying some local marijuana. (The store was in New Orleans.) Obviously, the observers had allayed their concerns over being observed.
As far as the other questions, I am reminded of an advertising professor in graduate school who pointed out the difficulty of putting yourself in the shoes of your ad audience. For example, he said that less than half the population had read a book over the previous year, a seemingly impossible fact to a classroom of students. It is often easy for us to see patterns in data that support our preconceived notions of the world. But this is also the "art" that a long time category manager brings to their job as they interpret the data. It often takes a combination of raw data and experience to really develop insight.
Whatever the statistical validity of the Hawthorne study, it is still intuitively pleasing.