Observing Consumers Beats Listening to Them

By Tom Ryan

Nate Bolt, co-founder and chief executive of user research firm Bolt Peters, argues that while tech entrepreneurs are encouraged to listen to potential customers, observing is a much better tool to see whether a new product will succeed in the market.

"The main problem with opinions is self-reporting bias," wrote Mr. Bolt in a column in the venture-capital blog, VentureBeat. "Opinions are often inconsistent with behaviors or other attitudes, especially when discussing hypotheticals."

As an example, he noted that Clippy, Microsoft's animated paperclip helper, came about after the company's researchers found that when asked, users roundly agreed that they wanted help working with their documents.

"But once people started actually using it in the real world, they hated it -- it might be one of the most hated features in the history of computing," said Mr. Bolt.

Mr. Bolt offered three ways tech entrepreneurs can gauge whether people would use a service:

1. Test ideas early by watching behavior: He suggests having eight people interact with a prototype or even wireframes or design makeups. In the tech world, a number of websites (Chalkmark, Pidoco, Balsamiq, etc.) allow companies to easily test prototypes. Wrote Mr. Bolt, "You can still ask all your needy questions about what they think after the session -- just don't take those too seriously."
2. Get all stakeholders to watch the research: Technical and business constraints obscure the basic question of whether the interface is any good.
3. Use unorthodox methods: Mr. Bolt noted that Apple claims it doesn't conduct user research but releasing products in generations provides the company with loads of reviews, task-specific complaints, crash reports, customer support issues, and Genius Bar feedback. "It's audacious, large-scale behavioral research," Mr. Bolt wrote. Similarly, long-beta testing periods have helped launch services such as Gmail.

Mr. Bolt concluded, "Do whatever you need to do to understand how people use your product. If it's a device meant to be used in cars, watch people use it cars; if it's a video game, avoid sterile lab environments. Just don't ask perfunctory, cookie-cutter survey questions to your potential customers, and expect that to ensure your product's usefulness."

Discussion Questions: What are the merits of observation versus listening in testing new products? What are some ways researchers can incorporate observational research alongside listening techniques?

• Stop listening to your customers - VentureBeat.com