When a group of people with differing levels of expertise disagree, it's not easy to work out who is right. Several studies suggest the answer is to look for the unexpectedly common answer, something that might prove useful in much more important contexts than your next trivia night.
Democracy works for value judgments, where my opinion is as good as yours, not so much for factual questions where you may have spent years learning up on a topic that someone else first heard about last Tuesday.
As Professor Drazen Prelec of Massachusetts Institute of Technology put it in Nature: “Democratic methods have serious limitations” when trying to establish objective truth. “They're biased for shallow, lowest common denominator information, at the expense of novel or specialized knowledge that is not widely shared.”
Alternatives have been proposed, such as asking people how confident they are in their answers. Prelec tested out a number of such methods using questions of the American state capitals, the value of art, and the chance a skin growth is cancerous.
When asked to name the capital of an American state, a majority of a randomly selected group will usually back the largest city – after all, at least they have heard of it. There will usually be a subgroup, however, who know the answer, whether because it's their home state, they are geography nerds who like to memorize these things, or they had an insistent school teacher.
Prelec found that not only does democracy fail in such cases, but weighting by people's confidence doesn't help a lot either.
What worked in Prelec's study was to ask people not just what answer they thought was right, but what they expected others would answer. If an answer was anticipated to get 5 percent support, but actually got 20 percent, it usually turned out to be right, despite being the minority position. Prelec calls this the “surprisingly popular” algorithm.
Prelec gave a group of Princeton students a list of cities and asked if each was their state's capital. They were also asked to predict the whole group's most popular answer.
When asked if Philadelphia is the capital of Pennsylvania, most participants said yes, and predicted others would do the same. The few who knew it's not were generally aware of their fellow participants' ignorance and also predicted most others would say yes. The fact that “no” exceeded expectations made it the winner in Prelec's algorithm.
Using 50 state capital questions and 80 other general knowledge questions, Prelec found the surprisingly popular method reduced error by 21-35 percent.
The same method proved effective when professional dermatologists were shown skin lesions and asked if they were cancerous, suggesting the work has applications when getting the answer right matters.
