No-one is good at judging spin – not even professionals

Following months of campaigning ahead of the US election, political scientists are suggesting that professional campaigners may not be getting the results they expect.

A study published in PNAS showsAmerican political staffers and professionals are no better than laypeople at spotting the efficacy of a political message.

The researchers also found that neither group could perform much better than random chance when judging the communications value of different messages.

“We found that neither political practitioners nor the mass public are particularly accurate in predicting which persuasive messages are more effective than others,” says co-author Associate Professor Joshua Kalla, a political scientist at Yale University, USA.

The researchers gathered 172 different messages on 21 topics in US politics, each a couple of sentences in length.

For instance, two messages used on the topic of legalising marijuana were:

“Today, marijuana trafficking is linked to a variety of crimes, from assault and murder to money laundering and smuggling. Legalization of marijuana would increase demand for the drug and almost certainly exacerbate drug-related crime, as well as cause a myriad of unintended but predictable consequences.” – from the Heritage Foundation

“Hundreds of thousands of Americans are arrested each year for marijuana-related offenses, the vast majority of which are for simple possession. By legalizing marijuana, government resources could be better spent on things like testing untested rape kits or investing in human needs such as mental health counseling, substance abuse treatment, and activities for at-risk teens.” – from the Marijuana Policy Project

The team tested these messages on 23,167 US study participants with an online survey. Participants were either not shown a persuasive message, or they were shown one.

Then, the researchers measured participants’ opinions on the political issue, and used this to judge the message’s effectiveness.

They then ran a second survey on 21,247 US laypeople, and 1,524 US political practitioners.

Most practitioners (83%) had worked in politics for more than 6 years, with 65% working in politics for more than a decade, and 91% of respondents said they had been involved in developing some form of political messaging. The sample was skewed politically, with 72% of practitioners identifying as Democrats, 11% as Republicans, and 17% as independent or other.

Survey respondents were asked to judge the efficacy of some of the 172 messages.

Lay people judged members of the public as more persuadable than they actually were, while practitioners did not. But other than that, both groups performed very similarly in terms of predicting how well each message might perform.

“This suggests that political practitioners who craft language intended to persuade have fairly poor intuitions about which messages people will find persuasive,” says Kalla.

But he adds that practitioners do have tools other than intuition for judging political messages.

“They could use survey experiments similar to what we did in this study,” says Kalla.

“We see political campaigns already doing that, and I suspect more will adopt such techniques moving forward.”