Which of these text messages do you think would be more likely to get a conservative voter out to vote?
- Tomorrow is Election Day for Governor! Your Voting Record is Public. Be a good citizen, be a Voter!
- Will we let them beat us? Friendly reminder to Vote for Gov tmrw.
Adam Schaeffer, the Director of Research and the co-founder of Evolving Strategies, posed that question to CPAC attendees at a panel titled, “Vaccines vs. Leeches: Using Experiments to Win Hearts, Minds, and Elections.”
He and his team have sent out both these messages to randomized subsets of voters, and it turned out that the first message had a statistically insignificant effect on voting, but the second turned out to raise turnout by 6.8 percentage points.
Schaeffer and others (including the growing team at Para Bellum) are trying to use experiments to guide outreach, testing tiny variations in messaging to find big, unexpected advantages. The Democrats, relying on the research of Alan Gerber and Donald Green, have been using experiments to increase turnout and maximize fundraising, and have outpaced the GOP’s efforts.
Experiments have the power to subvert the conventional wisdom of campaigns, since it’s easy to try out a new idea cheaply. In 2012, the Obama team found that they could maximize the chance that one of their emails would be opened with a simple, enigmatic subject line: “Hey.” Small changes can make a big difference.
In Schaeffer’s experiments, timing was critical. Although the second message produced good results in the morning, when the same message was sent in the afternoon, the results were still significant, but they were significantly negative. Voters who were contacted in the late afternoon had their turnout rates drop by -11.4 percentage points as a result.
Testing so many hypothesis and checking the impact of messages on so many tiny subgroups leaves candidates vulnerable to being mislead by statistical artifacts. Most commonly used significance tests have a one in twenty chance of being false positives. When a campaign tests hundreds of variations, some results are bound to seem significant in pilot tests but fail to preform when they’re applied to the whole electorate.
As a corrective, Schaeffer sometimes raises his standards for significance and checks how any result fits into his overall understanding of how the messages are received (e.g. all text messages became less effective in the afternoon). Campaign operatives can also look for guidance from the academic literature, but, Schaeffer cautions, because psychology experiments are usually run on people who are WEIRD (or Western, Educated, Industrialized, Rich, and Democratic), and may not generalize well to all voters.
Researchers like Schaeffer are expanding our understanding of the psychology of persuasion, but the published results so far are enough to be a little unsettling about their inclusion in politics. Robert Cialdini, the author of Influence, compiled studies that showed that the phrasing of a request can have an enormous effect on compliance rates.
In an experiment by Ellen Langer, when people tried to cut the line in front of a copy machine by saying, “Excuse me, I have five pages. May I use the Xerox machine?” the people in the line yielded 60% of the time. But, when the researchers said, “Excuse me, I have five pages. May I use the Xerox machine because I have to make some copies?” compliance shot up to 93%.
Everyone in the line was trying to make copies, so the “because” clause didn’t convey any new information and shouldn’t have had such a large impact. Subtle persuasive tactics of this type have the potential to decouple a voter’s compliance from their interests. Politicians can eke out major gains by using Cialdini’s techniques of reciprocity, social proof, authority and others, instead of winning voters’ loyalty by serving their interests.
When businesses use these clever, coercive messages, the results are less worrying. A hotel chain may use social feelings to increase the chance that visitors will reuse their towels (cards that cited the percent of other guests who reused towels prompted better compliance than those that just used environmental facts). But the stakes are lower than when a firm like Evolving Research puts images of eyes in their political email, to amp up pro-social feelings that may lead a voter to volunteer time or money.
Researchers and campaign staff can run useful experiments because they benefit from tight feedback loops, where the results of their work are quickly apparent. The long term consequence of their work, however, may be to weaken the feedback voters can provide their representatives, when their votes are bought by the phrasing of appeals, rather than their content.