The effectiveness of political actors in influencing public opinion by specific persuasive appeals is poorly understood. From laboratory experiments we have learned that persuasive stimuli (i.e. having participants watch a documentary or read a newspaper story) does have effects on attitudes. These experiments also reveal insights into to the psychological mechanisms of attitude change. We know less about the persistence of persuasive treatments over time (which is what matters in actual campaigns) and even less is known about the ecological validity of these experiments. What sounds like a technicality is crucial and addresses the core of persuasion research in political communication: It is unresolved to what degree and under which circumstances findings from artificial laboratory setting can be generalized to the real world, particularly the communication of actual political actors. Since research ultimately aims at understanding the world outside the lab, this is a crucial gap in our knowledge. Field-experiments are the natural candidate to investigate how persuasion works in applied settings. In recent years some scholars have started to conduct this kind of research with increasing intensity (among others, David Broockman has published several interesting studies on this topic)
Financed by Konrad-Adenauer-Stiftung and in collaboration with the German employers´ interest group Gesamtmetall we have conducted a persuasion experiment and present first and preliminary results. Andreas Jungherr is the principal investigator of this project and Matthias Mader and Harald Schoen also work on this. In the following, I will briefly present some slides from a presentation Andreas will give at the International Conference for Political Communication #IKPK16 in Berlin.
In this study, subjects received a mailing sent on behalf of ‚Gesamtmetall‘ on the issue of a free trade agreement (TTIP). Subjects (N=8.000) were randomly exposed to either a postal mailing under field conditions or to an identical facsimile as part of an online survey or no to treatment at all.
Attitudes on the issue in questions -TTIP- vary considerably. More people (42,9%) oppose the proposed trade than than people who support it (32,1%). 25 percent of respondents chose the scale mid-points of our three-item-construct on TTIP-attitudes.
Preliminary analyses of the treatment effect shows substantial persuasion among both experimental groups with larger effect sizes in the survey-experiment. In post-treatment wave 2 48% of respondents opposed TTIP and 27% supported TTIP. Those who have received a postal letter from the interest group which argued for the benefits of TTIP have considerably more favorable attitudes towards this trade deal. The share of supporter grew to 32% and the share of opposed respondents shrinked to 44%. (Yes, the effect is statistically significant at p<0.001). As expected, the effect is even stronger when the panelists have not received a physical letter, but have read the letter as part of our online survey. 45 percent (!) supported TTIP with only 32.5% opposing TTIP in this condition. So, having read the arguments by the German association ‚Gesamtmetall‘ basically reverts the opinion among respondents in the survey-experimental-condition from a solid plurality against TTIP to a solid plurality of ttip-supporters.
The lower (but still substantially important) effect size in the field-experimental condition compared to the survey-experimental condition has a variety of (very interesting) causes. Among them are non-compliance (simply not opening or reading the letter) and the bigger distance in time between treatment administration and measurement in the field-experimental mode. However, importantly, while the effects of the treatment naturally decline over time, they are persistent one week after the administration of the treatment.
The persuasion effects are highly statistically significant in the field- and in the survey-experimental-condition (and no, the result is no artifact of our trichotomization of the DV). Proper statistical analyses were already conducted. However, the relieving point is that even without fancy regression analyses our results are robust and effects can be detected with simple instruments of comparisons of means that are comprehensible without statistical education.
This posting gave a glimpse into the persuasion effect of our experiment. More generally, this is an exciting project because it allows tapping into a variety of different kinds of unanswered questions about how attitudes develop and change. We have conducted extensive analyses on the predictors explaining attitudes toward TTIP and free trade which will be reported some time later. The research design of parallel experiments is very interesting from a methodological perspective and hopefully leads to a better understanding of differences and commonalities of field- and survey-experiments. One example for this is the chance to investigate how selective exposure and motivated reasoning (established phenomena in lab experiments) will behave in the field and how this will influence the actual treatment effects in the field compared to the effect in the survey-experiment. The wealth of data we have will also allow us to tap into the psychological mechanisms behind the persuasion we have observed. Lastly, we had included another small experimental variation concerning the content of the latter (one-sided vs. two-sided).
Andreas will talk about some of these topics this Monday at #IKPK16 in greater detail. I just want to give a peek into one phenomenon I find particularly interesting. Pundits and scholars alike often argue that the findings on persuasion would not replicate outside the lab because citizens can often choose what message to consume and people would be motivated to avoid political messages that do not match their pre-existing beliefs. Therefore, persuasive appeals would mostly preach to the converted.
Our data does not support this notion of selective exposure. The pre-treatment attitude toward TTIP or union-membership is not or only weakly correlated with whether an individual has decided to read the letter. This is the case, although we had printed on the envelope „PRO TTIP!“ in the biggest letters possible and it also states that it was sent by a pro-business interest group. We have also conducted some analyses showing not much evidence for biased processing (motivated reasoning) either. So, there are interesting weeks ahead for those involved in project, in which we can learn more about how persuasion works in an applied setting.
Outstanding findings! Keep me posted, please!