When the Whole Is Greater than the Sum of Its Parts

In a new study, now published with the American Political Science Review, we show how established operationalization strategies of multi-dimensional concepts can systematically lead to wrong conclusions. Focusing on populist attitudes, we demonstrate simple methods to align theory and measurement.

Our argument refers to a specific but common type of multi-dimensional concepts which are sometimes called ‘non-compensatory’. Multi-dimensional concepts are non-compensatory when higher values on one component cannot offset lower values on another.

Think of democracy: If we believe that a country only counts as a democracy if it provides both rule of law and free elections then no valid measure of democracy will assign high democracy scores when ‚rule of law‘ scores are low even when the elections are extraordinarily fair.

However, not all social science studies consider this seemingly minor but consequential issue when putting multi-dimensional concepts into empirical practice. Focusing on populist attitudes, we show that these measurement-concept inconsistencies can lead to wrong conclusions.

Populism is an essentially contested concept. Yet, most scholars now agree that populist attitudes are multi-dimensional (e.g.: anti-elitism + Manicheanism + Sovereignty). Importantly, there is also widespread agreement on the idea that populist attitudes lie at the intersection of the concept’s sub-dimensions. Hence, the unique property of populist attitudes is the co-existence of its components. Put differently, we only consider citizens (or leaders) as populists if they accept anti-elitist views AND a Manichean outlook AND support popular sovereignty. Importantly, it is this non-compensatory concept property that distinguishes populist attitudes from other established public opinion constructs (eg cynicism, efficacy, ethnocentrism), that makes populist attitudes worthwhile as a concept and that makes it more than the sum of its parts. Yet, existing studies on populism at the mass level rarely transfer this crucial concept feature into empirical practice. Hence, some populism studies do not measure what they intend to measure and reported results do not necessarily speak about the concept under investigation.

The Causal and the Ontological Perspectives on the Relationship Between Concept and Concept Attributes

We argue that the most-often used operationalization approach (CFA or average scores) is rooted in a measurement paradigm that is often applied to latent constructs and which implicitly views the relationship between concept and concept components as causal (‘Bollen approach’).  We argue that a different perspective is needed when multi-dimensional concepts are non-compensatory. This ontological perspective we advocate allows for two operationalization strategies that account for this concept property, namely the Sartori and the Goertz approaches.

A straightforward Goertz-procedure is to use the minimum value of the concept components. The Sartori-approach entails setting thresholds on each concept components. Both approaches ensure to only assign high values to individuals if they score high on ALL concept components. These distinctions may seem like nitpicking. But they can make a crucial difference for substantive conclusions concerning nature and correlates of populist attitudes.

Correlations Between Goertz and Bollen Concept Structures of Populist Attitudes

Our preferred operationalization approach (Goertz) and the established approach (Bollen) result in different populism scores. Correlations between them are 0.4 to 0.9 Note: these scores were derived from the same data-generating process. They only differ in the aggregation rule!

Consider institutional trust. Bivariate Correlations with Institutional TrustBollen composite scores suggest that higher levels of populist attitudes go along with lower levels of trust. Yet, apparently, this association is driven entirely by anti-elitist orientations and not by the distinct concept of populist attitudes as an attitudinal syndrome at the intersection of ALL subdimensions. When operationalizing populist attitudes in a way that accounts for the non-compensatory relationship of the subdimensions (Goertz), then the seeming association between institutional trust and populist attitudes disappears.

Screenshot Shiny Web Application

We examined a large number of correlations in many datasets with different populism scales. In most cases, Goertz index indicates weaker associations with substantive variables than Bollen. In many cases, the conclusions do not differ. But too often they do,sometimes drastically. Using the Shiny Web Application, you can examine yourself how the disparities of the Sartori, Bollen and Goertz concept structures vary with scales and countries. Also, select a country and scale of your choice and see correlations with variables of interest. In addition, we use the Shiny App to demonstrate the relevance of researcher discretion. Specifically, using the Sartori approach we show how the estimated share of ‘populists’ various with specific details of the operationalization.

In addition to the Shiny Web Application, the study provides extensive Supplementary Materials. If your research deals with (non-compensatory) multi-dimensional concepts you might want to have a look at our step-by-step guide in Supplement 2. If you study populist attitudes (using the Schulz et al. Akkerman et al., Castanho Silva et al., Oliver/Rahn scale or the CSES scale of populist attitudes) you might want to have a look at Supplement 6 in which we discuss each scale and suitable operationalization strategies.

StudyShiny Web ApplicationReplication MaterialsSupplementary Materials

Democracy without democrats?

One-third of the world population lives in countries that move away from liberal democracy. Apparently, liberal self-governance is under stress. Democracy’s unexpected fragility has fueled scholarly interest in the regime preferences of ordinary citizens. Are democracy’s attitudinal foundations eroding? Do citizens turn away from liberal democracy and towards illiberal forms of governing?

We investigate these questions in a series of papers that are currently under review. Notably, we do not find consistent evidence for the erosion of democratic support in any of our studies. There is no coherent pattern of declining democratic support neither overtime nor across generations. With respect to Germany, we have summarized some of our findings [German] for a general audience for the blog of the German Political Science Association.

Almost universally, both young and older citizens want to live in a democracy and they endorse democratic self-governance as the preferred system of governing. In the same vein, there is no stark increase in support for non-democratic alternatives to governing either. However, while citizens continue to support democracy in the abstract they are not as supportive of how democracy’s central institutions work in practice. A final, notable result of our research is the remarkable re-politicization that has occurred over the past two decades. 

Altogether, however, the overall picture is that the attitudinal foundations of democracy remain intact. Yet, these findings do not suggest that democracy is safe. Citizen attitudes towards democracy are but one indicator of a healthy democracy. What is more: Attitudes are not enough. What also matters is that citizens translate their attitudes into political practice when it really matters. 

Publication in West European Politics: New political parties through the voters’ eyes

Why do citizens at the ballot box prefer new political parties over established political parties?

Scholars mainly studied the formation of political parties on the macro-level. Because party establishment is by definition tied to electoral accomplishments, macro-level studies on successful party establishment explicitly, or implicitly, presuppose assumptions about the behaviour of individual citizens whose votes are necessary for electoral success. Against that backdrop, I argue that to explain the conditions of successful party establishment we need to understand the motivational underpinnings of voting for a new party on the individual level.

Using cross-national voter surveys and long-term panel data from Germany (2005-2013) and the Netherlands (1998-2002), the study contrasts the implicit assumptions about voter behaviour of two macro-level theories on party emergence: the equilibrium- and the protest-based approaches on party emergence. The empirical analysis shows that there are merits to both approaches, but in many cases, the individual-level assumptions of both models do not square with the observed voting behaviour.

Determinants of Voting for a New Political Party

Although proximity to a new party matters, the findings do not support the equilibrium perspective’s tenet that new parties gain votes from citizens whose views were not represented in the preceding election. Moreover, political discontent was found a fertile soil for new parties to gain electoral support, but the relationship between discontent and voting is more complex than theoretically suggested. These findings on individual voter behaviour may inform further theoretical work on the successful establishment of new political parties. 

Pre-Print; Original articleReplication Material

Interviews on Assessing and Improving the Credibility of Social Science Research

The MZES Open Social Science Conference 2019 is over. 80 participants from all world continents (except Oceania) have participated, discussing and practicing „New Standards in Research Transparency and Reproducibility“.

The entire conference was live-streamed, sparking vivid debates on twitter with several hundred tweets using the #ossc19 hashtag. If you have missed the conference, you can all watch all keynote speeches, presentations and discussions on YouTube.

Keynote lecture: Arthur Lupia | It’s Up to Us – Transparency and the Public Value of Science

After the conference, I gave two interviews to German Public radio on asseing and improving the credibility on social science research:

https://www.deutschlandfunk.de/sozialwissenschaftliche-forschung-nicht-so-glaubwuerdig-wie.676.de.html?dram:article_id=439728

https://www.swr.de/swr2/programm/sendungen/impuls/replikationskrise-in-den-sozialwissenschaften/-/id=1853902/did=23333472/nid=1853902/sbkp7p/index.html

Talking about politics: When, with whom and to what effect?

When you talk to your spouse or friends, what do you talk about? Talk has political ramifications. According to moral psychologist Jonathan Haidt, human beings tend to disregard most information that contradict our pre-existing attitudes but social cues are the single influence with the potential to elicit a lasting change of mind.

However, although communication with friends and neighbours has the potential of transmitting new information and changing our minds, this potential will not materialize if we only converse with like-minded people. Considering ongoing debates about filter-bubbles and echo-chambers, this seems to be what is happening.

The Campaign Panel of the German Longitudinal Election Study allows shedding some light on the personal communication of ordinary citizens. In nine survey waves, respondents were repeatedly asked about the political conversations they have had in the past week: Whether they had any and with whom. Naturally, these measures are self-reported and therefore imperfect. Having these caveats in mind, here are some basic findings of research in collaboration with Lea Gärtner.

First, individuals differ strongly in whether and how often they talk about politics. Figure 1 shows the distribution of self-reported frequency of political conversations over the past week. Some people never touched upon this issue while a handful of people talked about politics every day.

Figure 1: Frequency of Political Conversations

We also investigated whether citizens only talk with like-minded people. In this case, „like-minded“ is operationalized as talking to a person who the respondent perceives as having the same vote intention. Specifically, we calculated the relative homogeneity of a person’s discussion network where „1“ denotes a respondent’s discussion network is not biased towards her own party. If the respondent intends to vote for CDU and the CDU polls at 30% then the discussion network is unbiased if it consists of 30% CDU voters. However, if 60% of the respondent’s conversation partners intend to vote CDU, then the respondent apparently prefers like-minded conversation partners and the coefficient would be „2“.

Figure 2: The Development of Homogeneity of Discussion Networks across the Election Campaign

Figure 2 provides three fundamental insights. First, citizens do prefer like-minded conversation partners. Regardless of party preference, the average citizen has twice as many conversation partners with identical vote preferences in her discussion network than we would have expected if discussion networks were unbiased. Second, the homogeneity of discussion networks does not increase when the election draws closer. Third, voters of AfD and (surprisingly?) voters of the Green Party have the strongest inclination to talk to like-minded people.

Finally, we should note that while we observe a bias towards congeniality, most of the respondents do not live in filter-bubbles. Although people prefer talking with like-minded others, other analyses show that most people also do talk to citizens with other points of view.

New publication in PVS: Why too many political science findings cannot be trusted and what we can do about it

For several months or years now I have been learning and thinking about the credibility of social scientific research. In the wake of this progress, I thoroughly revised my beliefs about the evidential value of published research findings. When I was conducting my master’s studies and was reading a scientific study published in a scientific journal by full-grown scientists, I believed and never questioned that was I was reading must be true. I knew that some studies were better than others and that scientists occasionally err. Still, when I heard that one of those full-grown scientists claimed that „most published research findings are false“ I was puzzled.

Years later I am being involved in organizing the MZES Open Social Science Conference 2019, which brings together scholars from around the globe to discuss research credibility and what we can do improve it. Now, I have penned a commentary on Open Science with a particular focus on political science. Rather than being fatalistic, the commentary hopes to be analytic and constructive. The article is a review of the existing meta-scientific literature on the credibility of quantitative findings in political science. At best, the paper will serve as a primer for those who are not yet versed in these discussions and it will contribute to the ongoing debates about research credibility.

The article is now published online at Springer PVS and will be published in print in the Issue 1/2019 of the German Political Science Quarterly.

Why Too Many Political Science Findings Cannot Be Trusted and What We Can Do About It: A Review of Meta-Scientific Research and a Call for Academic Reform

Ungated Pre-Print

Open Science Talk in Zurich: Warum wir sozialwissenschaftlichen Forschungsbefunden oft nicht trauen können

The IKMZ (Institut für Kommunikationswissenschaft und Medienforschung) invited me for a talk on the credibility of social scientific research. The presentation sparked an engaged debate about the incentive structures in academia and about what we can do to better align individual researcher rationality with the collective good of reliable research.

You can find the presentation here.