In a new study, now published with the American Political Science Review, we show how established operationalization strategies of multi-dimensional concepts can systematically lead to wrong conclusions. Focusing on populist attitudes, we demonstrate simple methods to align theory and measurement.
Our argument refers to a specific but common type of multi-dimensional concepts which are sometimes called ‘non-compensatory’. Multi-dimensional concepts are non-compensatory when higher values on one component cannot offset lower values on another.
Think of democracy: If we believe that a country only counts as a democracy if it provides both rule of law and free elections then no valid measure of democracy will assign high democracy scores when ‚rule of law‘ scores are low even when the elections are extraordinarily fair.
However, not all social science studies consider this seemingly minor but consequential issue when putting multi-dimensional concepts into empirical practice. Focusing on populist attitudes, we show that these measurement-concept inconsistencies can lead to wrong conclusions.
Populism is an essentially contested concept. Yet, most scholars now agree that populist attitudes are multi-dimensional (e.g.: anti-elitism + Manicheanism + Sovereignty). Importantly, there is also widespread agreement on the idea that populist attitudes lie at the intersection of the concept’s sub-dimensions. Hence, the unique property of populist attitudes is the co-existence of its components. Put differently, we only consider citizens (or leaders) as populists if they accept anti-elitist views AND a Manichean outlook AND support popular sovereignty. Importantly, it is this non-compensatory concept property that distinguishes populist attitudes from other established public opinion constructs (eg cynicism, efficacy, ethnocentrism), that makes populist attitudes worthwhile as a concept and that makes it more than the sum of its parts. Yet, existing studies on populism at the mass level rarely transfer this crucial concept feature into empirical practice. Hence, some populism studies do not measure what they intend to measure and reported results do not necessarily speak about the concept under investigation.
We argue that the most-often used operationalization approach (CFA or average scores) is rooted in a measurement paradigm that is often applied to latent constructs and which implicitly views the relationship between concept and concept components as causal (‘Bollen approach’). We argue that a different perspective is needed when multi-dimensional concepts are non-compensatory. This ontological perspective we advocate allows for two operationalization strategies that account for this concept property, namely the Sartori and the Goertz approaches.
A straightforward Goertz-procedure is to use the minimum value of the concept components. The Sartori-approach entails setting thresholds on each concept components. Both approaches ensure to only assign high values to individuals if they score high on ALL concept components. These distinctions may seem like nitpicking. But they can make a crucial difference for substantive conclusions concerning nature and correlates of populist attitudes.
Our preferred operationalization approach (Goertz) and the established approach (Bollen) result in different populism scores. Correlations between them are 0.4 to 0.9 Note: these scores were derived from the same data-generating process. They only differ in the aggregation rule!
Consider institutional trust. 
Bollen composite scores suggest that higher levels of populist attitudes go along with lower levels of trust. Yet, apparently, this association is driven entirely by anti-elitist orientations and not by the distinct concept of populist attitudes as an attitudinal syndrome at the intersection of ALL subdimensions. When operationalizing populist attitudes in a way that accounts for the non-compensatory relationship of the subdimensions (Goertz), then the seeming association between institutional trust and populist attitudes disappears.
We examined a large number of correlations in many datasets with different populism scales. In most cases, Goertz index indicates weaker associations with substantive variables than Bollen. In many cases, the conclusions do not differ. But too often they do,sometimes drastically. Using the Shiny Web Application, you can examine yourself how the disparities of the Sartori, Bollen and Goertz concept structures vary with scales and countries. Also, select a country and scale of your choice and see correlations with variables of interest. In addition, we use the Shiny App to demonstrate the relevance of researcher discretion. Specifically, using the Sartori approach we show how the estimated share of ‘populists’ various with specific details of the operationalization.
In addition to the Shiny Web Application, the study provides extensive Supplementary Materials. If your research deals with (non-compensatory) multi-dimensional concepts you might want to have a look at our step-by-step guide in Supplement 2. If you study populist attitudes (using the Schulz et al. Akkerman et al., Castanho Silva et al., Oliver/Rahn scale or the CSES scale of populist attitudes) you might want to have a look at Supplement 6 in which we discuss each scale and suitable operationalization strategies.
Study – Shiny Web Application – Replication Materials – Supplementary Materials
