Why too many political science findings cannot be trusted and what we can do about it

For several months or years now I have been learning and thinking about the credibility of social scientific research. In the wake of this progress, I thoroughly revised my beliefs about the evidential value of published research findings. When I was conducting my master’s studies and was reading a scientific study published in a scientific journal by full-grown scientists, I believed and never questioned that was I was reading must be true. I knew that some studies were better than others and that scientists occasionally err. Still, when I heard that one of those full-grown scientists claimed that „most published research findings are false„, I was puzzled.

Years later I am being involved in organizing the MZES Open Social Science Conference 2019, which brings together scholars from around the globe to discuss research credibility and what we can do improve it. Now, I have penned a commentary that reviews the existing meta-scientific literature on the credibility of quantitative findings in political science. Rather than fatalistic, the commentary hopes to be analytic and constructive. At best, the paper will serve as a primer for those who are not yet versed in these discussions and it will contribute to the ongoing debates about research credibility.

The German Political Science Quarterly (Politische Vierteljahresschrift) will generously publish this commentary at the end of this year. The commentary is on SocArxiv in a very draft. Comments and feedback are very welcome!

Why too many political science findings cannot be trusted and what we can do about it: Assessing, explaining and improving the credibility of our discipline’s evidence base

Release of unique longitudinal survey data: Repeatedly questioned respondents of the Short-term Campaign Panel 2013 and 2017

For which party have citizens voted back in 2013 who cast their ballot for the AfD in 2017? Were these voters AfD-adherents all along or did they abstain at the previous election or were they even voters of leftist parties at the previous elections?

In survey research, there are two ways of finding out: First, you can ask citizens after the federal elections 2017 about their vote choice at the recent election and then you also ask them to remember their vote choice at the previous election. The problem is that human beings are bad at remembering things. Just as we forget where we put our keys (only 5 minutes ago!) we also forget who we voted for four years ago. We are very bad at remembering. Research by Ruth Dassonneville shows that only half of those voters who have actually switched party choices between elections do remember that they had switched.

Therefore, the better option is having asked those voters already in 2013 about their vote choices and then to ask them again right after the elections in 2017.  This requires tracing respondents for a long period of time which means a lot of survey administration (and comes with other caveats, such as panel attrition). However, in addition to the accurate recall, such longitudinal survey data provide so many other benefits. For each individual, you can trace how his or his opinions about Merkel and immigrants (and how the personality or individual sense of efficacy changed) over all these years.

Such a longitudinal dataset is now available for the German Federal Elections 2013 and 2017.

  • Respondents surveyed 18 times
  • before/after German federal elections 2013 & 2017
  • N=2,725
  • about 5,000 substantive variables
  • free, open and well documented (German only at the moment)

You can find the data here.

This dataset is provided by German Longitudinal Election Survey. Other GLES panel datasets include GLES 2017 Campaign Panel (web, N=20,000), Long-Term Panel surveys (based on probability facetoface sampling), RCS (2 waves, telephone).

MZES Open Social Science Conference: Call for Papers published

With my amazing colleagues Nate Breznau and Eike Mark Rinke, I have the pleasure to organize a conference on a topic that is of personal importance for me. It concerns the practices and politics of doing research and is about establishing new standards of reproducibility and transparency in social science research.

Here is our call for papers. Please forward it to friends and colleagues and consider participating. Feel free to contact me with any questions you might have.

CfP — MZES Open Social Science Conference 2019: Practicing New Standards in Transparency and Reproducibility

This conference is a forum for practicing and discussing credibility, transparency and replicability in the social sciences.

About a decade ago, John Ioannidis claimed that “most published research findings are false”. While seeming outrageous at the time, a growing body of meta-scientific research in the behavioral and social sciences substantiated this claim, causing uncertainty about the trustworthiness of published scientific findings. We believe that threats to the validity of published findings in the social sciences are widespread and systemic. Therefore, this conference promotes introspection about the current state of social science research and exchange on the opportunities for institutional and methodological improvement in the future.

The conference is supported by the Berkeley Initiative for Transparency in the Social Sciences (BITSS) and will take place from 25-27 January 2019 in Mannheim, Germany.

Conference Websitetwitter

Keynote speakers

Jeremy Freese (Stanford University)
Thomas König (APSR, University of Mannheim)
Arthur Lupia (OSF, University of Michigan)
Julia Rohrer (100% CI, Leipzig University)

Participate in the conference

  • Give a talk: We call for researchers to advance discussion, debate, literature synthesis, or methods in open social science. We welcome methodological advances, e.g., p-curve analysis, systematic reviews, pre-analysis planning, and replication. We welcome general research findings that apply best practices of open science while conducting the research – Abstract submission DL: 22 August 2018  Read more
  • MZES-GESIS Pre-Registration Challenge:  We call for researchers to participate in a competition to win funding or survey time for the most innovative and rigorous pre-registration plan for a social science study. – Abstract submission DL: 22 August 2018  Read more
  • OSSC19 Crowdsourced Replication Initiative: We call for researchers to replicate and expand a previously published cross-national macro-comparative study. The goal is to explore and develop crowdsourcing methods and generate research surpassing what a single researcher could achieve. The replication comes from the field of immigration and social policy, but we encourage social science researchers of all disciplines and levels to participate. All full participants will be co-authors on the final paper. – Registration DL: 27 July 2018   Read more

Participate as a guest in Mannheim during the conference or during the subsequent Open Science Workshop, offered in collaboration with the Berkeley Initiative for Transparency in the Social Sciences (BITSS). Or use the live stream online.


Article at LSE Business Review: For free trade, but against TTIP: public opinion and the backlash against globalisation

LSE Business Review has published an article of ours (Jungherr, Mader, Schoen, Wuttke) summarizing the paper „Context-driven attitude formation: the difference between supporting free trade in the abstract and supporting specific trade agreements“, published in Review of International Political Economy. 

Considering current discussions about the eminent trade war between the US and Europe and related discussions about „TTIP light“, we hope this piece has some value for explaining public support and opposition towards free trade and specific free trade agreements:

LSE Business Review: For free trade, but against TTIP: public opinion and the backlash against globalisation

Why some people care about politics and others hardly bother at blog – Blog Post at International Society of Political Psychology

Theory in a nutshell

Why some people value political engagement and even find pleasure in engaging with politics while others hardly bother about the political domain, is a crucial question for the functioning of democratic societies. In my dissertation, I investigate what we can learn from motivation studies (largely neglected by political scientists) to understand the roots of political engagement. My contribution is to investigate the nonpolitical origins of political engagement in early childhood experiences, more specifically, the level of need satisfaction provided in the parental home during socialization’s formative phase.

I had the opportunity to write about these thoughts and the findings from two analyses of longitudinal cohort data at the ISSP blog. Please feel free to provide any comments and criticism.

Link to ISPP blog post on the roots of political engagement

[MZES 2019] Open Social Science Conference: Practicing New Standards in Research Integrity and Reproducibility

Open Science refers to a broad, transdisciplinary movement aiming to open up research processes to promote the integrity, reproducibility and value of scholarly work, in part as a response to pressing problems facing science. This movement thoroughly changed the field of psychology, and clearly a similar fundamental rearrangement is coming in all social sciences, for example political science, sociology, and communication.

With Nate Breznau and Eike Rinke, I have the pleasure of organizing  a meta-scientific conference, which discusses, advances and pushes for Open Science in research on politics and society.

Find preliminary information on our website.

Mannheim, Jan 25th-27th 2019.

Conference structure
Stream 1: Advancing Open Science: Problems in and solutions for research on politics and society
Stream 2: Doing Open Science: Applications in research on politics on society
Stream 3: Debating Open Science: Perspectives on the State of Social Science
Pre-registration challenge
Crowdsourced replication effort
BITSS Workshop on reproducible research

Key note speakers
Jeremy Freese (Stanford University)
Thomas König (APSR, University of Mannheim)
Arthur Lupia (OSF, University of Michigan)
Julia Rohrer (100% CI, Uni Leipzig)

How to participate
In June, we will distribute the CfP and information on the crowdsourced replication effort and the pre-registration challenge.

Follow us on twitter: @opensocsci

New publication [German, Zeitschrift für Parlamentsfragen]: A swing in the final weeks of the campaign: evidence from a multi-wave panel survey on the 2017 state election in Lower Saxony.

The state election in Lower Saxony in October 2017 walked in the shadows of the preceding election on the federal level. In light of both campaigns’ temporal proximity, this study investigates the development of political attitudes towards the federal and the state level and their relative influence on voter behavior in Lower Saxony. We make use of a multi-wave panel survey of 1,346 eligible voters in Lower Saxony, which was carried out in the framework of the German Longitudinal Election Study (GLES). Cross-sectional and longitudinal analyses show that a significant minority of respondents based their voting decision in the state election on state-specific considerations. State-specific attitudes varied strongly and to a large extent independently of attitudes towards parties and politicians at the federal level and went hand in hand with changes in voting intentions. The Lower Saxony SPD, in particular, was able to promote the popularity of its leading candidate, thereby leaving behind the CDU at the ballot boxes. [ZParl, vol. 49 (2018), no. 1, pp. 22 – 39]



Ein Umschwung in den letzten Wochen des Landtagswahlkampfes: Befunde einer mehrwelligen Wiederholungsbefragung zur niedersächsischen Landtagswahl 2017

Die niedersächsische Landtagswahl im Oktober 2017 fand im Schatten der vorhergehenden Bundestagswahl statt. Vor dem Hintergrund der zeitlichen Nähe beider Wahlgänge untersucht dieser Aufsatz die Entwicklung bundes- und landesbezogener Einstellungen im Wahlkampfverlauf sowie deren Einfluss auf das Wählerverhalten in Niedersachsen. Eine mehrwellige Wiederholungsbefragung von 1.346 niedersächsischen Wahlberechtigten, die im Rahmen der German Longitudinal Election Study (GLES) durchgeführt wurde, dient als Grundlage der Untersuchung. In quer- und längsschnittlichen Analysen wird gezeigt, dass eine bedeutende Minderheit der Befragten den Wahlentscheidungen auf der Landesebene ebenenspezifische Erwägungen zu Grunde legt. Landesbezogene Einstellungen waren einer von der Bundesebene weitgehend unabhängigen Wahlkampfdynamik unterworfen, die die Wahlabsichten des Elektorats beeinflusst zu haben scheinen. So konnte gerade die niedersächsische SPD die Popularität ihres Spitzenkandidaten in den letzten Wahlkampfwochen steigern und dadurch die CDU in einem Schlussspurt als stärkste Kraft ablösen. [ZParl, 49. Jg. (2018), H. 1, S. 22 – 39]

replication material
Published panel survey data on Lower Saxony

German Campaign Panel Publicly Available!

The 2017 German elections (and its aftermath) were extraordinary. At the campaign’s outset, the SPD experienced the strongest surge in public approval ever observed in German polling but months later attitudes had shifted again and the party ended up with its lowest election result in decades. The decline of Germany’s oldest democratic party was accompanied by the re-entry of the Free Democrats into German parliament and the rise of a populist right-wing party propagating stances on immigration and German history that for long were treated as „taboo“ in public discourse. Finally, the elections brought a new parliament but, for the first time in German history, no government.

With the publication of the 2017 GLES campaign panel you now have data to investigate these elections that might once mark a historic transition in German politics.

Data download & documentation

Key facts

  • Eight survey waves (more to come)
  • 22,526 respondents
  • side-study on parallel provincial snap election
  • extensive documentation (English translation soon)
  • harmonized with all other GLES components (face 2 face survey, media analysis, RCS etc.)

Next steps

  • A few weeks after the federal election in Germany, the citizens of Lower Saxony had the chance to vote again in a provincial snap election. Shortly after the provincial snap elections were called, we included an extensive survey battery on attitudes towards politicians and parties in Lower Saxony to assess how the parallel campaigns on the federal and on the provincial levels impact level-specific attitudes (survey wave 5ff.). These survey measure are included in the published data file. We conducted an additional post-election survey wave, exclusively for respondent from Lower Saxony. This wave will be added soon.
  • After a decision has been made about whether our country will either have a government again or will face new elections, we will conduct another survey wave (wave 9), which then will be added to the data set.
  • English translations will follow.
  • More information on survey measures (which instrument in which waves etc.) will follow.

Data download & documentation

New publication in [Review of International Political Economy]: Context-Driven Attitude Formation – The Difference Between Supporting Free Trade in the Abstract and Supporting Specific Trade Agreements

Contextual model of attitude formation

Many studies use the same factors to explain attitudes toward specific trade agreements and attitudes toward the principle of free trade and thus treat both objects as interchangeable. Contemporary trade agreements, however, often reach beyond trade in the narrow sense. Consequently, factors unrelated to free trade may affect citizens’ evaluations of these agreements. We propose a model of attitude formation toward specific trade agreements that includes the societal context as a constitutive feature. We expect salient aspects of an agreement to activate corresponding predispositions. Empirically, we compare how this contextual model and a standard model perform in explaining German citizens’ attitudes toward free trade and the Transatlantic Trade and Investment Partnership (TTIP). The results show that the standard model performs well in explaining public opinion on the principle of free trade but is less useful in explaining attitudes toward TTIP. The latter were driven by postures toward transatlantic cooperation, predispositions toward the role of interest groups in politics, and market regulation – aspects salient in German public discourse about TTIP. In sum, we find ample evidence for the need to differentiate between the two attitude objects and for our contextual model of attitude formation.

The article was was published in the Review of International Political Economy (with Andreas Jungherr, Matthias Mader, Harald Schoen). It is Open Access. You can find replication material here.

Kurzfassung: Wie Partizipation wirkt – Attraktivere Parteien durch mehr Mitbestimmung?

Die Hauptbefunde der kürzlich veröffentlichte Studie (Party Politics, mit Andreas Jungherr und Harald Schoen) zur Wirkung innerparteilicher Partizipation haben wir kompakt und (hoffentlich) verständlich beim Wissenschaftsblog DeFacto zusammengefasst.

Unsere Untersuchung zeigt, dass die Konsequenzen eines innerparteilichen Referendums über den Entscheid in der Sachfrage hinausreichen :

Ein Mitgliedervotum verändert, wie Mitglieder auf ihre Partei blicken. Wer über Sachfragen abstimmen darf, fühlt sich als Parteimitglied ermächtigt. Mitglieder lernen im Diskussionsprozess über das Verhalten der Parteispitzen und bewerten sie stärker auf Grund ihrer eigenen inhaltlichen Überzeugungen.

Weitere Informationen sind auf einer studienbegleitenden Webseite zu finden. Kommentare sind willkommen!

Wie Partizipation wirkt: Attraktivere Parteien durch mehr Mitbestimmung?