The Journal of Politics, a major and widely read publication in political science, has modified its editorial policy which breaks ground for political science. From now on*, all experimental studies submitted to JOP must have a pre-registration plan that was submitted before data collection. I want to explain why this policy is a step forward but it may not be the right one. And why another instrument helps us achieve all those goals we want to accomplish while avoiding some of the drawbacks of this policy. (Spoiler: The magical key is ‚Registered Reports‘)
Why the fuss?
No political science journal has ever put strict pre-registration policies in place. While the Transparency Openness Guidelines, published in Science a few years ago to promote openness in its various forms, embraces this policy, not many journals across disciplines have made pre-registrations mandatory.
What this policy can accomplish
The policy promotes pre-registration. Pre-registration makes the research process more observable, by giving readers a time-stamped snapshot of the researchers‘ thinking and intentions before data collection. This helps because 1) it discloses selective reporting as readers see whether authors have reported all outcomes they intended to analyze 2) it discloses p-hacking as readers can see whether authors deviated from the original analysis plan to obtain better-looking results 3) it discloses HARKing and allows distinguishing between exploratory and confirmatory research as readers can see whether authors constructed a hypothesis after seeing the data or whether a hypothesis was put a decisive test that could also have failed (which is not the case for exploratory work).
The better policy: Registered Reports
So while there are good reasons to promote pre-registration, this specific policy comes with unfortunate drawbacks (see below). Importantly, there is a better alternative that provides more favorable outcomes and fewer detriments. The Pareto-optimal solution – or JACKPOT as normal people would say – is a new article type called Registered Reports. Offered by 275 journals, Registered Reports reverse the order of the scientific publication process: Studies are reviewed before data collection based on the merits of theory, research question, and design. If the study meets the criteria of reviewers and editors, then the study is conditionally accepted for publication and the data will be collected and analyzed based on the pre-registration plan and then the study WILL BE PUBLISHED (even when the results are null or not spectacular – because the reviewers consider the research worthwhile regardless of the outcomes).
The drawbacks Registered Reports could circumvent
Twitter today was happy to discuss something non-COVID-related that also did not suggest the end of democracy is near. So, various twitter threads applaud the new policy or point to drawbacks. Here is how Registered Reports would address these drawbacks:
1) Required pre-registrations disadvantage scholars with fewer resources… (tweet by Hans Hassell)
because scholars with fewer resources cannot afford to field a second pre-registered and confirmatory study after having made a discovery in an exploratory study. Registered Reports do no solve this problem but may alleviate these concerns. A registered report with in-principle-acceptance after peer review WILL be published. With publication guaranteed, authors still have to collect the data for the pre-registered analysis but the researcher is freed from worries that the money might not return investment (will my research get published? will be research get published even if the results are null?).
2) The policy is inconsistent. Inconsistency causes psychological discomfort. Discomfort should be avoided (tweet by Michael Bang Petersen)
Whereas the new JOP policy only requires pre-registration from experimental studies (why not observational? etc…), Registered Reports are open to all studies regardless of research methodology or type of data – as long as pre-registration is feasible. This makes for a much cleaner policy and also allows for eg observational and qualitative studies.
3) Required pre-registration without results-blind review does not help much and fills file-drawers with unpublished studies (tweets by Matt Blackwell , Thomas Leeper)
The great advantage of Registered Reports is that they do not only disclose p-hacking, selective reporting and HARKing but also alleviate publication biases – all at the same time! Registered Reports shift the focus from eye-catching results to important research questions and rigorous research designs. All studies will be published if they meet the journal’s criteria for a solid design and an important research question.
4) The policy is too imprecise and invites low-quality pre-registrations (tweet by Matt Blackwell)
One big problem of pre-registration as they exist in academic reality is that they are often poorly done or not complied with in the final study. This is different for registered reports where the pre-analysis plan is an essential document in the review process. Because there are no empirical analyses or results to be judged, reviewers closely look at the *intended* analysis to judge its appropriateness for the research question. What is best, because the data is not yet collected, the plan and the study’s design can still be modified when the reviewers have good ideas for improvement. All of this makes for a more collaborative review process that is often experienced as more rewarding by reviewers and authors.
Which will be the first major journal to offer Registered Reports?
In 2019, a few dozen political scientists signed an open letter advocating for Registered Reports at the APSR where a new editorial team had just been announced. Actually, the letter was not so open as we had never publicized it because we did not want to put any public pressure on the journal. Although we have not received a substantive response from the journal on the proposal, it now seems now clear that Registered Reports will not be introduced at the APSR anytime soon. Nonetheless, I am convinced it is only a matter of time until we can submit Registered Reports to one of the top journals in our field (see Figure below). The only question is, which journal will be the first.
The JOP’s policy promotes the use of pre-registration in the discipline. However, generally speaking, Open Science practices are spreading so fast that some well-intended policies will inevitably be suboptimal. We should therefore always keep an open mind to see if there is room for improvement. We will see how this new policy will play out. Maybe we can use the gained experience to then take the next step: launch Registered Reports at a major journal in our discipline.
* after the transition period of one year
