Happy to share a pre-print of one of the most fascinating projects I have been involved in.
With Nate Breznau and Eike Rinke we have coordinated the research efforts of 163 social scientists who simultaneously be independently investigated the same research question with the same data. The result of that exercise reveals a previously hidden universe of scientific uncertainty that changes how interpret and approach scientific inquiry more. More info below.
Abstract
This study explores how analytical choices of researchers affect the reliability of scientific findings. Current lack-of-reliability discussions focus on systematic biases. We broaden the lens to include idiosyncratic decisions in data analysis that lead researchers to diverging results and conclusions. We coordinated and observed decisions among 73 research-teams as they independently tested the same hypothesis using the same data. Results show that in this typical secondary data research situation, the universe of pathways from data to results is so vast that each analysis was unique in some way. Teams reported divergent findings with contradictory substantive implications that could not be explained by differences in researchers’ expertise, prior beliefs, and expectations. This calls for greater humility and clarity in presentation of scientific findings. Idiosyncratic variation may also be a cause for why many hypotheses remain highly contested, particularly in large-scale social and behavioral research.
