Crowdsourcing Hypothesis Tests

None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None; None, None

Crowdsourcing Hypothesis Tests

Making transparent how design choices shape research results

Journal Article (2020)

Author(s)

Justin F. Landy (Nova Southeastern University)

Miaolei Liam (University of Warwick, Warwick Business School)

Isabel L. Ding (National University of Singapore)

Domenico Viganola (George Mason University)

Warren Tierney (Kemmy Business School, Mary Immaculate College)

Anna Dreber (University of Innsbruck, Stockholm School of Economics)

Magnus Johannesson (Stockholm School of Economics)

Thomas Pfeiffer (Massey University)

Charles R. Ebersole (University of Kassel, University of Virginia)

Quentin F. Gronau (Universiteit van Amsterdam)

Alexander Ly (Universiteit van Amsterdam)

Don Van Den Bergh (Universiteit van Amsterdam, Max Planck Institute for Human Development)

Maarten Marsman (University of Roehampton, Universiteit van Amsterdam)

Koen Derks (University of Roehampton, Nyenrode Business Universiteit)

Eric Jan Wagenmaker (Pacific Lutheran University, Nova Southeastern University)

Andrew Proctor (Stockholm School of Economics, Universität Bremen)

Daniel M. Bartels (University of Chicago)

Christopher W. Bauman (University of California)

William J. Brady (New York University)

Felix Cheung (The University of Hong Kong)

Andrei Cimpian (University of Cologne, New York University)

Simone Dohle (University of Cologne, Michigan State University)

M. Brent Donnellan (Michigan State University, Universitetet i Oslo)

Adam Hahn (University of Michigan, University of Cologne)

Michael P. Hall (University of Michigan)

William Jiménez-Leal (Universidad de los Andes, University of Andes)

David J. Johnson (University of Maryland, Department of Sociology)

Richard E. Lucas (Michigan State University, Harz University of Applied Sciences and University of Bamberg)

BenoÎt Monin (Stanford University)

Andres Montealegre (Universidad de los Andes, University of Kassel)

Elizabeth Mullen (San José State University)

Jun Pang (Renmin University of China, Renmin Business School)

Jennifer Ray (New York University, Renmin Business School)

Diego A. Reinero (New York University, Western Kentucky University)

Jesse Reynolds (Stanford University, Abilene Christian University)

Walter Sowden (University of Michigan)

Daniel Storage (University of Denver)

Runkun Su (Duke University, NUS Business School)

Christina M. Tworek (HarrisX)

Jay J. Van Bavel (Masaryk University, New York University)

Daniel Walco (New York Yankees)

Julian Wills (National University of Singapore, New York University)

Xiaobing Xu (Tsinghua University, Hainan University)

Kai Chi Yam (Rotman School of Management, Catholic University of the Sacred Heart)

Xiaoyu Yang (Rotman School of Management, Tsinghua University)

Yen Ping Chang (Institute of Sociology)

Tina S.T. Huang (Instituto Universitário de Lisboa (ISCTE-IUL))

Samuel G.B. Johnson (University of Bath)

Oscar Oviedo-Trespalacios (Queensland University of Technology)

Yoo Jin Lee (Insead)

Affiliation

External organisation

DOI related publication

https://doi.org/10.1037/bul0000220

Forecasting Crowdsourcing Conceptual replications Research robustness Scientific transparency

To reference this document use:

https://resolver.tudelft.nl/uuid:ac7d95d2-c6ef-443b-89c9-9c0dbb268e29

More Info

expand_more

Publication Year

2020

Language

English

Affiliation

External organisation

Journal title

Psychological Bulletin

Issue number

5

Volume number

146

Pages (from-to)

451-479

Downloads counter

346

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

No files available

Metadata only record. There are no files for this record.