Reproducibility in the absence of selective reporting: An illustration from large‐scale brain asymmetry research
- Human brain mapping
The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p-hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multisite collaborative study, we mapped brain anatomical left–right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we revisited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and significance threshold. In this sense, the results within each dataset were viewed as coming from separate studies in an “ideal publishing environment,” that is, free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.
Xiang‐Zhen Kong, ENIGMA Laterality Working Group, Clyde Francks, Xiang‐Zhen Kong, Samuel R Mathias, Tulio Guadalupe, Christoph Abé, Ingrid Agartz, Theophilus N Akudjedu, Andre Aleman, Saud Alhusaini, Nicholas B Allen, David Ames, Ole A Andreassen, Alejandro Arias Vasquez, Nicola J Armstrong, Phil Asherson, Felipe Bergo, Mark E Bastin, Albert Batalla, Jochen Bauer, Bernhard T Baune, Ramona Baur‐Streubel, Joseph Biederman, Sara K Blaine, Premika Boedhoe, Erlend Bøen, Anushree Bose, Janita Bralten, Daniel Brandeis, Silvia Brem, Henry Brodaty, Dilara Yüksel, Samantha J Brooks, Jan Buitelaar, Christian Bürger, Robin Bülow, Vince Calhoun, Anna Calvo, Erick Jorge Canales‐Rodríguez, Dara M Cannon, Elisabeth C Caparelli, Francisco X Castellanos, Fernando Cendes, Tiffany Moukbel Chaim‐Avancini, Kaylita Chantiluke, Qun‐lin Chen, Xiayu Chen, Yuqi Cheng, Anastasia Christakou, Vincent P Clark, David Coghill, Colm G Connolly, Annette Conzelmann, Aldo Córdova‐Palomera, Janna Cousijn, Tim Crow, Ana Cubillo, Udo Dannlowski, Sara Ambrosino de Bruttopilo, Patrick de Zeeuw, Ian J Deary, Damion V Demeter, Adriana Di Martino, Erin W Dickie, Bruno Dietsche, Nhat Trung Doan, Colin P Doherty, Alysa Doyle, Sarah Durston, Eric Earl, Stefan Ehrlich, Carl Johan Ekman, Torbjørn Elvsåshagen, Jeffery N Epstein, Damien A Fair, Stephen V Faraone, Guillén Fernández, Claas Flint, Geraldo Busatto Filho, Katharina Förster, Jean‐Paul Fouche, John J Foxe, Thomas Frodl, Paola Fuentes‐Claramonte, Janice M Fullerton, Hugh Garavan, Danielle do Santos Garcia, Ian H Gotlib, Anna E Goudriaan, Hans Jörgen Grabe, Nynke A Groenewold, Dominik Grotegerd, Oliver Gruber, Tiril Gurholt, Jan Haavik, Tim Hahn, Narelle K Hansell, Mathew A Harris, Catharina A Hartman, Maria del Carmen Valdés Hernández, Dirk Heslenfeld, Robert Hester, Derrek Paul Hibar, Beng‐Choon Ho, Tiffany C Ho, Pieter J Hoekstra, Ruth J van Holst, Martine Hoogman, Marie F Høvik, Fleur M Howells, Kenneth Hugdahl, Chaim Huyser, Martin Ingvar, Akari Ishikawa, Anthony James, Neda Jahanshad, Terry L Jernigan, Erik G Jönsson, Vasily Kaleda, Clare Kelly, Michael Kerich, Matcheri S Keshavan, Sabin Khadka, Tilo Kircher, Gregor Kohls, Kerstin Konrad, Ozlem Korucuoglu, Bernd Krämer, Axel Krug, Jonna Kuntsi, Jun Soo Kwon, Nanda Lambregts‐Rommelse, Mikael Landén, Luisa Lázaro, Irina Lebedeva, Rhoshel Lenroot, Klaus‐Peter Lesch, Qinqin Li, Kelvin O Lim, Jia Liu, Christine Lochner, Edythe D London, Valentina Lorenzetti, Michelle Luciano, Maartje Luijten, Astri J Lundervold, Scott Mackey, Frank P MacMaster, Sophie Maingault