Tags

Type your tag names separated by a space and hit enter

Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology.
PLoS One 2016; 11(5):e0153049Plos

Abstract

We describe a method of quantifying the effect of Questionable Research Practices (QRPs) on the results of meta-analyses. As an example we simulated a meta-analysis of a controversial telepathy protocol to assess the extent to which these experimental results could be explained by QRPs. Our simulations used the same numbers of studies and trials as the original meta-analysis and the frequencies with which various QRPs were applied in the simulated experiments were based on surveys of experimental psychologists. Results of both the meta-analysis and simulations were characterized by 4 metrics, two describing the trial and mean experiment hit rates (HR) of around 31%, where 25% is expected by chance, one the correlation between sample-size and hit-rate, and one the complete P-value distribution of the database. A genetic algorithm optimized the parameters describing the QRPs, and the fitness of the simulated meta-analysis was defined as the sum of the squares of Z-scores for the 4 metrics. Assuming no anomalous effect a good fit to the empirical meta-analysis was found only by using QRPs with unrealistic parameter-values. Restricting the parameter space to ranges observed in studies of QRP occurrence, under the untested assumption that parapsychologists use comparable QRPs, the fit to the published Ganzfeld meta-analysis with no anomalous effect was poor. We allowed for a real anomalous effect, be it unidentified QRPs or a paranormal effect, where the HR ranged from 25% (chance) to 31%. With an anomalous HR of 27% the fitness became F = 1.8 (p = 0.47 where F = 0 is a perfect fit). We conclude that the very significant probability cited by the Ganzfeld meta-analysis is likely inflated by QRPs, though results are still significant (p = 0.003) with QRPs. Our study demonstrates that quantitative simulations of QRPs can assess their impact. Since meta-analyses in general might be polluted by QRPs, this method has wide applicability outside the domain of experimental parapsychology.

Authors+Show Affiliations

Experimental Psychology, University of Groningen, Groningen, The Netherlands. Brain & Cognition, University of Amsterdam, Amsterdam, The Netherlands.LFR, Palo Alto, California, United States of America.Brain & Cognition, University of Amsterdam, Amsterdam, The Netherlands.

Pub Type(s)

Journal Article
Meta-Analysis

Language

eng

PubMed ID

27144889

Citation

Bierman, Dick J., et al. "Testing for Questionable Research Practices in a Meta-Analysis: an Example From Experimental Parapsychology." PloS One, vol. 11, no. 5, 2016, pp. e0153049.
Bierman DJ, Spottiswoode JP, Bijl A. Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology. PLoS ONE. 2016;11(5):e0153049.
Bierman, D. J., Spottiswoode, J. P., & Bijl, A. (2016). Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology. PloS One, 11(5), pp. e0153049. doi:10.1371/journal.pone.0153049.
Bierman DJ, Spottiswoode JP, Bijl A. Testing for Questionable Research Practices in a Meta-Analysis: an Example From Experimental Parapsychology. PLoS ONE. 2016;11(5):e0153049. PubMed PMID: 27144889.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology. AU - Bierman,Dick J, AU - Spottiswoode,James P, AU - Bijl,Aron, Y1 - 2016/05/04/ PY - 2015/09/15/received PY - 2016/03/20/accepted PY - 2016/5/5/entrez PY - 2016/5/6/pubmed PY - 2017/4/7/medline SP - e0153049 EP - e0153049 JF - PloS one JO - PLoS ONE VL - 11 IS - 5 N2 - We describe a method of quantifying the effect of Questionable Research Practices (QRPs) on the results of meta-analyses. As an example we simulated a meta-analysis of a controversial telepathy protocol to assess the extent to which these experimental results could be explained by QRPs. Our simulations used the same numbers of studies and trials as the original meta-analysis and the frequencies with which various QRPs were applied in the simulated experiments were based on surveys of experimental psychologists. Results of both the meta-analysis and simulations were characterized by 4 metrics, two describing the trial and mean experiment hit rates (HR) of around 31%, where 25% is expected by chance, one the correlation between sample-size and hit-rate, and one the complete P-value distribution of the database. A genetic algorithm optimized the parameters describing the QRPs, and the fitness of the simulated meta-analysis was defined as the sum of the squares of Z-scores for the 4 metrics. Assuming no anomalous effect a good fit to the empirical meta-analysis was found only by using QRPs with unrealistic parameter-values. Restricting the parameter space to ranges observed in studies of QRP occurrence, under the untested assumption that parapsychologists use comparable QRPs, the fit to the published Ganzfeld meta-analysis with no anomalous effect was poor. We allowed for a real anomalous effect, be it unidentified QRPs or a paranormal effect, where the HR ranged from 25% (chance) to 31%. With an anomalous HR of 27% the fitness became F = 1.8 (p = 0.47 where F = 0 is a perfect fit). We conclude that the very significant probability cited by the Ganzfeld meta-analysis is likely inflated by QRPs, though results are still significant (p = 0.003) with QRPs. Our study demonstrates that quantitative simulations of QRPs can assess their impact. Since meta-analyses in general might be polluted by QRPs, this method has wide applicability outside the domain of experimental parapsychology. SN - 1932-6203 UR - https://www.unboundmedicine.com/medline/citation/27144889/Testing_for_Questionable_Research_Practices_in_a_Meta_Analysis:_An_Example_from_Experimental_Parapsychology_ L2 - http://dx.plos.org/10.1371/journal.pone.0153049 DB - PRIME DP - Unbound Medicine ER -