Tags

Type your tag names separated by a space and hit enter

Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study.
BMJ Evid Based Med. 2021 Jun 03 [Online ahead of print]BE

Abstract

OBJECTIVE

The academic and scientific community has reacted at pace to gather evidence to help and inform about COVID-19. Concerns have been raised about the quality of this evidence. The aim of this review was to map the nature, scope and quality of evidence syntheses on COVID-19 and to explore the relationship between review quality and the extent of researcher, policy and media interest.

DESIGN AND SETTING

A meta-research: systematic review of reviews.

INFORMATION SOURCES

PubMed, Epistemonikos COVID-19 evidence, the Cochrane Library of Systematic Reviews, the Cochrane COVID-19 Study Register, EMBASE, CINAHL, Web of Science Core Collection and the WHO COVID-19 database, searched between 10 June 2020 and 15 June 2020.

ELIGIBILITY CRITERIA

Any peer-reviewed article reported as a systematic review, rapid review, overview, meta-analysis or qualitative evidence synthesis in the title or abstract addressing a research question relating to COVID-19. Articles described as meta-analyses but not undertaken as part of a systematic or rapid review were excluded.

STUDY SELECTION AND DATA EXTRACTION

Abstract and full text screening were undertaken by two independent reviewers. Descriptive information on review type, purpose, population, size, citation and attention metrics were extracted along with whether the review met the definition of a systematic review according to six key methodological criteria. For those meeting all criteria, additional data on methods and publication metrics were extracted.

RISK OF BIAS

For articles meeting all six criteria required to meet the definition of a systematic review, AMSTAR-2 ((A MeaSurement Tool to Assess systematic Reviews, version 2.0) was used to assess the quality of the reported methods.

RESULTS

2334 articles were screened, resulting in 280 reviews being included: 232 systematic reviews, 46 rapid reviews and 2 overviews. Less than half reported undertaking critical appraisal and a third had no reproducible search strategy. There was considerable overlap in topics, with discordant findings. Eighty-eight of the 280 reviews met all six systematic review criteria. Of these, just 3 were rated as of moderate or high quality on AMSTAR-2, with the majority having critical flaws: only a third reported registering a protocol, and less than one in five searched named COVID-19 databases. Review conduct and publication were rapid, with 52 of the 88 systematic reviews reported as being conducted within 3 weeks, and a half published within 3 weeks of submission. Researcher and media interest, as measured by altmetrics and citations, was high, and was not correlated with quality.

DISCUSSION

This meta-research of early published COVID-19 evidence syntheses found low-quality reviews being published at pace, often with short publication turnarounds. Despite being of low quality and many lacking robust methods, the reviews received substantial attention across both academic and public platforms, and the attention was not related to the quality of review methods.

INTERPRETATION

Flaws in systematic review methods limit the validity of a review and the generalisability of its findings. Yet, by being reported as 'systematic reviews', many readers may well regard them as high-quality evidence, irrespective of the actual methods undertaken. The challenge especially in times such as this pandemic is to provide indications of trustworthiness in evidence that is available in 'real time'.

PROSPERO REGISTRATION NUMBER

CRD42020188822.

Authors+Show Affiliations

NIHR ARC South West Peninsula, University of Exeter Medical School, University of Exeter, Exeter, UK r.a.abbott@exeter.ac.uk.NIHR ARC South West Peninsula, University of Exeter Medical School, University of Exeter, Exeter, UK.NIHR ARC South West Peninsula, University of Exeter Medical School, University of Exeter, Exeter, UK.NIHR ARC South West Peninsula, University of Exeter Medical School, University of Exeter, Exeter, UK.NIHR ARC South West Peninsula, University of Exeter Medical School, University of Exeter, Exeter, UK.University of Exeter Medical School, University of Exeter, Exeter, UK.NIHR ARC South West Peninsula, University of Exeter Medical School, University of Exeter, Exeter, UK.NIHR ARC South West Peninsula, University of Exeter Medical School, University of Exeter, Exeter, UK.

Pub Type(s)

Journal Article
Review

Language

eng

PubMed ID

34083212

Citation

Abbott, Rebecca, et al. "Characteristics, Quality and Volume of the First 5 Months of the COVID-19 Evidence Synthesis Infodemic: a Meta-research Study." BMJ Evidence-based Medicine, 2021.
Abbott R, Bethel A, Rogers M, et al. Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. BMJ Evid Based Med. 2021.
Abbott, R., Bethel, A., Rogers, M., Whear, R., Orr, N., Shaw, L., Stein, K., & Thompson Coon, J. (2021). Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. BMJ Evidence-based Medicine. https://doi.org/10.1136/bmjebm-2021-111710
Abbott R, et al. Characteristics, Quality and Volume of the First 5 Months of the COVID-19 Evidence Synthesis Infodemic: a Meta-research Study. BMJ Evid Based Med. 2021 Jun 3; PubMed PMID: 34083212.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. AU - Abbott,Rebecca, AU - Bethel,Alison, AU - Rogers,Morwenna, AU - Whear,Rebecca, AU - Orr,Noreen, AU - Shaw,Liz, AU - Stein,Ken, AU - Thompson Coon,Jo, Y1 - 2021/06/03/ PY - 2021/03/24/received PY - 2021/05/24/accepted PY - 2021/6/4/entrez PY - 2021/6/5/pubmed PY - 2021/6/5/medline KW - COVID-19 KW - evidence-based practice KW - public health JF - BMJ evidence-based medicine JO - BMJ Evid Based Med N2 - OBJECTIVE: The academic and scientific community has reacted at pace to gather evidence to help and inform about COVID-19. Concerns have been raised about the quality of this evidence. The aim of this review was to map the nature, scope and quality of evidence syntheses on COVID-19 and to explore the relationship between review quality and the extent of researcher, policy and media interest. DESIGN AND SETTING: A meta-research: systematic review of reviews. INFORMATION SOURCES: PubMed, Epistemonikos COVID-19 evidence, the Cochrane Library of Systematic Reviews, the Cochrane COVID-19 Study Register, EMBASE, CINAHL, Web of Science Core Collection and the WHO COVID-19 database, searched between 10 June 2020 and 15 June 2020. ELIGIBILITY CRITERIA: Any peer-reviewed article reported as a systematic review, rapid review, overview, meta-analysis or qualitative evidence synthesis in the title or abstract addressing a research question relating to COVID-19. Articles described as meta-analyses but not undertaken as part of a systematic or rapid review were excluded. STUDY SELECTION AND DATA EXTRACTION: Abstract and full text screening were undertaken by two independent reviewers. Descriptive information on review type, purpose, population, size, citation and attention metrics were extracted along with whether the review met the definition of a systematic review according to six key methodological criteria. For those meeting all criteria, additional data on methods and publication metrics were extracted. RISK OF BIAS: For articles meeting all six criteria required to meet the definition of a systematic review, AMSTAR-2 ((A MeaSurement Tool to Assess systematic Reviews, version 2.0) was used to assess the quality of the reported methods. RESULTS: 2334 articles were screened, resulting in 280 reviews being included: 232 systematic reviews, 46 rapid reviews and 2 overviews. Less than half reported undertaking critical appraisal and a third had no reproducible search strategy. There was considerable overlap in topics, with discordant findings. Eighty-eight of the 280 reviews met all six systematic review criteria. Of these, just 3 were rated as of moderate or high quality on AMSTAR-2, with the majority having critical flaws: only a third reported registering a protocol, and less than one in five searched named COVID-19 databases. Review conduct and publication were rapid, with 52 of the 88 systematic reviews reported as being conducted within 3 weeks, and a half published within 3 weeks of submission. Researcher and media interest, as measured by altmetrics and citations, was high, and was not correlated with quality. DISCUSSION: This meta-research of early published COVID-19 evidence syntheses found low-quality reviews being published at pace, often with short publication turnarounds. Despite being of low quality and many lacking robust methods, the reviews received substantial attention across both academic and public platforms, and the attention was not related to the quality of review methods. INTERPRETATION: Flaws in systematic review methods limit the validity of a review and the generalisability of its findings. Yet, by being reported as 'systematic reviews', many readers may well regard them as high-quality evidence, irrespective of the actual methods undertaken. The challenge especially in times such as this pandemic is to provide indications of trustworthiness in evidence that is available in 'real time'. PROSPERO REGISTRATION NUMBER: CRD42020188822. SN - 2515-4478 UR - https://www.unboundmedicine.com/medline/citation/34083212/Characteristics_quality_and_volume_of_the_first_5_months_of_the_COVID_19_evidence_synthesis_infodemic:_a_meta_research_study_ L2 - https://ebm.bmj.com/lookup/pmidlookup?view=long&pmid=34083212 DB - PRIME DP - Unbound Medicine ER -
Try the Free App:
Prime PubMed app for iOS iPhone iPad
Prime PubMed app for Android
Prime PubMed is provided
free to individuals by:
Unbound Medicine.