Tags

Type your tag names separated by a space and hit enter

Interrater agreement in the evaluation of discrepant imaging findings with the Radpeer system.
AJR Am J Roentgenol. 2012 Dec; 199(6):1320-7.AA

Abstract

OBJECTIVE

The Radpeer system is central to the quality assurance process in many radiology practices. Previous studies have shown poor agreement between physicians in the evaluation of their peers. The purpose of this study was to assess the reliability of the Radpeer scoring system.

MATERIALS AND METHODS

A sample of 25 discrepant cases was extracted from our quality assurance database. Images were made anonymous; associated reports and identities of interpreting radiologists were removed. Indications for the studies and descriptions of the discrepancies were provided. Twenty-one subspecialist attending radiologists rated the cases using the Radpeer scoring system. Multirater kappa statistics were used to assess interrater agreement, both with the standard scoring system and with dichotomized scores to reflect the practice of further review for cases rated 3 and 4. Subgroup analyses were conducted to assess subspecialist evaluation of cases.

RESULTS

Interrater agreement was slight to fair compared with that expected by chance. For the group of 21 raters, the kappa values were 0.11 (95% CI, 0.06-0.16) with the standard scoring system and 0.20 (95% CI, 0.13-0.27) with dichotomized scores. There was disagreement about whether a discrepancy had occurred in 20 cases. Subgroup analyses did not reveal significant differences in the degree of interrater agreement.

CONCLUSION

The identification of discrepant interpretations is valuable for the education of individual radiologists and for larger-scale quality assurance and quality improvement efforts. Our results show that a ratings-based peer review system is unreliable and subjective for the evaluation of discrepant interpretations. Resources should be devoted to developing more robust and objective assessment procedures, particularly those with clear quality improvement goals.

Authors+Show Affiliations

Department of Radiology, University of Washington, Box 359728, 325 9th Ave, Seattle, WA 98104, USA.No affiliation info availableNo affiliation info availableNo affiliation info availableNo affiliation info available

Pub Type(s)

Journal Article

Language

eng

PubMed ID

23169725

Citation

Bender, Leila C., et al. "Interrater Agreement in the Evaluation of Discrepant Imaging Findings With the Radpeer System." AJR. American Journal of Roentgenology, vol. 199, no. 6, 2012, pp. 1320-7.
Bender LC, Linnau KF, Meier EN, et al. Interrater agreement in the evaluation of discrepant imaging findings with the Radpeer system. AJR Am J Roentgenol. 2012;199(6):1320-7.
Bender, L. C., Linnau, K. F., Meier, E. N., Anzai, Y., & Gunn, M. L. (2012). Interrater agreement in the evaluation of discrepant imaging findings with the Radpeer system. AJR. American Journal of Roentgenology, 199(6), 1320-7. https://doi.org/10.2214/AJR.12.8972
Bender LC, et al. Interrater Agreement in the Evaluation of Discrepant Imaging Findings With the Radpeer System. AJR Am J Roentgenol. 2012;199(6):1320-7. PubMed PMID: 23169725.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Interrater agreement in the evaluation of discrepant imaging findings with the Radpeer system. AU - Bender,Leila C, AU - Linnau,Ken F, AU - Meier,Eric N, AU - Anzai,Yoshimi, AU - Gunn,Martin L, PY - 2012/11/22/entrez PY - 2012/11/22/pubmed PY - 2013/2/21/medline SP - 1320 EP - 7 JF - AJR. American journal of roentgenology JO - AJR Am J Roentgenol VL - 199 IS - 6 N2 - OBJECTIVE: The Radpeer system is central to the quality assurance process in many radiology practices. Previous studies have shown poor agreement between physicians in the evaluation of their peers. The purpose of this study was to assess the reliability of the Radpeer scoring system. MATERIALS AND METHODS: A sample of 25 discrepant cases was extracted from our quality assurance database. Images were made anonymous; associated reports and identities of interpreting radiologists were removed. Indications for the studies and descriptions of the discrepancies were provided. Twenty-one subspecialist attending radiologists rated the cases using the Radpeer scoring system. Multirater kappa statistics were used to assess interrater agreement, both with the standard scoring system and with dichotomized scores to reflect the practice of further review for cases rated 3 and 4. Subgroup analyses were conducted to assess subspecialist evaluation of cases. RESULTS: Interrater agreement was slight to fair compared with that expected by chance. For the group of 21 raters, the kappa values were 0.11 (95% CI, 0.06-0.16) with the standard scoring system and 0.20 (95% CI, 0.13-0.27) with dichotomized scores. There was disagreement about whether a discrepancy had occurred in 20 cases. Subgroup analyses did not reveal significant differences in the degree of interrater agreement. CONCLUSION: The identification of discrepant interpretations is valuable for the education of individual radiologists and for larger-scale quality assurance and quality improvement efforts. Our results show that a ratings-based peer review system is unreliable and subjective for the evaluation of discrepant interpretations. Resources should be devoted to developing more robust and objective assessment procedures, particularly those with clear quality improvement goals. SN - 1546-3141 UR - https://www.unboundmedicine.com/medline/citation/23169725/Interrater_agreement_in_the_evaluation_of_discrepant_imaging_findings_with_the_Radpeer_system_ L2 - https://www.ajronline.org/doi/10.2214/AJR.12.8972 DB - PRIME DP - Unbound Medicine ER -