Tags

Type your tag names separated by a space and hit enter

Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT.
Radiology. 2020 09; 296(3):E156-E165.R

Abstract

Background Coronavirus disease 2019 (COVID-19) and pneumonia of other diseases share similar CT characteristics, which contributes to the challenges in differentiating them with high accuracy. Purpose To establish and evaluate an artificial intelligence (AI) system for differentiating COVID-19 and other pneumonia at chest CT and assessing radiologist performance without and with AI assistance. Materials and Methods A total of 521 patients with positive reverse transcription polymerase chain reaction results for COVID-19 and abnormal chest CT findings were retrospectively identified from 10 hospitals from January 2020 to April 2020. A total of 665 patients with non-COVID-19 pneumonia and definite evidence of pneumonia at chest CT were retrospectively selected from three hospitals between 2017 and 2019. To classify COVID-19 versus other pneumonia for each patient, abnormal CT slices were input into the EfficientNet B4 deep neural network architecture after lung segmentation, followed by a two-layer fully connected neural network to pool slices together. The final cohort of 1186 patients (132 583 CT slices) was divided into training, validation, and test sets in a 7:2:1 and equal ratio. Independent testing was performed by evaluating model performance in separate hospitals. Studies were blindly reviewed by six radiologists without and then with AI assistance. Results The final model achieved a test accuracy of 96% (95% confidence interval [CI]: 90%, 98%), a sensitivity of 95% (95% CI: 83%, 100%), and a specificity of 96% (95% CI: 88%, 99%) with area under the receiver operating characteristic curve of 0.95 and area under the precision-recall curve of 0.90. On independent testing, this model achieved an accuracy of 87% (95% CI: 82%, 90%), a sensitivity of 89% (95% CI: 81%, 94%), and a specificity of 86% (95% CI: 80%, 90%) with area under the receiver operating characteristic curve of 0.90 and area under the precision-recall curve of 0.87. Assisted by the probabilities of the model, the radiologists achieved a higher average test accuracy (90% vs 85%, Δ = 5, P < .001), sensitivity (88% vs 79%, Δ = 9, P < .001), and specificity (91% vs 88%, Δ = 3, P = .001). Conclusion Artificial intelligence assistance improved radiologists' performance in distinguishing coronavirus disease 2019 pneumonia from non-coronavirus disease 2019 pneumonia at chest CT. © RSNA, 2020 Online supplemental material is available for this article.

Authors+Show Affiliations

From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).From the Department of Radiology, Xiangya Hospital, Central South University, Changsha 410008, China (H.X.B., Z.X., D.C.W., W.H.L.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (H.X.B., B.H., K.H., I.P., M.K.A.); Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa (R.W.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology. Massachusetts General Hospital, Boston, Mass (K.C.); Warren Alpert Medical School at Brown University, Providence, RI (H.X.B., K.H., T.M.L.T., J.W.C., I.P.); Department of Radiology, Yongzhou Central Hospital, Yongzhou, China (L.B.S.); Department of Radiology, Changde Second People's Hospital, Changde, China (J.M.); Department of Radiology, Affiliated Nan Hua Hospital, University of South China, Hengyang, China (X.L.J.); Department of Radiology, Loudi Central Hospital, Loudi, China (Q.H.Z.); Department of Radiology, Chenzhou Second People's Hospital, Chenzhou, China (P.F.H.); Department of Radiology, Zhuzhou Central Hospital, Zhuzhou, China (Y.H.L.); Department of Radiology, Yiyang City Center Hospital, Yiyang, China (F.X.F.); Department of Radiology, Brigham and Women's Hospital, Boston, Mass (R.Y.H.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (R.S.); and Department of Radiology, The First Hospital of Changsha, Changsha, China (Q.Z.Y.).

Pub Type(s)

Journal Article

Language

eng

PubMed ID

32339081

Citation

Bai, Harrison X., et al. "Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 From Pneumonia of Other Origin at Chest CT." Radiology, vol. 296, no. 3, 2020, pp. E156-E165.
Bai HX, Wang R, Xiong Z, et al. Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT. Radiology. 2020;296(3):E156-E165.
Bai, H. X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., Tran, T. M. L., Choi, J. W., Wang, D. C., Shi, L. B., Mei, J., Jiang, X. L., Pan, I., Zeng, Q. H., Hu, P. F., Li, Y. H., Fu, F. X., Huang, R. Y., Sebro, R., ... Liao, W. H. (2020). Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT. Radiology, 296(3), E156-E165. https://doi.org/10.1148/radiol.2020201491
Bai HX, et al. Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 From Pneumonia of Other Origin at Chest CT. Radiology. 2020;296(3):E156-E165. PubMed PMID: 32339081.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT. AU - Bai,Harrison X, AU - Wang,Robin, AU - Xiong,Zeng, AU - Hsieh,Ben, AU - Chang,Ken, AU - Halsey,Kasey, AU - Tran,Thi My Linh, AU - Choi,Ji Whae, AU - Wang,Dong-Cui, AU - Shi,Lin-Bo, AU - Mei,Ji, AU - Jiang,Xiao-Long, AU - Pan,Ian, AU - Zeng,Qiu-Hua, AU - Hu,Ping-Feng, AU - Li,Yi-Hui, AU - Fu,Fei-Xian, AU - Huang,Raymond Y, AU - Sebro,Ronnie, AU - Yu,Qi-Zhi, AU - Atalay,Michael K, AU - Liao,Wei-Hua, Y1 - 2020/04/27/ PY - 2020/4/28/pubmed PY - 2020/8/28/medline PY - 2020/4/28/entrez SP - E156 EP - E165 JF - Radiology JO - Radiology VL - 296 IS - 3 N2 - Background Coronavirus disease 2019 (COVID-19) and pneumonia of other diseases share similar CT characteristics, which contributes to the challenges in differentiating them with high accuracy. Purpose To establish and evaluate an artificial intelligence (AI) system for differentiating COVID-19 and other pneumonia at chest CT and assessing radiologist performance without and with AI assistance. Materials and Methods A total of 521 patients with positive reverse transcription polymerase chain reaction results for COVID-19 and abnormal chest CT findings were retrospectively identified from 10 hospitals from January 2020 to April 2020. A total of 665 patients with non-COVID-19 pneumonia and definite evidence of pneumonia at chest CT were retrospectively selected from three hospitals between 2017 and 2019. To classify COVID-19 versus other pneumonia for each patient, abnormal CT slices were input into the EfficientNet B4 deep neural network architecture after lung segmentation, followed by a two-layer fully connected neural network to pool slices together. The final cohort of 1186 patients (132 583 CT slices) was divided into training, validation, and test sets in a 7:2:1 and equal ratio. Independent testing was performed by evaluating model performance in separate hospitals. Studies were blindly reviewed by six radiologists without and then with AI assistance. Results The final model achieved a test accuracy of 96% (95% confidence interval [CI]: 90%, 98%), a sensitivity of 95% (95% CI: 83%, 100%), and a specificity of 96% (95% CI: 88%, 99%) with area under the receiver operating characteristic curve of 0.95 and area under the precision-recall curve of 0.90. On independent testing, this model achieved an accuracy of 87% (95% CI: 82%, 90%), a sensitivity of 89% (95% CI: 81%, 94%), and a specificity of 86% (95% CI: 80%, 90%) with area under the receiver operating characteristic curve of 0.90 and area under the precision-recall curve of 0.87. Assisted by the probabilities of the model, the radiologists achieved a higher average test accuracy (90% vs 85%, Δ = 5, P < .001), sensitivity (88% vs 79%, Δ = 9, P < .001), and specificity (91% vs 88%, Δ = 3, P = .001). Conclusion Artificial intelligence assistance improved radiologists' performance in distinguishing coronavirus disease 2019 pneumonia from non-coronavirus disease 2019 pneumonia at chest CT. © RSNA, 2020 Online supplemental material is available for this article. SN - 1527-1315 UR - https://www.unboundmedicine.com/medline/citation/32339081/Artificial_Intelligence_Augmentation_of_Radiologist_Performance_in_Distinguishing_COVID_19_from_Pneumonia_of_Other_Origin_at_Chest_CT_ L2 - https://pubs.rsna.org/doi/10.1148/radiol.2020201491?url_ver=Z39.88-2003&amp;rfr_id=ori:rid:crossref.org&amp;rfr_dat=cr_pub=pubmed DB - PRIME DP - Unbound Medicine ER -