Tags

Type your tag names separated by a space and hit enter

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation.
J Med Internet Res. 2020 11 06; 22(11):e24018.JM

Abstract

BACKGROUND

COVID-19 has infected millions of people worldwide and is responsible for several hundred thousand fatalities. The COVID-19 pandemic has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods to meet these needs are lacking.

OBJECTIVE

The aims of this study were to analyze the electronic health records (EHRs) of patients who tested positive for COVID-19 and were admitted to hospitals in the Mount Sinai Health System in New York City; to develop machine learning models for making predictions about the hospital course of the patients over clinically meaningful time horizons based on patient characteristics at admission; and to assess the performance of these models at multiple hospitals and time points.

METHODS

We used Extreme Gradient Boosting (XGBoost) and baseline comparator models to predict in-hospital mortality and critical events at time windows of 3, 5, 7, and 10 days from admission. Our study population included harmonized EHR data from five hospitals in New York City for 4098 COVID-19-positive patients admitted from March 15 to May 22, 2020. The models were first trained on patients from a single hospital (n=1514) before or on May 1, externally validated on patients from four other hospitals (n=2201) before or on May 1, and prospectively validated on all patients after May 1 (n=383). Finally, we established model interpretability to identify and rank variables that drive model predictions.

RESULTS

Upon cross-validation, the XGBoost classifier outperformed baseline models, with an area under the receiver operating characteristic curve (AUC-ROC) for mortality of 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days. XGBoost also performed well for critical event prediction, with an AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, the unimputed XGBoost model achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers of mortality prediction.

CONCLUSIONS

We externally and prospectively trained and validated machine learning models for mortality and critical events for patients with COVID-19 at different time horizons. These models identified at-risk patients and uncovered underlying relationships that predicted outcomes.

Authors+Show Affiliations

The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Anesthesiology, Perioperative and Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Zena and Michael A. Wiener Cardiovascular Institute, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Harvard Medical School, Boston, MA, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Pamela Sklar Division of Psychiatric Genomics, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Pamela Sklar Division of Psychiatric Genomics, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Mount Sinai Data Warehouse, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Mount Sinai Heart, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Cardiology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Icahn Institute for Data Science and Genomic Technology, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Mount Sinai Data Office, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Division of Infectious Diseases, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Zena and Michael A. Wiener Cardiovascular Institute, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Icahn Institute for Data Science and Genomic Technology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Charles Bronfman Institute for Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Pathology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Zena and Michael A. Wiener Cardiovascular Institute, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Mount Sinai Heart, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Cardiology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Office of the Dean, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Anesthesiology, Perioperative and Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Digital Health Center, Hasso Plattner Institute, University of Potsdam, Potsdam, Germany.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Anesthesiology, Perioperative and Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Mount Sinai Heart, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Cardiology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Environmental Medicine and Public Health, Icahn School of Medicine at Mount Sinai, New York, NY, United States.Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Pamela Sklar Division of Psychiatric Genomics, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States. The Charles Bronfman Institute for Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.The Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, NY, United States. Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, United States.

Pub Type(s)

Journal Article
Research Support, N.I.H., Extramural
Validation Study

Language

eng

PubMed ID

33027032

Citation

Vaid, Akhil, et al. "Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation." Journal of Medical Internet Research, vol. 22, no. 11, 2020, pp. e24018.
Vaid A, Somani S, Russak AJ, et al. Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation. J Med Internet Res. 2020;22(11):e24018.
Vaid, A., Somani, S., Russak, A. J., De Freitas, J. K., Chaudhry, F. F., Paranjpe, I., Johnson, K. W., Lee, S. J., Miotto, R., Richter, F., Zhao, S., Beckmann, N. D., Naik, N., Kia, A., Timsina, P., Lala, A., Paranjpe, M., Golden, E., Danieletto, M., ... Glicksberg, B. S. (2020). Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation. Journal of Medical Internet Research, 22(11), e24018. https://doi.org/10.2196/24018
Vaid A, et al. Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation. J Med Internet Res. 2020 11 6;22(11):e24018. PubMed PMID: 33027032.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation. AU - Vaid,Akhil, AU - Somani,Sulaiman, AU - Russak,Adam J, AU - De Freitas,Jessica K, AU - Chaudhry,Fayzan F, AU - Paranjpe,Ishan, AU - Johnson,Kipp W, AU - Lee,Samuel J, AU - Miotto,Riccardo, AU - Richter,Felix, AU - Zhao,Shan, AU - Beckmann,Noam D, AU - Naik,Nidhi, AU - Kia,Arash, AU - Timsina,Prem, AU - Lala,Anuradha, AU - Paranjpe,Manish, AU - Golden,Eddye, AU - Danieletto,Matteo, AU - Singh,Manbir, AU - Meyer,Dara, AU - O'Reilly,Paul F, AU - Huckins,Laura, AU - Kovatch,Patricia, AU - Finkelstein,Joseph, AU - Freeman,Robert M, AU - Argulian,Edgar, AU - Kasarskis,Andrew, AU - Percha,Bethany, AU - Aberg,Judith A, AU - Bagiella,Emilia, AU - Horowitz,Carol R, AU - Murphy,Barbara, AU - Nestler,Eric J, AU - Schadt,Eric E, AU - Cho,Judy H, AU - Cordon-Cardo,Carlos, AU - Fuster,Valentin, AU - Charney,Dennis S, AU - Reich,David L, AU - Bottinger,Erwin P, AU - Levin,Matthew A, AU - Narula,Jagat, AU - Fayad,Zahi A, AU - Just,Allan C, AU - Charney,Alexander W, AU - Nadkarni,Girish N, AU - Glicksberg,Benjamin S, Y1 - 2020/11/06/ PY - 2020/09/01/received PY - 2020/10/02/accepted PY - 2020/10/02/revised PY - 2020/10/8/pubmed PY - 2020/11/25/medline PY - 2020/10/7/entrez KW - COVID-19 KW - EHR KW - TRIPOD KW - clinical informatics KW - cohort KW - electronic health record KW - hospital KW - machine learning KW - mortality KW - performance KW - prediction SP - e24018 EP - e24018 JF - Journal of medical Internet research JO - J Med Internet Res VL - 22 IS - 11 N2 - BACKGROUND: COVID-19 has infected millions of people worldwide and is responsible for several hundred thousand fatalities. The COVID-19 pandemic has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods to meet these needs are lacking. OBJECTIVE: The aims of this study were to analyze the electronic health records (EHRs) of patients who tested positive for COVID-19 and were admitted to hospitals in the Mount Sinai Health System in New York City; to develop machine learning models for making predictions about the hospital course of the patients over clinically meaningful time horizons based on patient characteristics at admission; and to assess the performance of these models at multiple hospitals and time points. METHODS: We used Extreme Gradient Boosting (XGBoost) and baseline comparator models to predict in-hospital mortality and critical events at time windows of 3, 5, 7, and 10 days from admission. Our study population included harmonized EHR data from five hospitals in New York City for 4098 COVID-19-positive patients admitted from March 15 to May 22, 2020. The models were first trained on patients from a single hospital (n=1514) before or on May 1, externally validated on patients from four other hospitals (n=2201) before or on May 1, and prospectively validated on all patients after May 1 (n=383). Finally, we established model interpretability to identify and rank variables that drive model predictions. RESULTS: Upon cross-validation, the XGBoost classifier outperformed baseline models, with an area under the receiver operating characteristic curve (AUC-ROC) for mortality of 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days. XGBoost also performed well for critical event prediction, with an AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, the unimputed XGBoost model achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers of mortality prediction. CONCLUSIONS: We externally and prospectively trained and validated machine learning models for mortality and critical events for patients with COVID-19 at different time horizons. These models identified at-risk patients and uncovered underlying relationships that predicted outcomes. SN - 1438-8871 UR - https://www.unboundmedicine.com/medline/citation/33027032/Machine_Learning_to_Predict_Mortality_and_Critical_Events_in_a_Cohort_of_Patients_With_COVID_19_in_New_York_City:_Model_Development_and_Validation_ L2 - https://www.jmir.org/2020/11/e24018/ DB - PRIME DP - Unbound Medicine ER -