Tags

Type your tag names separated by a space and hit enter

Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy.
Eur Urol. 2016 Apr; 69(4):547-550.EU

Abstract

Because surgical skill may be a key determinant of patient outcomes, there is growing interest in skill assessment. In the Michigan Urological Surgery Improvement Collaborative (MUSIC), we assessed whether peer and crowd-sourced (ie, layperson) video review of robot-assisted radical prostatectomy (RARP) could distinguish technical skill among practicing surgeons. A total of 76 video clips from 12 MUSIC surgeons consisted of one of four parts of RARP and underwent blinded review by MUSIC peer surgeons and prequalified crowd-sourced reviewers. Videos were rated for global skill (Global Evaluation Assessment of Robotic Skills) and procedure-specific skill (Robotic Anastomosis and Competency Evaluation). We fit linear mixed-effects models to estimate mean peer and crowd ratings for each video. Individual video ratings were aggregated to calculate surgeon skill scores. Peers (n=25) completed 351 video ratings over 15 d, whereas crowd-sourced reviewers (n=680) completed 2990 video ratings in 38 h. Surgeon global skill scores ranged from 15.8 to 21.7 (peer) and from 19.2 to 20.9 (crowd). Peer and crowd ratings demonstrated strong correlation for both global (r=0.78) and anastomosis (r=0.74) skills. The two groups consistently agreed on the rank order of lower scoring surgeons, suggesting a potential role for crowd-sourced methodology in the assessment of surgical performance. Lack of patient outcomes is a limitation and forms the basis of future study.

PATIENT SUMMARY

We demonstrated the large-scale feasibility of assessing the technical skill of robotic surgeons and found that online crowd-sourced reviewers agreed with experts on the rank order of surgeons with the lowest technical skill scores.

Authors+Show Affiliations

Department of Urology, University of Michigan, Ann Arbor, MI, USA. Electronic address: kghani@med.umich.edu.Department of Urology, University of Michigan, Ann Arbor, MI, USA.Department of Urology, University of Michigan, Ann Arbor, MI, USA.Department of Urology, University of Michigan, Ann Arbor, MI, USA.Department of Urology, Spectrum Health, Grand Rapids, MI, USA.Michigan Institute of Urology, Dearborn, MI, USA.Vattikuti Urology Institute, Henry Ford Health System, Detroit, MI, USA.Vattikuti Urology Institute, Henry Ford Health System, Detroit, MI, USA.Department of Biostatistics, University of Washington, Seattle, WA, USA.Department of Urology, University of Washington, Seattle, WA, USA.Department of Urology, University of Michigan, Ann Arbor, MI, USA.Vattikuti Urology Institute, Henry Ford Health System, Detroit, MI, USA.No affiliation info available

Pub Type(s)

Letter
Research Support, Non-U.S. Gov't

Language

eng

PubMed ID

26755338

Citation

Ghani, Khurshid R., et al. "Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill With Robot-assisted Radical Prostatectomy." European Urology, vol. 69, no. 4, 2016, pp. 547-550.
Ghani KR, Miller DC, Linsell S, et al. Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy. Eur Urol. 2016;69(4):547-550.
Ghani, K. R., Miller, D. C., Linsell, S., Brachulis, A., Lane, B., Sarle, R., Dalela, D., Menon, M., Comstock, B., Lendvay, T. S., Montie, J., & Peabody, J. O. (2016). Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy. European Urology, 69(4), 547-550. https://doi.org/10.1016/j.eururo.2015.11.028
Ghani KR, et al. Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill With Robot-assisted Radical Prostatectomy. Eur Urol. 2016;69(4):547-550. PubMed PMID: 26755338.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy. AU - Ghani,Khurshid R, AU - Miller,David C, AU - Linsell,Susan, AU - Brachulis,Andrew, AU - Lane,Brian, AU - Sarle,Richard, AU - Dalela,Deepansh, AU - Menon,Mani, AU - Comstock,Bryan, AU - Lendvay,Thomas S, AU - Montie,James, AU - Peabody,James O, AU - ,, Y1 - 2016/01/02/ PY - 2015/09/09/received PY - 2015/11/23/accepted PY - 2016/1/13/entrez PY - 2016/1/13/pubmed PY - 2016/12/15/medline KW - Crowd sourcing KW - Radical prostatectomy KW - Robotic surgery KW - Skills assessment SP - 547 EP - 550 JF - European urology JO - Eur. Urol. VL - 69 IS - 4 N2 - UNLABELLED: Because surgical skill may be a key determinant of patient outcomes, there is growing interest in skill assessment. In the Michigan Urological Surgery Improvement Collaborative (MUSIC), we assessed whether peer and crowd-sourced (ie, layperson) video review of robot-assisted radical prostatectomy (RARP) could distinguish technical skill among practicing surgeons. A total of 76 video clips from 12 MUSIC surgeons consisted of one of four parts of RARP and underwent blinded review by MUSIC peer surgeons and prequalified crowd-sourced reviewers. Videos were rated for global skill (Global Evaluation Assessment of Robotic Skills) and procedure-specific skill (Robotic Anastomosis and Competency Evaluation). We fit linear mixed-effects models to estimate mean peer and crowd ratings for each video. Individual video ratings were aggregated to calculate surgeon skill scores. Peers (n=25) completed 351 video ratings over 15 d, whereas crowd-sourced reviewers (n=680) completed 2990 video ratings in 38 h. Surgeon global skill scores ranged from 15.8 to 21.7 (peer) and from 19.2 to 20.9 (crowd). Peer and crowd ratings demonstrated strong correlation for both global (r=0.78) and anastomosis (r=0.74) skills. The two groups consistently agreed on the rank order of lower scoring surgeons, suggesting a potential role for crowd-sourced methodology in the assessment of surgical performance. Lack of patient outcomes is a limitation and forms the basis of future study. PATIENT SUMMARY: We demonstrated the large-scale feasibility of assessing the technical skill of robotic surgeons and found that online crowd-sourced reviewers agreed with experts on the rank order of surgeons with the lowest technical skill scores. SN - 1873-7560 UR - https://www.unboundmedicine.com/medline/citation/26755338/Measuring_to_Improve:_Peer_and_Crowd_sourced_Assessments_of_Technical_Skill_with_Robot_assisted_Radical_Prostatectomy_ L2 - https://linkinghub.elsevier.com/retrieve/pii/S0302-2838(15)01189-6 DB - PRIME DP - Unbound Medicine ER -