Tags

Type your tag names separated by a space and hit enter

Construct validity of the endoscopic sinus surgery simulator: II. Assessment of discriminant validity and expert benchmarking.
Arch Otolaryngol Head Neck Surg. 2007 Apr; 133(4):350-7.AO

Abstract

OBJECTIVES

To establish discriminant validity of the endoscopic sinus surgery simulator (ES3) (Lockheed Martin, Akron, Ohio) between various health care provider experience levels and to define benchmarking criteria for skills assessment.

DESIGN

Prospective multi-institutional comparison study.

SETTING

University-based tertiary care institution.

PARTICIPANTS

Ten expert otolaryngologists, 14 otolaryngology residents, and 10 medical students.

INTERVENTIONS

Subjects completed the ES3's virtual reality curriculum (10 novice mode, 10 intermediate mode, and 3 advanced mode trials). Performance scores were recorded on each trial. Performance differences were analyzed using analysis of variance for repeated measures (experience level as between-subjects factor).

MAIN OUTCOME MEASURES

Simulator performance scores, accuracy, time to completion, and hazard disruption.

RESULTS

The novice mode accurately distinguished the 3 groups, particularly at the onset of training (mean scores: senior otolaryngologists, 66.0; residents, 42.7; students, 18.3; for the paired comparisons between groups 1 and 2 and groups 1 and 3, P = .04 and .03, respectively). Subjects were not distinguished beyond trial 5. The intermediate mode only discriminated students from other subjects (P = .008). The advanced mode did not show performance differences between groups. Scores on the novice mode predicted those on the intermediate mode, which predicted advanced mode scores (r = 0.687), but no relationship was found between novice and advanced scores. All groups performed equally well and with comparable consistency at the outset of training. Expert scores were used to define benchmark criteria of optimal performance.

CONCLUSIONS

This study completes the construct validity assessment of the ES3 by demonstrating its discriminant capabilities. It establishes expert surgeon benchmark performance criteria and shows that the ES3 can train novice subjects to attain those. The refined analysis of trial performance scores could serve educational and skills assessment purposes. Current studies are evaluating the transfer of surgical skills acquired on the ES3 to the operating room (predictive validity).

Links

Publisher Full Text
Aggregator Full Text

Authors+Show Affiliations

Fried MP
Department of Otorhinolaryngology-Head and Neck Surgery, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY 10467, USA. mfried@montefiore.org
Sadoughi B
No affiliation info available
Weghorst SJ
No affiliation info available
Zeltsan M
No affiliation info available
Cuellar H
No affiliation info available
Uribe JI
No affiliation info available
Sasaki CT
No affiliation info available
Ross DA
No affiliation info available
Jacobs JB
No affiliation info available
Lebowitz RA
No affiliation info available
Satava RM
No affiliation info available

MeSH

Analysis of VarianceBenchmarkingClinical CompetenceComputer SimulationComputer-Assisted InstructionEducational MeasurementEducational TechnologyEndoscopyHumansParanasal Sinus DiseasesProspective StudiesUser-Computer Interface

Pub Type(s)

Comparative Study
Journal Article
Multicenter Study
Research Support, U.S. Gov't, P.H.S.

Language

eng

PubMed ID

17438249

Citation

Fried, Marvin P., et al. "Construct Validity of the Endoscopic Sinus Surgery Simulator: II. Assessment of Discriminant Validity and Expert Benchmarking." Archives of Otolaryngology--head & Neck Surgery, vol. 133, no. 4, 2007, pp. 350-7.
Fried MP, Sadoughi B, Weghorst SJ, et al. Construct validity of the endoscopic sinus surgery simulator: II. Assessment of discriminant validity and expert benchmarking. Arch Otolaryngol Head Neck Surg. 2007;133(4):350-7.
Fried, M. P., Sadoughi, B., Weghorst, S. J., Zeltsan, M., Cuellar, H., Uribe, J. I., Sasaki, C. T., Ross, D. A., Jacobs, J. B., Lebowitz, R. A., & Satava, R. M. (2007). Construct validity of the endoscopic sinus surgery simulator: II. Assessment of discriminant validity and expert benchmarking. Archives of Otolaryngology--head & Neck Surgery, 133(4), 350-7.
Fried MP, et al. Construct Validity of the Endoscopic Sinus Surgery Simulator: II. Assessment of Discriminant Validity and Expert Benchmarking. Arch Otolaryngol Head Neck Surg. 2007;133(4):350-7. PubMed PMID: 17438249.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Construct validity of the endoscopic sinus surgery simulator: II. Assessment of discriminant validity and expert benchmarking. AU - Fried,Marvin P, AU - Sadoughi,Babak, AU - Weghorst,Suzanne J, AU - Zeltsan,Michael, AU - Cuellar,Hernando, AU - Uribe,José I, AU - Sasaki,Clarence T, AU - Ross,Douglas A, AU - Jacobs,Joseph B, AU - Lebowitz,Richard A, AU - Satava,Richard M, PY - 2007/4/18/pubmed PY - 2007/5/16/medline PY - 2007/4/18/entrez SP - 350 EP - 7 JF - Archives of otolaryngology--head & neck surgery JO - Arch Otolaryngol Head Neck Surg VL - 133 IS - 4 N2 - OBJECTIVES: To establish discriminant validity of the endoscopic sinus surgery simulator (ES3) (Lockheed Martin, Akron, Ohio) between various health care provider experience levels and to define benchmarking criteria for skills assessment. DESIGN: Prospective multi-institutional comparison study. SETTING: University-based tertiary care institution. PARTICIPANTS: Ten expert otolaryngologists, 14 otolaryngology residents, and 10 medical students. INTERVENTIONS: Subjects completed the ES3's virtual reality curriculum (10 novice mode, 10 intermediate mode, and 3 advanced mode trials). Performance scores were recorded on each trial. Performance differences were analyzed using analysis of variance for repeated measures (experience level as between-subjects factor). MAIN OUTCOME MEASURES: Simulator performance scores, accuracy, time to completion, and hazard disruption. RESULTS: The novice mode accurately distinguished the 3 groups, particularly at the onset of training (mean scores: senior otolaryngologists, 66.0; residents, 42.7; students, 18.3; for the paired comparisons between groups 1 and 2 and groups 1 and 3, P = .04 and .03, respectively). Subjects were not distinguished beyond trial 5. The intermediate mode only discriminated students from other subjects (P = .008). The advanced mode did not show performance differences between groups. Scores on the novice mode predicted those on the intermediate mode, which predicted advanced mode scores (r = 0.687), but no relationship was found between novice and advanced scores. All groups performed equally well and with comparable consistency at the outset of training. Expert scores were used to define benchmark criteria of optimal performance. CONCLUSIONS: This study completes the construct validity assessment of the ES3 by demonstrating its discriminant capabilities. It establishes expert surgeon benchmark performance criteria and shows that the ES3 can train novice subjects to attain those. The refined analysis of trial performance scores could serve educational and skills assessment purposes. Current studies are evaluating the transfer of surgical skills acquired on the ES3 to the operating room (predictive validity). SN - 0886-4470 UR - https://www.unboundmedicine.com/medline/citation/17438249/Construct_validity_of_the_endoscopic_sinus_surgery_simulator:_II__Assessment_of_discriminant_validity_and_expert_benchmarking_ L2 - https://jamanetwork.com/journals/jamaotolaryngology/fullarticle/10.1001/archotol.133.4.350 DB - PRIME DP - Unbound Medicine ER -