Tags

Type your tag names separated by a space and hit enter

Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery.
Am J Rhinol Allergy. 2009 May-Jun; 23(3):354-9.AJ

Abstract

BACKGROUND

Otolaryngology residency programs are required by the Accreditation Council for Graduate Medical Education to evaluate residents' operative competency. Many such tools based on the model of objective structured assessment of technical skills (OSATSs) have been developed in other surgical specialties, but no such instruments exist for otolaryngologic procedures except for tonsillectomy. Endoscopic sinus surgery (ESS) is among the most common rhinologic procedure and lends itself to objective evaluation of operative competency. The purpose of this study was to develop and test a new tool for ESS, focusing on feasibility, content and construct validity, and interrater agreement that can be used for such assessment in the operating room and the cadaver dissection course.

METHODS

Faculty input via the modified Delphi technique helped develop the content of a new OSATS-based instrument. The instrument underwent serial improvements based on 3 years of endoscopic sinus surgery (ESS) cadaver courses. All evaluations were used to calculate construct validity while paired observations were used to determine interrater agreement. Regional and national faculty input was incorporated for increasing generalizability. Internal consistency was calculated using Cronbach's alpha.

RESULTS

A total of 51 assessments were completed for 28 residents who were evaluated by 15 faculty members as they performed ESS on cadavers over a period of 3 years. A high degree of internal consistency (0.99) and feasibility was noted for the instrument, which took 7 minutes to complete. The interrater agreement improved with focused faculty development for the 3rd year of the course.

CONCLUSION

Our results and experience suggest that a feasible, reliable, and valid instrument for objective evaluation of operative competency can be developed for ESS. Further experience at other otolaryngology programs and efforts focused on faculty development will be needed to enhance faculty buy-in. The instrument can be used for formative and summative feedback as well as for identifying residents needing remediation.

Links

Publisher Full Text

Authors+Show Affiliations

Lin SY
Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins Hospital, Baltimore, Maryland 21287, USA.
Laeeq K
No affiliation info available
Ishii M
No affiliation info available
Kim J
No affiliation info available
Lane AP
No affiliation info available
Reh D
No affiliation info available
Bhatti NI
No affiliation info available

MeSH

Clinical CompetenceEducational MeasurementEndoscopyHumansInternship and ResidencyOtolaryngologyParanasal SinusesPilot Projects

Pub Type(s)

Journal Article

Language

eng

PubMed ID

19490815

Citation

Lin, Sandra Y., et al. "Development and Pilot-testing of a Feasible, Reliable, and Valid Operative Competency Assessment Tool for Endoscopic Sinus Surgery." American Journal of Rhinology & Allergy, vol. 23, no. 3, 2009, pp. 354-9.
Lin SY, Laeeq K, Ishii M, et al. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy. 2009;23(3):354-9.
Lin, S. Y., Laeeq, K., Ishii, M., Kim, J., Lane, A. P., Reh, D., & Bhatti, N. I. (2009). Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. American Journal of Rhinology & Allergy, 23(3), 354-9. https://doi.org/10.2500/ajra.2009.23.3275
Lin SY, et al. Development and Pilot-testing of a Feasible, Reliable, and Valid Operative Competency Assessment Tool for Endoscopic Sinus Surgery. Am J Rhinol Allergy. 2009 May-Jun;23(3):354-9. PubMed PMID: 19490815.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. AU - Lin,Sandra Y, AU - Laeeq,Kulsoom, AU - Ishii,Masaru, AU - Kim,Jean, AU - Lane,Andrew P, AU - Reh,Douglas, AU - Bhatti,Nasir I, PY - 2009/6/4/entrez PY - 2009/6/6/pubmed PY - 2009/7/31/medline SP - 354 EP - 9 JF - American journal of rhinology & allergy JO - Am J Rhinol Allergy VL - 23 IS - 3 N2 - BACKGROUND: Otolaryngology residency programs are required by the Accreditation Council for Graduate Medical Education to evaluate residents' operative competency. Many such tools based on the model of objective structured assessment of technical skills (OSATSs) have been developed in other surgical specialties, but no such instruments exist for otolaryngologic procedures except for tonsillectomy. Endoscopic sinus surgery (ESS) is among the most common rhinologic procedure and lends itself to objective evaluation of operative competency. The purpose of this study was to develop and test a new tool for ESS, focusing on feasibility, content and construct validity, and interrater agreement that can be used for such assessment in the operating room and the cadaver dissection course. METHODS: Faculty input via the modified Delphi technique helped develop the content of a new OSATS-based instrument. The instrument underwent serial improvements based on 3 years of endoscopic sinus surgery (ESS) cadaver courses. All evaluations were used to calculate construct validity while paired observations were used to determine interrater agreement. Regional and national faculty input was incorporated for increasing generalizability. Internal consistency was calculated using Cronbach's alpha. RESULTS: A total of 51 assessments were completed for 28 residents who were evaluated by 15 faculty members as they performed ESS on cadavers over a period of 3 years. A high degree of internal consistency (0.99) and feasibility was noted for the instrument, which took 7 minutes to complete. The interrater agreement improved with focused faculty development for the 3rd year of the course. CONCLUSION: Our results and experience suggest that a feasible, reliable, and valid instrument for objective evaluation of operative competency can be developed for ESS. Further experience at other otolaryngology programs and efforts focused on faculty development will be needed to enhance faculty buy-in. The instrument can be used for formative and summative feedback as well as for identifying residents needing remediation. SN - 1945-8924 UR - https://www.unboundmedicine.com/medline/citation/19490815/Development_and_pilot_testing_of_a_feasible_reliable_and_valid_operative_competency_assessment_tool_for_endoscopic_sinus_surgery_ L2 - https://journals.sagepub.com/doi/10.2500/ajra.2009.23.3275?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub=pubmed DB - PRIME DP - Unbound Medicine ER -