Tags

Type your tag names separated by a space and hit enter

Multi-institutional validation of a web-based core competency assessment system.
J Surg Educ. 2007 Nov-Dec; 64(6):390-4.JS

Abstract

OBJECTIVE

The Association of Program Directors in Surgery and the Division of Education of the American College of Surgeons developed and implemented a web-based system for end-of-rotation faculty assessment of ACGME core competencies of residents. This study assesses its reliability and validity across multiple programs.

METHODS

Each assessment included ratings (1-5 scale) on 23 items reflecting the 6 core competencies. A total of 4241 end-of-rotation assessments were completed for 332 general surgery residents (> or =5 evaluations each) at 5 sites during the 2004-2005 and 2005-2006 academic years. The mean rating for each resident on each item was computed for each academic year. The mean rating of items representing each competency was computed for each resident. Additional data included USMLE and ABSITE scores, PGY, and status in program (categorical, designated preliminary, and undesignated preliminary).

RESULTS

Coefficient alpha was greater than 0.90 for each competency score. Mean ratings for each competency increased significantly (p < 0.01) as a function of PGY. Mean ratings for professionalism and interpersonal/communication skills (IPC) were significantly higher than all other competencies at all PGY levels. Competency ratings of PGY 1 residents correlated significantly with USMLE Step I, ranging from (r = 0.26, p < 0.01) for Professionalism to (r = 0.41, p < 0.001) for Systems-Based Practice. Ratings of Knowledge (r = 0.31, p < 0.01), Practice-Based Learning & Improvement (PBLI; r = 0.22, p < 0.05), and Systems-Based Practice (r = 0.20, p < 0.05) correlated significantly with 2005 ABSITE Total Percentile. Ratings of all competencies correlated significantly with the 2006 ABSITE Total Percentile Score (range: r = 0.20, p < 0.05 for professionalism to r = 0.35, p < 0.001 for knowledge). Categorical and designated preliminary residents received significantly higher ratings (p < 0.05) than nondesignated preliminaries for knowledge, patient care, PBLI, and systems-based practice only.

CONCLUSIONS

Faculty ratings of core competencies are internally consistent. The pattern of statistically significant correlations between competency ratings and USMLE and ABSITE scores supports the postdictive and concurrent validity, respectively, of faculty perceptions of resident knowledge. The pattern of increased ratings as a function of PGY supports the construct validity of faculty ratings of resident core competencies.

Authors+Show Affiliations

Tabuenca A
Loma Linda University Medical Center, Loma Linda, California, USA.
Welling R
No affiliation info available
Sachdeva AK
No affiliation info available
Blair PG
No affiliation info available
Horvath K
No affiliation info available
Tarpley J
No affiliation info available
Savino JA
No affiliation info available
Gray R
No affiliation info available
Gulley J
No affiliation info available
Arnold T
No affiliation info available
Wolfe K
No affiliation info available
Risucci DA
No affiliation info available

MeSH

AdultClinical CompetenceEducational MeasurementGeneral SurgeryHumansInternetInternship and ResidencyReproducibility of Results

Pub Type(s)

Journal Article
Multicenter Study
Validation Study

Language

eng

PubMed ID

18063275

Citation

Tabuenca, Arnold, et al. "Multi-institutional Validation of a Web-based Core Competency Assessment System." Journal of Surgical Education, vol. 64, no. 6, 2007, pp. 390-4.
Tabuenca A, Welling R, Sachdeva AK, et al. Multi-institutional validation of a web-based core competency assessment system. J Surg Educ. 2007;64(6):390-4.
Tabuenca, A., Welling, R., Sachdeva, A. K., Blair, P. G., Horvath, K., Tarpley, J., Savino, J. A., Gray, R., Gulley, J., Arnold, T., Wolfe, K., & Risucci, D. A. (2007). Multi-institutional validation of a web-based core competency assessment system. Journal of Surgical Education, 64(6), 390-4.
Tabuenca A, et al. Multi-institutional Validation of a Web-based Core Competency Assessment System. J Surg Educ. 2007 Nov-Dec;64(6):390-4. PubMed PMID: 18063275.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Multi-institutional validation of a web-based core competency assessment system. AU - Tabuenca,Arnold, AU - Welling,Richard, AU - Sachdeva,Ajit K, AU - Blair,Patrice G, AU - Horvath,Karen, AU - Tarpley,John, AU - Savino,John A, AU - Gray,Richard, AU - Gulley,Julie, AU - Arnold,Teresa, AU - Wolfe,Kevin, AU - Risucci,Donald A, PY - 2007/03/19/received PY - 2007/06/20/revised PY - 2007/06/21/accepted PY - 2007/12/8/pubmed PY - 2008/2/26/medline PY - 2007/12/8/entrez SP - 390 EP - 4 JF - Journal of surgical education JO - J Surg Educ VL - 64 IS - 6 N2 - OBJECTIVE: The Association of Program Directors in Surgery and the Division of Education of the American College of Surgeons developed and implemented a web-based system for end-of-rotation faculty assessment of ACGME core competencies of residents. This study assesses its reliability and validity across multiple programs. METHODS: Each assessment included ratings (1-5 scale) on 23 items reflecting the 6 core competencies. A total of 4241 end-of-rotation assessments were completed for 332 general surgery residents (> or =5 evaluations each) at 5 sites during the 2004-2005 and 2005-2006 academic years. The mean rating for each resident on each item was computed for each academic year. The mean rating of items representing each competency was computed for each resident. Additional data included USMLE and ABSITE scores, PGY, and status in program (categorical, designated preliminary, and undesignated preliminary). RESULTS: Coefficient alpha was greater than 0.90 for each competency score. Mean ratings for each competency increased significantly (p < 0.01) as a function of PGY. Mean ratings for professionalism and interpersonal/communication skills (IPC) were significantly higher than all other competencies at all PGY levels. Competency ratings of PGY 1 residents correlated significantly with USMLE Step I, ranging from (r = 0.26, p < 0.01) for Professionalism to (r = 0.41, p < 0.001) for Systems-Based Practice. Ratings of Knowledge (r = 0.31, p < 0.01), Practice-Based Learning & Improvement (PBLI; r = 0.22, p < 0.05), and Systems-Based Practice (r = 0.20, p < 0.05) correlated significantly with 2005 ABSITE Total Percentile. Ratings of all competencies correlated significantly with the 2006 ABSITE Total Percentile Score (range: r = 0.20, p < 0.05 for professionalism to r = 0.35, p < 0.001 for knowledge). Categorical and designated preliminary residents received significantly higher ratings (p < 0.05) than nondesignated preliminaries for knowledge, patient care, PBLI, and systems-based practice only. CONCLUSIONS: Faculty ratings of core competencies are internally consistent. The pattern of statistically significant correlations between competency ratings and USMLE and ABSITE scores supports the postdictive and concurrent validity, respectively, of faculty perceptions of resident knowledge. The pattern of increased ratings as a function of PGY supports the construct validity of faculty ratings of resident core competencies. SN - 1931-7204 UR - https://www.unboundmedicine.com/medline/citation/18063275/Multi_institutional_validation_of_a_web_based_core_competency_assessment_system_ DB - PRIME DP - Unbound Medicine ER -