Tags

Type your tag names separated by a space and hit enter

Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods.
Eur J Dent Educ. 2009 Aug; 13(3):162-71.EJ

Abstract

AIM

Aim of this study is to elucidate which standard setting method is optimal to prevent incompetent students to pass and competent students to fail a dental Objective Structured Clinical Examination (OSCE).

MATERIAL AND METHODS

An OSCE with 14 test stations was used to assess the performance of 119 third year dental students in a training group practice. To establish the pass/fail standard per station, three standard setting methods were applied: the Angoff I method, the modified Angoff II with reality check and the Borderline Regression (BR) method. For the final decision about passing or failing the complete OSCE, three methods were compared: total compensatory (TC), a partial compensatory (PC) within clusters of competence and a non-compensatory (NC) model. The reliability of the pass/fail standard of the three methods was indicated by the root mean square error (RMSE). As a criterion measure, a sample of the students (n = 89) was rated in the clinic by their instructors and accordingly these students were divided into two groups: competent and incompetent students. The students' clinical rating (considered for this study as 'true qualification') was compared with the pass-fail classification resulting from the OSCE. Undeserved passing of an incompetent student was considered as more damaging than failing a competent student.

RESULTS

The BR method showed more acceptable results than the two Angoff methods. In terms of pass rate the BR method showed the highest pass rates: for the TC model the Angoff method I and II and the BR showed pass rates of 86.6%, 86.6% and 97.5% respectively. For the PC model the pass rates were 30.3%, 34.5% and 61.3%, and for the NC model the pass rates were 0.8%, 1.7% and 7.6%. The BR method showed lower RMSEs (higher reliability): for the TC model the RMSEs were 1.3%, 1.0% and 0.3% for the Angoff I, Angoff II and BR method respectively, and for the PC model the RMSE of the clusters of competence range was 2.0-3.7% for Angoffs I; 1.8-2.2% for Angoff II and 0.6-0.7% for the BR method. In terms of incorrect decisions, the BR method had a higher loss due to incorrect decisions for the TC model than for the PC model which is in accordance with the results of other studies in medical education.

CONCLUSIONS

Therefore we conclude that the BR method in a PC model provides defensible pass/fail standards and seems to be the optimal choice for OSCEs in health education.

Links

Aggregator Full Text

Authors+Show Affiliations

Schoonheim-Klein M
Department of Periodontology, Academic Centre for Dentistry Amsterdam, Amsterdam, The Netherlands. m.schoonheim@acta.nl
Muijtjens A
No affiliation info available
Habets L
No affiliation info available
Manogue M
No affiliation info available
van der Vleuten C
No affiliation info available
van der Velden U
No affiliation info available

MeSH

Clinical CompetenceEducation, DentalEducational MeasurementFemaleHumansMaleRegression Analysis

Pub Type(s)

Comparative Study
Journal Article

Language

eng

PubMed ID

19630935

Citation

Schoonheim-Klein, M, et al. "Who Will Pass the Dental OSCE? Comparison of the Angoff and the Borderline Regression Standard Setting Methods." European Journal of Dental Education : Official Journal of the Association for Dental Education in Europe, vol. 13, no. 3, 2009, pp. 162-71.
Schoonheim-Klein M, Muijtjens A, Habets L, et al. Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. Eur J Dent Educ. 2009;13(3):162-71.
Schoonheim-Klein, M., Muijtjens, A., Habets, L., Manogue, M., van der Vleuten, C., & van der Velden, U. (2009). Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. European Journal of Dental Education : Official Journal of the Association for Dental Education in Europe, 13(3), 162-71. https://doi.org/10.1111/j.1600-0579.2008.00568.x
Schoonheim-Klein M, et al. Who Will Pass the Dental OSCE? Comparison of the Angoff and the Borderline Regression Standard Setting Methods. Eur J Dent Educ. 2009;13(3):162-71. PubMed PMID: 19630935.
* Article titles in AMA citation format should be in sentence-case
TY - JOUR T1 - Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. AU - Schoonheim-Klein,M, AU - Muijtjens,A, AU - Habets,L, AU - Manogue,M, AU - van der Vleuten,C, AU - van der Velden,U, PY - 2009/7/28/entrez PY - 2009/7/28/pubmed PY - 2009/10/30/medline SP - 162 EP - 71 JF - European journal of dental education : official journal of the Association for Dental Education in Europe JO - Eur J Dent Educ VL - 13 IS - 3 N2 - AIM: Aim of this study is to elucidate which standard setting method is optimal to prevent incompetent students to pass and competent students to fail a dental Objective Structured Clinical Examination (OSCE). MATERIAL AND METHODS: An OSCE with 14 test stations was used to assess the performance of 119 third year dental students in a training group practice. To establish the pass/fail standard per station, three standard setting methods were applied: the Angoff I method, the modified Angoff II with reality check and the Borderline Regression (BR) method. For the final decision about passing or failing the complete OSCE, three methods were compared: total compensatory (TC), a partial compensatory (PC) within clusters of competence and a non-compensatory (NC) model. The reliability of the pass/fail standard of the three methods was indicated by the root mean square error (RMSE). As a criterion measure, a sample of the students (n = 89) was rated in the clinic by their instructors and accordingly these students were divided into two groups: competent and incompetent students. The students' clinical rating (considered for this study as 'true qualification') was compared with the pass-fail classification resulting from the OSCE. Undeserved passing of an incompetent student was considered as more damaging than failing a competent student. RESULTS: The BR method showed more acceptable results than the two Angoff methods. In terms of pass rate the BR method showed the highest pass rates: for the TC model the Angoff method I and II and the BR showed pass rates of 86.6%, 86.6% and 97.5% respectively. For the PC model the pass rates were 30.3%, 34.5% and 61.3%, and for the NC model the pass rates were 0.8%, 1.7% and 7.6%. The BR method showed lower RMSEs (higher reliability): for the TC model the RMSEs were 1.3%, 1.0% and 0.3% for the Angoff I, Angoff II and BR method respectively, and for the PC model the RMSE of the clusters of competence range was 2.0-3.7% for Angoffs I; 1.8-2.2% for Angoff II and 0.6-0.7% for the BR method. In terms of incorrect decisions, the BR method had a higher loss due to incorrect decisions for the TC model than for the PC model which is in accordance with the results of other studies in medical education. CONCLUSIONS: Therefore we conclude that the BR method in a PC model provides defensible pass/fail standards and seems to be the optimal choice for OSCEs in health education. SN - 1600-0579 UR - https://www.unboundmedicine.com/medline/citation/19630935/Who_will_pass_the_dental_OSCE_Comparison_of_the_Angoff_and_the_borderline_regression_standard_setting_methods_ L2 - https://doi.org/10.1111/j.1600-0579.2008.00568.x DB - PRIME DP - Unbound Medicine ER -