Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score

Factors Associated With the Quality of the Patient-Doctor Relationship: A Cross-Sectional Study of Ambulatory Mexican Patients With Rheumatic Diseases

BACKGROUND

The patient-doctor relationship (PDR) is a complex phenomenon with strong cultural determinants, which impacts health-related outcomes and, accordingly, does have ethical implications. The study objective was to describe the PDR from medical encounters between 600 Mexican outpatients with rheumatic diseases and their attending rheumatologists, and to identify factors associated with a good PDR.

METHODS

A cross-sectional study was performed. Patients completed the PDRQ-9 (Patient-Doctor Relationship Questionnaire, 9 items), the HAQ-DI (Health Assessment Questionnaire Disability Index), the Short-Form 36 items (SF-36), a pain-visual analog scale, and the Ideal Patient Autonomy Scale. Relevant sociodemographic, disease-related, and treatment-related variables were obtained. Patients assigned a PDRQ-9 score to each patient-doctor encounter. Regression analysis was used to identify factors associated with a good PDR, which was defined based on a cutoff point established using the borderline performance method.

RESULTS

Patients were primarily middle-aged female subjects (86%), with substantial disease duration (median, 11.1 years), without disability (HAQ-DI within reference range, 55.3%), and with deteriorated quality of life (SF-36 out of reference range, 73.7%-78.6%). Among them, 36.5% had systemic lupus erythematosus and 31.8% had rheumatoid arthritis. There were 422 patients (70.3%) with a good PDR and 523 medical encounters (87.2%) involved certified rheumatologists.Patient paternalistic ideal of autonomy (odds ratio [OR], 3.029; 95% confidence interval [CI], 1.793-5.113), SF-36 score (OR, 1.014; 95% CI, 1.003-1.025), female sex (OR, 0.460; 95% CI, 0.233-0.010), and being certified rheumatologist (OR, 1.526; 95% CI, 1.059-2.200) were associated with a good PDR.

CONCLUSIONS

Patient-related factors and the degree of experience of the attending physician impact the quality of the PDR, in Mexican outpatients with rheumatic diseases.

Entrustment within an objective structured clinical examination (OSCE) progress test: Bridging the gap towards competency-based medical education

PURPOSE

Progress testing aligns well with competency-based medical education (CBME) frameworks, which stress the importance of continuous improvement. Entrustment is a useful assessment concept in CBME models. The purpose of this study was to explore the use of an entrustability rating scale within the context of an objective structured clinical examination (OSCE) Progress Test.

METHODS

A 9-case OSCE Progress Test was administered to Internal Medicine residents (PGYs 1-4). Residents were assessed using a checklist (CL), global rating scale (GRS), training level rating scale (TLRS), and entrustability scale (ENT). Reliability was calculated using Cronbach's alpha. Differences in performance by training year were explored using ANOVA and effect sizes were calculated using partial eta-squared. Examiners completed a post-examination survey.

RESULTS

Ninety one residents and forty two examiners participated in the OSCE. Inter-station reliability was high for all instruments. There was an overall effect of training level for all instruments (p < 0.001). Effect sizes were large. 88% of examiners completed the survey. Most (62%) indicated feeling comfortable in making entrustment decisions during the OSCE.

CONCLUSIONS

An entrustability scale can be used in an OSCE Progress Test to generate highly reliable ratings that discriminate between learners at different levels of training.

Differences in expectations of passing standards in communication skills for pre-clinical and clinical medical students

OBJECTIVE

Communication and interpersonal skills (CIS) are essential elements of competency-based education. We examined defensible CIS passing levels for medical students completing basic sciences (second-year students) and clinical training (fourth-year students), using five standard setting methods.

METHODS

A 14-item CIS scale was used. Data from second-year (n = 190) and fourth-year (n = 170) students were analyzed using descriptive statistics and generalizability studies. Fifteen judges defined borderline CIS performance. Cut scores and fail rates from five standard setting methods (Angoff, Borderline-Group, Borderline-Regression, Contrasting-Groups, and Normative methods) were examined.

RESULTS

CIS performance was similar during second-year (Mean = 74%, SD = 6%) and fourth-year (Mean = 72%, SD = 5%) students. Judges using the Angoff method expected greater competence at the fourth-year level, as reflected in the Angoff cut scores (second-year = 53% with 0% fail, fourth-year = 66% with 10% fail). Cut scores from the remaining methods did not differentiate between training levels. We found evidence of case specificity.

CONCLUSION

Performance on CIS may be case specific. Passing standards for communication skills may require employing approaches such as the Angoff method that are sensitive to expectations of learner performance for different levels of training, competencies, and milestone levels.

PRACTICE IMPLICATIONS

Institutions that want to encourage continued growth in CIS should apply appropriate standard setting methods.

Borderline grades in high stakes clinical examinations: resolving examiner uncertainty

BACKGROUND

Objective Structured Clinical Exams are used to increase reliability and validity, yet they only achieve a modest level of reliability. This low reliability is due in part to examiner variance which is greater than the variance of students. This variance often represents indecisiveness at the cut score with apparent confusion over terms such as "borderline pass". It is amplified by a well reported failure to fail.

METHODS

A borderline grade (meaning performance is neither a clear pass nor a clear fail) was introduced in a high stakes undergraduate medical clinical skills exam to replace a borderline pass grade (which was historically resolved as 50%) in a 4 point scale (distinction, pass, borderline, fail). Each Borderline grade was then resolved into a Pass or Fail grade by a formula referencing the difficulty of the station and the performance in the same domain by the student in other stations. Raw pass or fail grades were unaltered. Mean scores and 95%CI were calculated per station and per domain for the unmodified and the modified scores/grades (results are presented on error bars). To estimate the defensibility of these modifications, similar analysis took place for the P and the F grades which resulted from the modification of the B grades.

RESULTS

Of 14,634 observations 4.69% were Borderline. Application of the formula did not impact the mean scores in each domain but the failure rate for the exam increased from 0.7 to 4.1%. Examiners and students expressed satisfaction with the Borderline grade, resolution formula and outcomes. Mean scores (by stations and by domains respectively) of students whose B grades were modified to P were significantly higher than their counterparts whose B grades were modified to F.

CONCLUSIONS

This study provides a feasible and defensible resolution to situations where the examinee's performance is neither a clear pass nor a clear fail, demonstrating the application of the resolution of borderline formula in a high stakes exam. It does not create a new performance standard but utilises real data to make judgements about these small number of candidates. This is perceived as a fair approach to Pass/Fail decisions.

Cut-scores revisited: feasibility of a new method for group standard setting

BACKGROUND

Standard setting is one of the most contentious topics in educational measurement. Commonly-used methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard setting method yields the highest validity.

METHODS

The method described and piloted in this study asked expert judges to estimate the scores on a real MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole. The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to derive the cut-score and the confidence in it.

RESULTS

In this example the new method's cut-score was higher than the judges' estimate. The method also yielded estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence one may have in the accuracy of that cut-score.

CONCLUSIONS

This new standard-setting method offers some advances, and possibly advantages, in that the decisions being asked of judges are based on firmer constructs, and it takes into account variation among judges.

Enhancing the defensibility of examiners' marks in high stake OSCEs

BACKGROUND

Most assessments in health professions education consist of knowledge-based examinations as well as practical and clinical examinations. Among the most challenging aspects of clinical assessments is decision making related to borderline grades assigned by examiners. Borderline grades are commonly used by examiners when they do not have sufficient information to make clear pass/fail decisions. The interpretation of these borderline grades is rarely discussed in the literature. This study reports the application of the Objective Borderline Method (version 2, henceforth: OBM2) to a high stakes Objective Structured Clinical Examination undertaken at the end of the final year of a Medicine program in Australia.

METHODS

The OBM2 uses all examination data to reclassify borderline grades as either pass or fail. Factor analysis was used to estimate the suitability of data for application of OBM2. Student's t-tests, utilising bootstrapping, were used to compare the OBM2 with 'traditional' results. Interclass correlations were used to estimate the association between the grade reclassification and all other grades in this examination.

RESULTS

The correlations between scores for each station and pass/fail outcomes increased significantly after the mark reclassification, yet the reclassification did not significantly impact on students' total scores. Examiners, students and program leaders expressed high levels of satisfaction and the Faculty's Curriculum Development Committee has decided that the OBM2 will be used for all future clinical examinations. Implications of the OBM2 are discussed.

CONCLUSIONS

The OBM2 provides a feasible, defensible and acceptable solution for classification of borderline grades as either pass or fail.

Does quantity ensure quality? Standardized OSCE-stations for outcome-oriented evaluation of practical skills at different medical faculties

BACKGROUND

Practical skills are often assessed using Objective Structured Clinical Skill Exams (OSCE). Nevertheless, in Germany, interchange and agreement between different medical faculties or a general agreement on the minimum standard for passing is lacking.

METHODS

We developed standardized OSCE-stations for assessing structured clinical examination of knee and shoulder joint with identical checklists and evaluation standards. These were implemented into the OSCE-course at five different medical faculties. Learning objectives for passing the stations were agreed beforehand. At each faculty, one reference examiner scored independently of the local examiner. Outcome of the students at the standardized station was compared between faculties and correlated to their total outcome at the OSCE, to their results at the Part One of the National Medical Licensing Examination as a reference test during medical studies and to their previous amount of lessons in examining joints.

RESULTS

Comparing the results of the reference examiner, outcome at the station differed significantly between some of the participating medical faculties. Depending on the faculty, mean total results at the knee-examination-station differed from 64.4% to 77.9% and at the shoulder-examination-station from 62.6% to 79.2%. Differences were seen in knowledge-based items and also in competencies like communication and professional manner. There was a weak correlation between outcome at the joint-examination-OSCE-station and Part One of the National Medical Licensing Examination, and a modest correlation between outcome at the joint-examination-station and total OSCE-result. Correlation to the previous amount of lessons in examining joint was also weak.

CONCLUSION

Although addressing approved learning objectives, different outcomes were achieved when testing a clinical skill at different medical faculties with a standardized OSCE-station. Results can be used as a tool for evaluating lessons, training and curricula at the different sites. Nevertheless, this study shows the importance of information exchange and agreement upon certain benchmarks and evaluation standards when assessing practical skills.

The method used to set the pass mark in an objective structured clinical examination defines the performance of candidates for certification as rheumatologists

BACKGROUND

The Mexican Accreditation Council for Rheumatology certifies trainees (TR) on an annual basis using both a multiple-choice question (MCQ) test and an objective structured clinical examination (OSCE). For 2013 and 2014, the OSCE pass mark (PM) was set by criterion referencing as ≥6 (CPM), whereas overall rating of borderline performance method (BPM) was added for 2015 and 2016 accreditations. We compared OSCE TR performance according to CPM and BPM, and examined whether correlations between MCQ and OSCE were affected by PM.

METHODS

Forty-three (2015) and 37 (2016) candidates underwent both tests. Altogether, OSCE were integrated by 15 validated stations; one evaluator per station scored TR performance according to a station-tailored check-list and a Likert scale (fail, borderline, above range) of overall performance. A composite OSCE score was derived for each candidate. Appropriate statistics were used.

RESULTS

Mean (±standard derivation [SD]) MCQ test scores were 6.6±0.6 (2015) and 6.4±0.6 (2016) with 5 candidates receiving a failing score each year. Mean (±SD) OSCE scores were 7.4±0.6 (2015) and 7.3±0.6 (2016); no candidate received a failing CPM score in either 2015 or 2016 OSCE, although 21 (49%) and 19 (51%) TR, respectively, received a failing BPM score (calculated as 7.3 and 7.4, respectively). Stations for BPM ranged from 4.5 to 9.5; overall, candidates showed better performance in CPM. In all, MCQ correlated with composite OSCE, r=0.67 (2015) and r=0.53 (2016); P≤.001. Trainees with a passing BPM score in OSCE had higher MCQ scores than those with a failing score.

CONCLUSIONS

Overall, OSCE-PM selection impacted candidates' performance but had a limited affect on correlation between clinical and practical examinations.

A Work-Centered Approach for Setting Passing Scores on Performance-Based Assessments

Standardized patient examinations are being used for high-stakes decisions (e.g., graduation, licensure, and certification) with growing frequency. Concurrently, research on methods to determine the passing score for these types of performance-based assessments has increased. A wide variety of approaches have been considered in the past several years, many based on traditional techniques developed for use with multiple-choice examinations. More and more, techniques that center on review of examinee work have been employed, often resulting in the establishment of defensible, reproducible standards. The purpose of this article is to describe and evaluate a work-centered approach for determining the passing score for a performance-based assessment of clinical skills. A description of an approach to set passing scores that involves expert review of examinee work is presented. Application and evaluation of the method are illustrated using examination materials obtained from a high-stakes certification assessment.

Dental Undergraduate Views of Objective Structured Clinical Examinations (OSCEs): A Literature Review

Objective Structured Clinical Examinations (OSCEs) are widely used in dental undergraduate assessment, often determining progression or graduation. Student evaluation of this assessment process is important, and this includes identifying the views of the student. The aim of this paper is to present a review of the current literature regarding dental student perceptions of OSCEs. A search of the PubMed database covering the period 1975 to 2015 identified 121 possible papers from which only six were suitable for review. The remaining papers were excluded due to them not reporting on dental undergraduate views. Students perceived the OSCE to be a valid assessment in three studies, but not in one. The educational benefit of an OSCE is well supported by these studies. OSCEs can induce high levels of anxiety compared to other forms of assessments, but this did not affect student performance. The majority of students would chose to have a similar format of assessment again, showing support for OSCEs. Further research using larger cohorts of students could be undertaken in order to support these finding which would give added evidence for the continuing use of OSCEs as a valid method of both dental undergraduate education and assessment.

How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE)

BACKGROUND

The goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method.

METHODS

The results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for "will the borderline/advanced beginner examinee respond correctly to this item?" The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods.

RESULTS

A paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01).

CONCLUSION

The results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.

Setting pass scores for assessment of technical performance by surgical trainees

BACKGROUND

One of the major challenges of competency-based training is defining a score representing a competent performance. The objective of this study was to set pass scores for the Objective Structured Assessment of Technical Skill.

METHODS

Pass scores for the examination were set using three standard setting methods applied to data collected prospectively from first-year surgical residents (trainees). General surgery residents were then assigned an overall pass-fail status for each method. Using a compensatory model, residents passed the eight station examinations if they met the overall pass score; using a conjunctive model, residents passed if they met the overall pass score and passed at least 50 per cent of the stations. The consistency of the pass-fail decision across the three methods, and between a compensatory and conjunctive model, were compared.

RESULTS

Pass scores were stable across all three methods using data from 513 residents, 133 of whom were general surgeons. Consistency of the pass-fail decision across the three methods was 95.5 and 93.2 per cent using compensatory and conjunctive models respectively. Consistency of the pass-fail status between compensatory and conjunctive models for all three methods was also very high (91.7, 95.5 and 96.2 per cent).

CONCLUSION

Consistency in pass-fail status between the various methods builds evidence of validity for the set scores. These methods can be applied and studied across a variety of assessment platforms, helping to increase the use of standard setting for competency-based training.

Standard setting in OSCEs: a borderline approach

BACKGROUND

The evaluation of clinical skills and competencies is a high-stakes process carrying significant consequences for the candidate. Hence, it is mandatory to have a robust method to justify the pass score in order to maintain a valid and reliable objective structured clinical examination (OSCE). The aim was to trial the borderline approach using the two-domain global rating scale for standard setting in the OSCE.

METHODS

For each domain, a set of six-point (from 5 to 0) scales were used to reflect high and low divisions within the 'pass', 'borderline' and 'fail' categories. Scores on the two individual global scales were summed to create a 'summed global rating'. Similarly task-based checklists for individual stations were summed to get a total score. It is mandatory to have a robust method to justify the pass score in order to maintain a valid and reliable OSCE RESULTS: The Pearson's correlation between task-based checklist scoring and the two-domain global rating scale were moderate and significant. The highest R(2) coefficient of 0.479 was obtained for station 7, and the lowest R(2) value was 0.241 for station 14.

DISCUSSION

There was a significant positive correlation between the two scales; however, the R(2) value was not satisfactory except for station 7. The pass mark for the OSCE according to the borderline method was 64 per cent, which is higher than the arbitrarily set pass mark of 50 per cent.

CONCLUSIONS

This study confirms that the two-domain global rating scale is appropriate to assess the abilities of students within the framework of an OSCE. The strong relationships between the two-domain global rating scale and task-based checklists provide evidence that the two-domain global rating scale can be used to genuinely assess students' proficiencies.

Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A Validity Study

CONSTRUCT: Authentic standard setting methods will demonstrate high convergent validity evidence of their outcomes, that is, cutoff scores and pass/fail decisions, with most other methods when compared with each other.

BACKGROUND

The objective structured clinical examination (OSCE) was established for valid, reliable, and objective assessment of clinical skills in health professions education. Various standard setting methods have been proposed to identify objective, reliable, and valid cutoff scores on OSCEs. These methods may identify different cutoff scores for the same examinations. Identification of valid and reliable cutoff scores for OSCEs remains an important issue and a challenge.

APPROACH

Thirty OSCE stations administered at least twice in the years 2010-2012 to 393 medical students in Years 2 and 3 at Aga Khan University are included. Psychometric properties of the scores are determined. Cutoff scores and pass/fail decisions of Wijnen, Cohen, Mean-1.5SD, Mean-1SD, Angoff, borderline group and borderline regression (BL-R) methods are compared with each other and with three variants of cluster analysis using repeated measures analysis of variance and Cohen's kappa.

RESULTS

The mean psychometric indices on the 30 OSCE stations are reliability coefficient = 0.76 (SD = 0.12); standard error of measurement = 5.66 (SD = 1.38); coefficient of determination = 0.47 (SD = 0.19), and intergrade discrimination = 7.19 (SD = 1.89). BL-R and Wijnen methods show the highest convergent validity evidence among other methods on the defined criteria. Angoff and Mean-1.5SD demonstrated least convergent validity evidence. The three cluster variants showed substantial convergent validity with borderline methods.

CONCLUSIONS

Although there was a high level of convergent validity of Wijnen method, it lacks the theoretical strength to be used for competency-based assessments. The BL-R method is found to show the highest convergent validity evidences for OSCEs with other standard setting methods used in the present study. We also found that cluster analysis using mean method can be used for quality assurance of borderline methods. These findings should be further confirmed by studies in other settings.

Progress testing: is there a role for the OSCE?

CONTEXT

The shift from a time-based to a competency-based framework in medical education has created a need for frequent formative assessments. Many educational programmes use some form of written progress test to identify areas of strength and weakness and to promote continuous improvement in their learners. However, the role of performance-based assessments, such as objective structured clinical examinations (OSCEs), in progress testing remains unclear.

OBJECTIVE

The aims of this paper are to describe the use of an OSCE to assess learners at different stages of training, describe a structure for reporting scores, and provide evidence for the psychometric properties of different rating tools.

METHODS

A 10-station OSCE was administered to internal medicine residents in postgraduate years (PGYs) 1-4. Candidates were assessed using a checklist (CL), a global rating scale (GRS) and a training level rating scale (TLRS). Reliability was calculated for each measure using Cronbach's alpha. Differences in performance by year of training were explored using analysis of variance (anova). Correlations between scores obtained using the different rating instruments were calculated.

RESULTS

Sixty-nine residents participated in the OSCE. Inter-station reliability was greater (0.88) using the TLRS compared with the CL (0.84) and GRS (0.79). Using all three rating instruments, scores varied significantly by year of training (p < 0.001). Scores from the different rating instruments were highly correlated: CL and GRS, r = 0.93; CL and TLRS, r = 0.90, and GRS and TLRS, r = 0.94 (p < 0.001). Candidates received feedback on their performance relative to examiner expectations for their PGY level.

CONCLUSIONS

Scores were found to have high reliability and demonstrated significant differences in performance by year of training. This provides evidence for the validity of using scores achieved on an OSCE as markers of progress in learners at different levels of training. Future studies will focus on assessing individual progress on the OSCE over time.

Summative assessment of undergraduates' communication competence in challenging doctor-patient encounters. Evaluation of the Düsseldorf CoMeD-OSCE

OBJECTIVE

To evaluate the summative assessment (OSCE) of a communication training programme for dealing with challenging doctor-patient encounters in the 4th study year.

METHODS

Our OSCE consists of 4 stations (breaking bad news, guilt and shame, aggressive patients, shared decision making), using a 4-item global rating (GR) instrument. We calculated reliability coefficients for different levels, discriminability of single items and interrater reliability. Validity was estimated by gender differences and accordance between GR and a checklist.

RESULTS

In a pooled sample of 456 students in 3 OSCEs over 3 terms, total reliability was α=0.64, reliability coefficients for single stations were >0.80, and discriminability in 3 of 4 stations was within the range of 0.4-0.7. Except for one station, interrater reliability was moderate to strong. Reliability on item level was poor and pointed to some problems with the use of the GR.

CONCLUSION

The application of the GR on regular undergraduate medical education shows moderate reliability in need of improvement and some traits of validity. Ongoing development and evaluation is needed with particular regard to the training of the examiners.

PRACTICE IMPLICATIONS

Our CoMeD-OSCE proved suitable for the summative assessment of communication skills in challenging doctor-patient encounters.

Musculoskeletal ultrasound objective structured clinical examination: an assessment of the test

OBJECTIVE

To determine the reliability and validity of an objective structured clinical examination (OSCE) for musculoskeletal ultrasound (MSUS).

METHODS

A 9-station OSCE was administered to 35 rheumatology fellows trained in MSUS and to 3 expert faculty (controls). Participants were unaware of joint health (5 diseased/4 healthy). Faculty assessors (n = 9) graded image quality with predefined checklists and a 0-5 global rating, blinded to who performed the study. Interrater reliability, correlation between a written multiple choice question examination (MCQ) and OSCE performance, and comparison of fellow OSCE results with those of the faculty were measured to determine OSCE reliability, concurrent validity, and construct validity.

RESULTS

Assessors' interrater reliability was good (intraclass correlation coefficient [ICC] 0.7). Score reliability was good in the normal wrist and ankle stations (ICC 0.7) and moderate in the abnormal wrist and ankle stations (ICC 0.4). MCQ grades significantly correlated with OSCE grades (r = 0.52, P < 0.01). The fellows in the bottom quartile of the MCQ scored 3.07 on the OSCE, significantly worse than the top quartile fellows (3.32) and the faculty (3.29; P < 0.01). Scores also significantly discriminated bottom quartile fellows from faculty in the normal wrist and ankle stations (3.38 versus 3.78; P < 0.01), but not in the abnormal stations (3.37 versus 3.49; P = 0.08).

CONCLUSION

MSUS OSCE is a reliable and valid method for evaluation of MSUS skill. Normal joint assessment stations are more reliable than abnormal joint assessment stations and better discriminate poorly performing fellows from faculty. Therefore, MSUS OSCE with normal joints can be used for the assessment of MSUS skill competency.

The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments

The decision to pass or fail a medical student is a 'high stakes' one. The aim of this study is to introduce and demonstrate the feasibility and practicality of a new objective standard-setting method for determining the pass/fail cut-off score from borderline grades. Three methods for setting up pass/fail cut-off scores were compared: the Regression Method, the Borderline Group Method, and the new Objective Borderline Method (OBM). Using Year 5 students' OSCE results from one medical school we established the pass/fail cut-off scores by the abovementioned three methods. The comparison indicated that the pass/fail cut-off scores generated by the OBM were similar to those generated by the more established methods (0.840 ≤ r ≤ 0.998; p < .0001). Based on theoretical and empirical analysis, we suggest that the OBM has advantages over existing methods in that it combines objectivity, realism, robust empirical basis and, no less importantly, is simple to use.

Global rating scale for the assessment of paramedic clinical competence

OBJECTIVE

The aim of this study was to develop and critically appraise a global rating scale (GRS) for the assessment of individual paramedic clinical competence at the entry-to-practice level.

METHODS

The development phase of this study involved task analysis by experts, contributions from a focus group, and a modified Delphi process using a national expert panel to establish evidence of content validity. The critical appraisal phase had two raters apply the GRS, developed in the first phase, to a series of sample performances from three groups: novice paramedic students (group 1), paramedic students at the entry-to-practice level (group 2), and experienced paramedics (group 3). Using data from this process, we examined the tool's reliability within each group and tested the discriminative validity hypothesis that higher scores would be associated with higher levels of training and experience.

RESULTS

The development phase resulted in a seven-dimension, seven-point adjectival GRS. The two independent blinded raters scored 81 recorded sample performances (n = 25 in group 1, n = 33 in group 2, n = 23 in group 3) using the GRS. For groups 1, 2, and 3, respectively, interrater reliability reached 0.75, 0.88, and 0.94. Intrarater reliability reached 0.94 and the internal consistency ranged from 0.53 to 0.89. Rater differences contributed 0-5.7% of the total variance. The GRS scores assigned to each group increased with level of experience, both using the overall rating (means = 2.3, 4.1, 5.0; p < 0.001) and considering each dimension separately. Applying a modified borderline group method, 54.9% of group 1, 13.4% of group 2, and 2.9% of group 3 were below the cut score.

CONCLUSION

The results of this study provide evidence that the scores generated using this scale can be valid for the purpose of making decisions regarding paramedic clinical competence.

Pediatrics milestone project: next steps toward meaningful outcomes assessment

In the September 2010 issue of JGME, the Pediatric Milestones Working Group published "The Pediatrics Milestones: Conceptual Framework, Guiding Principles, and Approach to Development", a document that describes the construction of the first iteration of the Pediatric Milestones. These Milestones were developed by the Working Group as a group of practical behavioral expectations for each of the 52 sub-competencies. In constructing these Milestones, the authors were cognizant of the need to ground the Milestones themselves in evidence, theories or other conceptual frameworks that would provide the basis for the ontogeny of development for each sub-competency. During this next phase of the Milestones development, the process will continue with consultation with content experts and consideration of assessment of Milestones. We have described possible measurement tools, explored threats to validity, establishment of benchmarks, and possible approaches to reporting of performance. The vision of the Pediatrics Milestone Project is to understand the development of a pediatrician from entry into medical school through the twilight of a physician's career, and the work will require a collaborative effort of the undergraduate and graduate medical education communities, and the accrediting and certifying bodies.

Assessment of technical skills: best practices

Assessment is an important aspect of veterinary education from the point of view of setting standards, driving learning, providing feedback, and reassuring society that veterinarians are competent to assume the responsibilities entrusted to them. However, no single format exists that can, by itself, assess the complex mixture of knowledge and skills essential to the veterinarian's role. The areas that are most challenging to assess are those involving behaviors and attitudes. These include the various technical skills required for diagnosis and treatment. One approach, aimed at retaining validity but improving reliability compared with traditional, more subjective methods, first described in medicine 35 years ago, is the Objective Structured Clinical Examination (OSCE), which has been introduced into veterinary education as the Objective Structured Practical Veterinary Examination (OSPVE) and run at the Royal Veterinary College since 2004. This approach is good for the assessment of competence in relation to isolated techniques and whole procedures but has been criticized for the way in which these are tested out of context. However, further development of structured clinical assessments, such as the mini-Clinical Examination and the Direct Observation of Procedural Skills, may help address some of these limitations, and the use of multi-source feedback, particularly client feedback, may allow the further domains of professional behaviors, attitudes, and communication to be judged and developed.

Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods

AIM

Aim of this study is to elucidate which standard setting method is optimal to prevent incompetent students to pass and competent students to fail a dental Objective Structured Clinical Examination (OSCE).

MATERIAL AND METHODS

An OSCE with 14 test stations was used to assess the performance of 119 third year dental students in a training group practice. To establish the pass/fail standard per station, three standard setting methods were applied: the Angoff I method, the modified Angoff II with reality check and the Borderline Regression (BR) method. For the final decision about passing or failing the complete OSCE, three methods were compared: total compensatory (TC), a partial compensatory (PC) within clusters of competence and a non-compensatory (NC) model. The reliability of the pass/fail standard of the three methods was indicated by the root mean square error (RMSE). As a criterion measure, a sample of the students (n = 89) was rated in the clinic by their instructors and accordingly these students were divided into two groups: competent and incompetent students. The students' clinical rating (considered for this study as 'true qualification') was compared with the pass-fail classification resulting from the OSCE. Undeserved passing of an incompetent student was considered as more damaging than failing a competent student.

RESULTS

The BR method showed more acceptable results than the two Angoff methods. In terms of pass rate the BR method showed the highest pass rates: for the TC model the Angoff method I and II and the BR showed pass rates of 86.6%, 86.6% and 97.5% respectively. For the PC model the pass rates were 30.3%, 34.5% and 61.3%, and for the NC model the pass rates were 0.8%, 1.7% and 7.6%. The BR method showed lower RMSEs (higher reliability): for the TC model the RMSEs were 1.3%, 1.0% and 0.3% for the Angoff I, Angoff II and BR method respectively, and for the PC model the RMSE of the clusters of competence range was 2.0-3.7% for Angoffs I; 1.8-2.2% for Angoff II and 0.6-0.7% for the BR method. In terms of incorrect decisions, the BR method had a higher loss due to incorrect decisions for the TC model than for the PC model which is in accordance with the results of other studies in medical education.

CONCLUSIONS

Therefore we conclude that the BR method in a PC model provides defensible pass/fail standards and seems to be the optimal choice for OSCEs in health education.

A summative, Objective, Structured, Clinical Examination in ENT used to assess postgraduate doctors after one year of ENT training, as part of the Diploma of Otorhinolaryngology, Head and Neck Surgery

CONTEXT

'High stakes' postgraduate medical examinations should conform to current educational standards. In the UK and Ireland, national assessments in surgery are devised and managed through the examination structure of the Royal Colleges of Surgeons. Their efforts are not reported in the medical education literature. In the current paper, we aim to clarify this process.

OBJECTIVES

To replace the clinical section of the Diploma of Otorhinolaryngology with an Objective, Structured, Clinical Examination, and to set the level of the assessment at one year of postgraduate training in the specialty.

METHODS

After 'blueprinting' against the whole curriculum, an Objective, Structured, Clinical Examination comprising 25 stations was divided into six clinical stations and 19 other stations exploring written case histories, instruments, test results, written communication skills and interpretation skills. The pass mark was set using a modified borderline method and other methods, and statistical analysis of the results was performed.

RESULTS

The results of nine examinations between May 2004 and May 2008 are presented. The pass mark varied between 68 and 82 per cent. Internal consistency was good, with a Cronbach's alpha value of 0.99 for all examinations and split-half statistics varying from 0.96 to 0.99. Different standard settings gave similar pass marks.

CONCLUSIONS

We have developed a summative, Objective, Structured, Clinical Examination for doctors training in otorhinolaryngology, reported herein. The objectives and standards of setting a high quality assessment were met.

Sources of variation in performance on a shared OSCE station across four UK medical schools

CONTEXT

High-stakes undergraduate clinical assessments should be based on transparent standards comparable between different medical schools. However, simply sharing questions and pass marks may not ensure comparable standards and judgements. We hypothesised that in multicentre examinations, teaching institutions contribute to systematic variations in students' marks between different medical schools through the behaviour of their markers, standard-setters and simulated patients.

METHODS

We embedded a common objective structured clinical examination (OSCE) station in four UK medical schools. All students were examined by a locally trained examiner as well as by a centrally provided examiner. Central and local examiners did not confer. Pass scores were calculated using the borderline groups method. Mean scores awarded by each examiner group were also compared. Systematic variations in scoring between schools and between local and central examiners were analysed.

RESULTS

Pass scores varied slightly but significantly between each school, and between local and central examiners. The patterns of variation were usually systematic between local and central examiners (either consistently lower or higher). In some cases scores given by one examiner pair were significantly different from those awarded by other pairs in the same school, implying that other factors (possibly simulated patient behaviour) make a significant difference to student scoring.

CONCLUSIONS

Shared undergraduate clinical assessments should not rely on scoring systems and standard setting which fail to take into account other differences between schools. Examiner behaviour and training and other local factors are important contributors to variations in scores between schools. The OSCE scores of students from different medical schools should not be directly compared without taking such systematic variations into consideration.

Setting standards for teaching evaluation data: an application of the contrasting groups method

BACKGROUND

Teaching evaluations are widely used in retention and remediation decisions. Typically evaluations are reviewed in a global manner and some gestalt is reached.

PURPOSE

Apply the Contrasting Groups standard setting methodology to faculty teaching dossiers, to examine resulting "pass-rates" and precision of the decisions.

METHODS

Ten faculty judges set standards for teaching dossiers using a Contrasting Groups approach. Blinded dossiers summarizing clinical (N = 47) and classroom teaching (N = 37) were sorted into piles labeled Unsatisfactory, Satisfactory, Excellent, and Superior. Cut-points were the midpoints between aggregated judge-level mean performances of dossiers placed within adjacent levels.

RESULTS

For the total faculty, the percentage assigned to groups labeled Unsatisfactory through Superior, were 4.1%, 5.9%, 26.1%, and 63.9% and 6.6%, 24.8%, 44.1%, and 24.5% for clinical and classroom teaching respectively. Standard error of measurement was between .20 and .25 with 5 to 7 judges.

CONCLUSIONS

Standard setting methods applied to faculty evaluation data produce precise results. Future work can examine standard stability and acceptability, and methods to combine across teaching venues.

Sharpening the eye of the OSCE with critical action analysis

PURPOSE

When interpreting performance scores on an objective structured clinical examination (OSCE), are all checklist items created equal? Although assigning priority through checklist item weighting is often used to emphasize the clinical importance of selected checklist items, the authors propose the use of critical action analysis as an additional method for analyzing and discriminating clinical performance in clinical skill assessment exercises. A critical action is defined as an OSCE checklist item whose performance is critical to ensure an optimal patient outcome and avoid medical error. In this study, the authors analyzed a set of clerkship OSCE performance outcome data and compared the results of critical action analysis versus traditional checklist item performance scores.

METHOD

OSCE performance scores of 398 third-year clerkship students from 2003 to 2006 at the University of Virginia School of Medicine were analyzed using descriptive statistics and a logistic regression model. Through a consensus process, 10 of 25 OSCE cases were identified as containing critical actions.

RESULTS

Students who scored above the median correctly performed the critical actions more often than those scoring lower. However, for 9 of 10 cases, 6% to 46% of higher-scoring students failed to perform the critical action correctly.

CONCLUSIONS

Failure to address this skill assessment outcome is a missed opportunity to more fully understand and apply the results of such examinations to the clinical performance development of medical students. Including critical action analysis in OSCE data interpretation sharpens the eye of the OSCE and enhances its value in clinical skill assessment.

Competency domains in an undergraduate Objective Structured Clinical Examination: their impact on compensatory standard setting

CONTEXT

Following a 15-week attachment in paediatrics and child health, general practice and dermatology medical students in their second clinical year at this medical school undertake a high-stakes assessment including an objective structured clinical examination (OSCE). There were 2 hypotheses. Firstly, groups of similar stations map to competency domains identifiable by factor analysis. Secondly, poor performance in individual domains is compensated for by achieving the required standard of performance across the whole assessment.

METHODS

A total of 647 medical students were assessed by an OSCE during 5 individual examination sittings (diets) over 2 years. Ten scoring stations in the OSCE were analysed and confirmatory factor analysis performed comparing a 1-factor model (where all the stations are discrete entities related to one underlying domain) with a 3-factor model (where the stations load onto 3 domains from a previously reported exploratory factor analysis).

RESULTS

The 3-factor model yielded a significantly better fit to the data (chi(2 )=( )15.3, P < 0.01). Assessing the compensation data of 1 diet, 29 of 127 students failed in 1 or more domains described, whereas only 5 failed if compensation was allowed across all domains.

DISCUSSION

Confirmatory factor analysis showed a significant fit of the data to previously described competency domains for a high-stakes undergraduate OSCE. Compensation within but not between competency domains would provide a more robust standard, improve validity, and substantially reduce the pass rate.

Estimating the minimum number of judges required for test-centred standard setting on written assessments. do discussion and iteration have an influence?

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers, respectively. Data collected has been analysed to investigate whether reliable standards can be achieved for small-scale, medical school-based assessments, to establish the minimum number of judges required and the effect of a discussion phase on reliability. The root mean squared error (RMSE) has been used as a measure of reliability and used to compute 95% confidence intervals for comparison to the examination statistics. The RMSE has been used to calculate the minimum number of judges required to obtain a predetermined minimum level of reliability, and the effect of the number of judges and number of items have been examined. Values of the RMSE obtained vary from 0.9 to 2.2%. Using average variances across each paper type, the minimum number of judges to obtain a RMSE of less than 2% is 10 or more judges before discussion or 6 or more judges after discussion. The results indicate that including a discussion phase improves the reliability and reduces the minimum number of judges required. Decision studies indicate that increasing the number of questions included in the assessments would not significantly improve the reliability of the standard setting.

Integration of e-learning technologies in an interprofessional health science course

BACKGROUND

Advances in information and communication technology are influencing instructional formats and delivery modes for post secondary teaching and learning.

AIMS

The purpose of this study was to determine whether interprofessional team process skills traditionally taught in a small group face-to-face classroom setting could be taught in a blended learning environment; without compromising the pedagogical approach and collaborative Group Investigation Model (Sharan & Sharan 1992) used in the course.

METHOD

A required interprofessional team development course designed to teach health science students (Medicine, Nursing, Pharmacy, Occupational Therapy, Physical Therapy, Dentistry, Dental Hygiene, Medical Laboratory Science, and Nutrition) team process skills was redesigned from a 100% face-to-face delivery format to a blended learning format where 70% of the instruction was delivered using a new synchronous virtual classroom technology (Elluminate www.elluminate.com) in conjunction with asynchronous technology (WebCT). It was hypothesized there would be no significant difference between the blended learning format and the traditional face-to-face format in the development of interprofessional team knowledge, skills and attitudes. The two formats were evaluated on demographic information, computer experience, and interprofessional team attitudes, knowledge and skills.

RESULTS

The three main findings are: (a) no significant differences between student groups on achieving team process skills, (b) an observation of differences between the groups on team dynamics, and (c) a more positive achievement of course learning objectives perceived by students in the blended learning class.

CONCLUSIONS

The results provide evidence to support our blended learning format without compromising pedagogy. They also suggest that this format enhances students' perceptions of their learning.

Assessment of clinical performance: gathering evidence

The focus of assessment of clinical performance has moved from over-reliance on individual tools to constructing a coherent assessment programme. The purpose of such an assessment programme is to gather high-quality evidence to make well-informed decisions. This requires clarity on the decisions to be made and an ability to gather a sufficient amount of high-quality data. The assessment programme should be aligned to doing the job well so that a successful assessment result reflects what is valued. A variety of assessments over a variety of times, matched against the areas of interest and value, enhances both reliability and validity. Workplace-based assessment tools can complement centralized assessment tools. Multiple snapshots, even if some are not totally in focus, give a better picture than one poorly aimed photograph.

Description and impact of using a standard-setting method for determining pass/fail scores in a surgery clerkship

BACKGROUND

Research on performance evaluation highlights the importance of using multiple measures to develop an accurate profile of students, yet we found no literature describing the use of a standard-setting method for determining a pass/fail cutoff for a clerkship based on multiple assessment methods.

METHOD

Steps in setting an absolute standard for a pass/fail grade are described. The new cut-off score was used to compare what decisions would have been made had it been applied in previous clerkships.

RESULTS

We successfully applied the Hofstee method to ascertain a new standard pass/fail cutoff for our total surgery clerkship score. Had this absolute score been used in 4 prior clerkships, 150 instead of 152 would have passed the clerkship, and 10 instead of 8 would have failed the clerkship.

CONCLUSIONS

A standard-setting method can be applied to a final clinical clerkship grade even when multiple performance measures are used.

Assuring the quality of high-stakes undergraduate assessments of clinical competence

In the UK, and in many Commonwealth countries, a university degree is accepted by registration bodies as an indication of competence to practice as a PRHO or intern. Concerns have been raised that the quality of university examinations may not always be sufficient for such high-stakes decision-making. Assessments of clinical competence are subject to many potential sources of error. The search for standardization, and high validity and reliability, demands the identification and reduction of measurement errors and biases due to poor test design or variation in test items, judges, patients or examination procedures. Generalizability and other research studies have identified where the likely sources of error might arise and have been taken into account in the development of published guidelines on international best practice, which institutions should strive to follow. The purpose of this paper is to describe the development of the integrated final-year assessment of clinical competence at the University of Sheffield. The aim was to introduce a range of strategies to ensure the examination met the best practice guidelines. These included blueprinting the assessment to achieve a high degree of content validity; lengthening the examination by adding a written component to the OSCE component to ensure an adequate level of reliability; providing training and feedback for examiners and simulated patients; paying attention to item development; and providing statistical information to assist the examination committee in standard setting and decision-making. This evidence-based approach should be readily achievable by all medical schools.

What to do about medical students with unsatisfactory consultation skills?

If it is accepted that new doctors must be 'Fit to Practice' in a standard clinical house job, they should be competent in basic communication skills. Although these skills may be assessed as part of a routine OSCE-style exam in the course of the curriculum, the question is raised whether students who fail to demonstrate a minimal level of competence in this area should to be allowed to progress to the next stage of the course and eventually graduate. This paper describes our experiences with introducing 'barrier' stations in communication skills into the OSCE. Students who failed these stations, irrespective of their overall exam performance, undertook remedial studies in a compulsory two-week directed study module followed by a four-station OSCE.

Standard setting in a small scale OSCE: a comparison of the Modified Borderline-Group Method and the Borderline Regression Method

When setting standards, administrators of small-scale OSCEs often face several challenges, including a lack of resources, a lack of available expertise in statistics, and difficulty in recruiting judges. The Modified Borderline-Group Method is a standard setting procedure that compensates for these challenges by using physician examiners and is easy to use making it a good choice for small scale OSCEs. Unfortunately, the use of this approach may introduce a new challenge. Because a small scale OSCE has a small number of examinees, there may be few examinees in the borderline range, which could introduce an unintentional bias. A standard setting method called The Borderline Regression Method will be described. This standard setting method is similar to the Modified Borderline-Group Method but incorporates a linear regression approach allowing the cut score to be set using the scores from all examinees and not from a subset. The current study uses confidence intervals to analyze the precision of cut scores derived from both approaches when applied to a small scale OSCE.

Basic life support skills training in a first year medical curriculum: six years' experience with two cognitive-constructivist designs

RATIONALE

Although the Basic Life Support (BLS) ability of a medical student is a crucial competence, poor BLS training programs have been documented worldwide. Better training designs are needed. This study aims to share detailed descriptions and the test results of two cognitive-constructivist training models for the BLS skills in the first year of medical curriculum.

METHOD

A BLS skills training module was implemented in the first year curriculum in the course of 6 years (1997-2003). The content was derived from the European Resuscitation Council Guidelines. Initially, a competence-based model was used and was upgraded to a cognitive apprenticeship model in 2000. The main performance-content type that was expected at the end of the course was: competent application of BLS procedures on manikins and peers at an OSCE as well as 60% achievement in a test consisting of 25 MCQ items. A retrospective cohort survey design using exam results and a self-completed anonymous student ratings' questionnaire were used in order to test models.

RESULTS

Training time for individual students varied from 21 to 29 hours. One thousand seven hundred and sixty students were trained. Fail rates were very low (1.0-2.2%). The students were highly satisfied with the module during the 6 years.

CONCLUSION

In the first year of the medical curriculum, a competence-based or cognitive apprenticeship model using cognitive-constructivist designs of skills training with 9 hours theoretical and 12-20 hours long practical sessions took place in groups of 12-17 students; medical students reached a degree of competence to sufficiently perform BLS skills on the manikins and their peers. The cognitive-constructivist designs for skills training are associated with high student satisfaction. However, the lack of controls limits the extrapolation of this conclusion.

The relationship between checklist scores on a communication OSCE and analogue patients' perceptions of communication

Many efforts to teach and evaluate physician-patient communication are based on two assumptions: first, that communication can be conceptualized as consisting of specific observable behaviors, and second, that physicians who exhibit certain behaviors are more effective in communicating with patients. These assumptions are usually implicit, and are seldom tested. The purpose of this study was to investigate whether specific communication behaviors are positively related to patients' perceptions of effective communication. Trained raters used a checklist to record the presence or absence of specific communication behaviors in 100 encounters in a communication Objective Structured Clinical Examination (OSCE). Lay volunteers served as analogue patients and rated communication during each encounter. Correlations between checklist scores and analogue patients' ratings were not significantly different from zero for four of five OSCE cases studied. Within each case, certain communication behaviors did appear to be related to patients' ratings, but the critical behaviors were not consistent across cases. We conclude that scores from OSCE communication checklists may not predict patients' perceptions of communication. Determinants of patient perceptions of physician communication may be more subtle, more complex, and more case-specific than we were able to capture with the current checklist.

Setting a standard for performance assessment of doctor-patient communication in general practice

CONTEXT

Continuing professional development (CPD) of general practitioners.

OBJECTIVE

Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated.

METHODS

To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A borderline regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs.

RESULTS

Both methods are applicable and reliable; the obtained standards are credible according to the GPs.

CONCLUSIONS

Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used - i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given - is important because methods differ in practical aspects.

Consistency of standards and stability of pass/fail decisions with examinee-based standard-setting methods in a small-scale objective structured clinical examination

PROBLEM STATEMENT AND BACKGROUND

The apparent feasibility and the face validity of the examinee-based methods such as the borderline-group methods provide support for their increasing adoption by health profession schools. Before that can occur, however, more information on the quality of the standards produced by these techniques is required. The purpose of the present study was to assess the quality of the standards produced on a small-scale objective structured clinical examination (OSCE) by the borderline-group and contrasting-groups examinee-based standard-setting procedures.

METHOD

These two examinee-based criterion-referenced standard-setting procedures were done for an undergraduate fourth-year surgical OSCE and the consistency of the standards and the decisions arising from the standards were assessed.

RESULTS

Both techniques provided consistent and realistic standards.

CONCLUSIONS

There is sufficient theoretical and empirical evidence to support the use of examinee-based standard-setting techniques in small-scale OSCEs that use expert examiners.

Trends in medical education research

The medical education community is reflecting increasingly on the role and nature of research in the field. Useful sources of data to include in these reflections are a description of the topics in which we are investing our energies, an analysis of the extent to which there is a sense of progress on these topics, and an examination of the mechanisms by which any progress has been achieved. This article presents the results of a thematic review of the medical education research literature in four key journals since the turn of the 21st century. It describes four examples of areas in which the community appears to be investing its energies: curriculum and teaching issues, skills and attitudes relevant to the structure of the profession, individual characteristics of medical students, and the evaluation of students and residents. A discussion of the recent publications in these domains highlights a distinction between thematic categories of research, in which many members of the community are working on the same topic, and programmatic lines of research, in which members of the community are working together toward the shared goal of consensual understanding. The author suggests that community-level, programmatic lines of research are necessary to build knowledge and understanding of a domain and that, in the absence of such communal effort, the value of research is limited to the uncoordinated accrual of information.

Techniques for measuring clinical competence: objective structured clinical examinations

The traditional clinical examination has been shown to have serious limitations in terms of its validity and reliability. The OSCE provides some answers to these limitations and has become very popular. Many variants on the original OSCE format now exist and much research has been done on various aspects of their use. Issues to be addressed relate to organization matters and to the quality of the assessment. This paper focuses particularly on the latter with respect to ways of ensuring content validity and achieving acceptable levels of reliability. A particular concern has been the demonstrable need for long examinations if high levels of reliability are to be achieved. Strategies for reducing the practical difficulties this raises are discussed. Standard setting methods for use with OSCEs are described.

Setting defensible performance standards on OSCEs and standardized patient examinations

Recently, standardized patient assessments and objective structured clinical examinations have been used for high-stakes certification and licensure decisions. In these testing situations, it is important that the assessments are standardized, the scores are accurate and reliable, and the resulting decisions regarding competence ar equitable and defensible. For the decisions to be valid, justifiable standards, or cut-scores, must beset. Unfortunately, unlike the body of research specifically dedicated to multiple-choice examinations, relatively little research has been conducted on standard-setting methods appropriate for use with performance-based assessments. The purpose of this article is to provide the reader with some guidance on how to set defensible standards on performance assessments, especially those that utilize standardized patients in simulated medical encounters. Various methods are discussed and contrasted, highlighting the relevant strengths and weaknesses. In addition, based on the prevailing literature and research, ideas for future studies and potential augmentations to current performance-based standard setting protocols are advanced.