Examination of clinical examiners

Interrater reliability in neurology objective structured clinical examination across specialties

PURPOSE

To assess interrater reliability and examiners' characteristics, especially specialty, associated with scoring of neurology objective structured clinical examination (OSCE).

MATERIAL AND METHODS

During a neurology mock OSCE, five randomly chosen students volunteers were filmed while performing 1 of the 5 stations. Video recordings were scored by physicians from the Lyon and Clermont-Ferrand university teaching hospitals to assess students performance using both a checklist scoring and a global rating scale. Interrater reliability between examiners were assessed using intraclass coefficient correlation. Multivariable linear regression models including video recording as random effect dependent variable were performed to detect factors associated with scoring.

RESULTS

Thirty examiners including 15 (50%) neurologists participated. The intraclass correlation coefficient of checklist scores and global ratings between examiners were 0.71 (CI95% [0.45-0.95]) and 0.54 (CI95% [0.28-0.91]), respectively. In multivariable analyses, no factor was associated with checklist scores, while male gender of examiner was associated with lower global rating (β coefficient = -0.37; CI 95% [-0.62-0.11]).

CONCLUSIONS

Our study demonstrated through a video-based scoring method that agreement among examiners was good using checklist scoring while moderate using global rating scale in neurology OSCE. Examiner's specialty did not affect scoring whereas gender was associated with global rating scale.

Comparing Objective Structured Clinical Examinations and Traditional Clinical Examinations in the Summative Evaluation of Final-Year Medical Students

BACKGROUND

Medical schools have traditionally assessed medical students using long and short cases. Objective structured clinical examination (OSCE) has been found to be more reliable.

AIM

To compare OSCE and traditional method of assessment in the summative assessment of final-year medical students.

METHODOLOGY

This was a retrospective cross-sectional study conducted at Enugu State University of Science and Technology College of Medicine. The Department of Internal Medicine organized clinical examinations consisting of long and short cases. The Department of Surgery organized an OSCE consisting of two parts (picture OSCE and clinical OSCE). Students' scores in both internal medicine and surgery were collated and subjected to analysis with SPSS version 23 (IBM; SPSS, Chicago, IL, USA). Pearson's correlation was used to assess the correlations, paired t-test was used to compare the mean scores, and Cronbach's alpha was used to assess the reliability. P < 0.05 was considered statistically significant.

RESULTS

Out of the 73 candidates, 41 were female and 32 were male giving a female: male ratio of 1.3:1. Using paired sample t test, there were significant differences between the mean score in long case (mean = 52.86, standard deviation [SD] = 4.315) and mean score in clinical OSCE (mean = 58.356, SD = 7.906), t (72) = -7.181, P = 0.000; mean score in short case (mean = 52.86, SD = 4.097) and mean score in picture OSCE (mean = 48.580, SD = 8.992, t (72) =4.558, P = 0.000; no significant difference between the mean total score in internal medicine clinicals (mean = 105.712, SD = 6.680) and mean total score in surgery clinicals (mean = 106.915, SD = 15.846), t (72) = -0.788, P = 0.433. The Cronbach's alpha for traditional examination and OSCE was 0.437 and 0.863, respectively.

CONCLUSION

OSCE gives a similar mean score to traditional method, but OSCE is more reliable.

Visions and reality: the idea of competence-oriented assessment for German medical students is not yet realised in licensing examinations

Objective: Competence orientation, often based on the CanMEDS model, has become an important goal for modern curricula in medical education. The National Competence Based Catalogue of Learning Objectives for Undergraduate Medical Education (NKLM) has been adopted in Germany. However, it is currently unknown whether the vision of competence orientation has also reached the licensing examination procedures. Methods: Therefore, a prospective, descriptive, single-centre, exemplary study design was applied to evaluate 4051 questions/tasks (from 28 examiners at 7 two-day licensing oral-practical exams) for undergraduate medical students at the University of Ulm. The oral and practical questions/tasks as well as the real bedside assessment were assigned to specific competence roles (NKLM section I), categories (NKLM section II) and taxonomy levels of learning domains. Results: Numerous questions/tasks were set per candidate (day 1/2: 70±24/86±19 questions) in the licensing oral-practical exam. Competence roles beyond the "medical expert" were scarcely considered. Furthermore, practical and communication skills at the bedside were hardly addressed (less than 3/15 min). Strikingly, there was a significant predominance of questions with a low-level taxonomy. Conclusions: The data indicate a misalignment of competence-oriented frameworks and the "real world" licensing practical-oral medical exam, which needs improvement in both evaluation and education processes.

The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective

The Objective Structured Clinical Examination (OSCE) was first described by Harden in 1975 as an alternative to the existing methods of assessing clinical performance (Harden et al. 1975). The OSCE was designed to improve the validity and reliability of assessment of performance, which was previously assessed using the long case and short case examinations. Since then the use of the OSCE has become widespread within both undergraduate and postgraduate clinical education. We recognise that the introduction of the OSCE into an existing assessment programme is a challenging process requiring a considerable amount of theoretical and practical knowledge. The two parts of this Guide are designed to assist all those who intend implementing the OSCE into their assessment systems. Part I addresses the theoretical aspects of the OSCE, exploring its historical development, its place within the range of assessment tools and its core applications. Part II offers more practical information on the process of implementing an OSCE, including guidance on developing OSCE stations, choosing scoring rubrics, training examiners and standardised patients and managing quality assurance processes. Together we hope these two parts will act as a useful resource both for those choosing to implement the OSCE for the first time and also those wishing to quality assure their existing OSCE programme.

Looking back to the future: a message for a new generation of medical educators

OBJECTIVE

Many changes in medical education have occurred during the author's 50-year career in the field. The aim of this paper is to describe 10 lessons worth recording for others engaged in the training of health care professionals.

THREE CAREER PHASES

The first phase in the author's career occurred during 1960-1972 in Glasgow, where an interest in medical education developed alongside an engagement in clinical medicine and research into iodine metabolism. The second phase took place during 1972-2002 in Dundee, where, after working for a period as a clinician, the author made a full-time commitment to medical education. The third phase, from 2002 to the present, has provided the opportunity to explore new horizons in medical education.

CONCLUSIONS

The following lessons have been learned. (i) People are important as role models and collaborators. (ii) Innovation in medical education is a complex process and research findings can easily be misinterpreted. (iii) Nudges, interventions that encourage rather than mandate change, are valuable. (iv) Students are important players in planning, delivering and evaluating a curriculum. Each student has different needs and aspirations. They are the 'digital natives'. (v) Offer stakeholders practical solutions to problems that can be implemented. (vi) There is always something to learn outside one's own practice. Go to a conference or read a journal in a related field. (vii) Time spent recording one's work and publishing reports based on it is rewarding. (viii) Learn from history. We don't need to keep reinventing the wheel. (ix) Obtain independent funding. (x) Finally, and most importantly, have fun. Working in medical education can be exciting, fulfilling and hugely enjoyable.

Objective Structured Clinical Examinations (OSCEs), psychiatry and the Clinical assessment of Skills and Competencies (CASC) same evidence, different judgement

BACKGROUND

The Objective Structured Clinical Examination (OSCE), originally developed in the 1970's, has been hailed as the "gold standard" of clinical assessments for medical students and is used within medical schools throughout the world. The Clinical assessment of Skills and Competencies (CASC) is an OSCE used as a clinical examination gateway, granting access to becoming a senior Psychiatrist in the UK.

DISCUSSION

Van der Vleuten's utility model is used to examine the CASC from the viewpoint of a senior psychiatrist. Reliability may be equivalent to more traditional examinations. Whilst the CASC is likely to have content validity, other forms of validity are untested and authenticity is poor. Educational impact has the potential to change facets of psychiatric professionalism and influence future patient care. There are doubts about acceptability from candidates and more senior psychiatrists.

SUMMARY

Whilst OSCEs may be the best choice for medical student examinations, their use in post graduate psychiatric examination in the UK is subject to challenge on the grounds of validity, authenticity and educational impact.

Defining the correctness of a diagnosis: differential judgments and expert knowledge

Approaches that use a simulated patient case to study and assess diagnostic reasoning usually use the correct diagnosis of the case as a measure of success and as an anchor for other measures. Commonly, the correctness of a diagnosis is determined by the judgment of one or more experts. In this study, the consistency of experts' judgments of the correctness of a diagnosis, and the structure of knowledge supporting their judgments, were explored using a card sorting task. Seven expert pediatricians were asked to sort into piles the diagnoses proposed by 119 individuals who had worked through a simulated patient case of Haemophilus influenzae Type B (HIB) meningitis. The 119 individuals had varying experience levels. The expert pediatricians were asked to sort the proposed diagnoses by similarity of content, and then to order the piles based on correctness, relative to the known correct diagnosis (HIB meningitis). Finally, the experts were asked to judge which piles contained correct or incorrect diagnoses. We found that, contrary to previous studies, experts shared a common conceptual framework of the diagnostic domain being considered and were consistent in how they categorized the diagnoses. However, similar to previous studies, the experts differed greatly in their judgment of which diagnoses were correct. This study has important implications for understanding expert knowledge, for scoring performance on simulated or real patient cases, for providing feedback to learners in the clinical setting, and for establishing criteria that define what is correct in studies of diagnostic error and diagnostic reasoning.

The long case and its modifications: a literature review

CONTEXT

This review provides a summary of the published literature on the suitability of the long case and its modifications for high-stakes assessment.

METHODS

Databases related to medicine were searched for articles published from 2000 to 2008, using the keywords 'long case', 'clinical examinations' and 'clinical assessment'. Reference lists of review articles were hand-searched. Articles related to the objective structured clinical examination were eliminated. Research-based articles with hard data were given more emphasis in this review than those based on opinion.

RESULTS

Eighteen articles were identified. The main disadvantage of the long case is its inability to sample the curriculum widely, resulting in low reliability. The main advantage of the long case is its ability to assess the candidate's overall (holistic) approach to the patient. Modifications to the long case attempt to: structure the format and the marking scheme; increase the number of examiners; observe the candidate's behaviour, and increase the number of cases.

CONCLUSIONS

The long case is a traditional clinical examination format for the assessment of clinical competence and assessment at this level is important. The starting point for the majority of recent research on the long case has been an acceptance of its low reliability and modifications to the format have been proposed. Further evidence of the efficacy of these modifications is required, however, before they can be recommended for summative assessment. If further research is to be undertaken on the long case, it should focus on finding practicable ways of sampling the curriculum widely to increase reliability while maintaining the holistic approach towards the patient, which represents the attraction of the long case.

The introduction and perception of an OSCE with an element of self- and peer-assessment

The purpose of the present study was to encourage reflective dental students by performing an educational Objective Structured Clinical Examination (OSCE) with an element of self- and peer-assessment. An interdisciplinary OSCE comprising cariology, endodontics and microbiology was set up for all third-year students. A blueprint secured representation of the skills to be tested, i.e. knowledge, interdisciplinary knowledge, communication, clinical reasoning and practical procedures. At each station positive and constructive feedback was given to the students based on predefined criteria. Further, the students received written marks after completion of the OSCE. At one station the feedback and marks were replaced by self- and peer-assessment performed by the students in groups after the OSCE. Afterwards, the 68 students and 8 teachers participating in the OSCE answered a questionnaire on their opinion and perception of the examination. The results showed good correlation between the marks given and the students' perception of task difficulty. Generally, there were no systematic variations in the marks given during the week or by individual assessors at the same station, except for one, as well as agreement with marks of the ordinary clinical assessment. The marks given during self- and peer-assessment differed widely, indicating a need for training in this aspect. The questionnaires revealed a very positive perception of the OSCE from both students and teachers. Thus, the majority found the examination relevant and of educational benefit, capable of improving the learning of the students and useful for assessment purposes. Also, the self- and peer-assessment was found useful by the students. In conclusion, this interdisciplinary OSCE stressing constructive feedback to the students was perceived very positively by students and teachers and recognised for its beneficial possibilities in education and assessment.

The use of a virtual patient case in an OSCE-based exam--a pilot study

BACKGROUND

This study focuses on a skills test based clinical assessment where 118 fourth-year medical students at the four teaching hospitals of Karolinska Institutet participated in the same 12-module OSCE. The goal of one of the twelve examination modules was to assess the students' skills and ability to solve a virtual patient (VP) case (the ISP system), which included medical history taking, lab tests, physical examinations and suggestion of a preliminary diagnosis.

AIMS

The primary aim of this study was to evaluate the potential of a VP as a possible tool for assessment of clinical reasoning and problem solving ability among medical students. The feeling of realism of the VP and its possible affective impact on the student's confidence were also investigated.

METHOD

We observed and analysed students' reactions, engagement and performance (activity log files) during their interactive sessions with the simulation. An individual human assistant was provided along with the computer simulation and the videotaped interaction student/assistant was then analysed in detail and related to the students' outcomes.

RESULTS

The results indicate possible advantages of using ISP-like systems for assessment. The VP was for instance able to reliably differentiate between students' performances but some weaknesses were also identified, like a confounding influence on students' outcomes by the assistants used. Significant differences, affecting the results, were found between the students in their degree of affective response towards the system as well as the perceived usefulness of assistance.

CONCLUSION

Students need to be trained beforehand in mastering the assessment tool. Rating compliance needs to be targeted before VP-based systems like ISP can be used in exams and if such systems would be used in high-stake exams, the use of human assistants should be limited and scoring rubrics validated (and preferably automated).

The Objective Structured Clinical Examination (OSCE) as a determinant of veterinary clinical skills

The Objective Structured Clinical Examination (OSCE) has become an excellent tool to evaluate many elements of a student's clinical skills, especially including communication with the patient (human medicine) or client (veterinary medicine); eliciting clinical information from these conversations; some aspects of the physical examination; and many areas of clinical evaluation and assessment. One key factor is that the examination can be structured to compare different students' abilities.

How should paediatric examiners be trained?

Analysis of both the content and process of examinations is central to planning the appropriate education and training of examiners in paediatric clinical examinations. This paper discusses the case for developing training, reviews the current literature, and suggests the desirable attributes of examiners and the components of a training programme. Potential areas of further research are also considered.

Prevalence of the insulin resistance syndrome in obesity

AIMS

To assess prevalence of the insulin resistance syndrome (IRS: obesity, abnormal glucose homoeostasis, dyslipidaemia, and hypertension) in obese UK children and adolescents of different ethnicities and to assess whether fasting data is sufficient to identify IRS in childhood obesity.

METHODS

A total of 103 obese (BMI >95th centile) children and adolescents 2-18 years of age referred for assessment underwent an oral glucose tolerance test, measurement of fasting lipids, and blood pressure determination. Main outcome measures were prevalence of components of IRS by modified WHO criteria, with IRS defined as > or =3 components (including obesity).

RESULTS

There were 67 girls (65%). BMI z-score ranged from 1.65 to 6.15, with 72% having a z-score > or =3.0. Abnormal glucose homoeostasis was identified in 46% (hyperinsulinism in 40%, impaired fasting glucose in 0.8%, impaired glucose tolerance in 11%). No subjects had silent type 2 diabetes. Dyslipidaemia was identified in 30% and hypertension in 32%. Thirty one per cent had obesity alone, 36% had two components, 28% had three, and 5% had all four components. Birth weight, BMI, and family history of IRS were not associated with risk of IRS. Higher age increased the risk of IRS; however the syndrome was seen in 30% of children under 12 years. The use of fasting glucose and insulin data for identifying IRS had a sensitivity of 88% and specificity of 100%.

CONCLUSIONS

One third of obese children and adolescents have the IRS; however type 2 diabetes is rare. Obese children with the IRS may form a high risk group to whom scarce intervention resources should be targeted. Further work is needed to develop appropriate screening programmes for IRS components in significantly obese children.

Objectivity in objective structured clinical examinations: checklists are no substitute for examiner commitment

PURPOSE

This study explored factors that contribute to objectivity in objective structured clinical examinations (OSCEs). The authors quantified the effect of examiners on interrater reliability and separated this effect from that of station construction, determined the effect of objectification on station reliability and validity, and explored examiner factors that may contribute to interrater reliability.

METHOD

Data came from examiners' mark sheets from four annual OSCEs (1997-2000). The OSCEs were conducted identically and simultaneously at three sites, within the University of Otago medical school in New Zealand, with two examiners at each station. The contribution to interrater correlations of station construction and mark sheet compared with examiners' contribution was partitioned out using a random-effects analysis of variance. For one OSCE, a multiple linear regression was used to determine the independent contributions to interrater reliability of the number of checklist items per mark sheet, examiner experience, and examiner involvement in station construction.

RESULTS

Station construction and mark sheets contributed 10.1% and examiners contributed 89.9% to the variation in interrater reliability. Following multivariate analysis, the number of items per mark sheet was negatively associated, and examiner involvement in station construction was positively associated, with interrater reliability. Examiner experience in examining or in clinical medicine was not associated with interrater reliability. There was a negative, but nonsignificant, correlation between number of items per mark sheet and that station's correlation with the aggregate OSCE mark.

CONCLUSIONS

The contribution of objective mark sheets to objectivity is relatively minor compared with examiners' contribution. Increasing the number of checklist items per mark sheet decreased both reliability and validity. Achieving objectivity requires diligent examiners who are involved in the whole assessment.

Variability in the assessment of advanced life support skills

The number of short 'life support' and emergency care courses available are increasing. Variability in examiner assessments has been reported previously in more traditional types of examinations but there is little data on the reliability of the assessments used on these newer courses. This study evaluated the reliability and consistency of instructor marking for the Resuscitation Council UK Advanced Life Support Course. Twenty five instructors from 15 centres throughout the UK were shown four staged video recorded defibrillation tests (one repeated) and three cardiac arrest simulation tests in order to assess inter-observer and intra-observer variability. These tests form part of the final assessment of competence on an Advanced Life Support course. Significant levels of variability were demonstrated between instructors with poor levels of agreement of 52-80% for defibrillation tests and 52-100% for cardiac arrest simulation tests. There was evidence of differences in the observation/recognition of errors and rating tendencies of instructors. Four instructors made a different pass/fail decision when shown defibrillation test 2 for a second time leading to only moderate levels of intra-observer agreement (kappa=0.43). In conclusion there is significant variability between instructors in the assessment of advanced life support skills, which may undermine the present assessment mechanisms for the advanced life support course. Validation of the assessment tools for the rapidly growing number of life support courses is required with urgent steps to improve reliability where required.

Educational portfolios in the assessment of general practice trainers: reliability of assessors

OBJECTIVES

This paper reports a project that assessed a series of portfolios assembled by a cohort of participants attending a course for prospective general practice trainers.

DESIGN

The reliability of judgements about individual 'components', together with an overall global judgement about performance were studied.

SETTING

NHSE South & West, King Alfred's College, Winchester and Institute of Community Studies, Bournemouth University.

SUBJECTS

Eight experienced general practice trainers recruited from around Wessex, which incorporates Hampshire, Dorset, Wiltshire and the Isle of Wight.

RESULTS

The reliability of individual assessor's judgements (i.e. their consistency) was moderate, but inter-rater reliability did not reach a level which could support making a safe summative judgement. The levels of reliability reached were similar to other subjective assessments and perhaps reflected individuality of personal agendas of both the assessed and the assessors, and variations in portfolio structure and content.

CONCLUSIONS

Suggestions for approaches in future are made.

Testing the raters: inter-rater reliability of standardized anaesthesia simulator performance

PURPOSE

Assessment of physician performance has been a subjective process. An anaesthesia simulator could be used for a more structured and standardized evaluation but its reliability for this purpose is not known. We sought to determine if observers witnessing the same event in an anaesthesia simulator would agree on their rating of anaesthetist performance.

METHODS

The study had the approval of the research ethics board. Two one-hour clinical scenarios were developed, each containing five anaesthetic problems. For each problem, a rating scale defined the appropriate score (no response to the situation: score = 0; compensating intervention defined as physiological correction: score = 1; corrective treatment: defined as definitive therapy score = 2). Video tape recordings, for assessment of inter-rater reliability, were generated through role-playing with recording of the two scenarios three times each resulting in a total of 30 events to be evaluated. Two clinical anaesthetists, uninvolved in the development of the study and the clinical scenarios, reviewed and scored each of the 30 problems independently. The scores produced by the two observers were compared using the kappa statistic of agreement.

RESULTS

The raters were in complete agreement on 29 of the 30 items. There was excellent inter-rater reliability (= 0.96, P < 0.001).

CONCLUSIONS

The use of videotapes allowed the scenarios to be scored by reproducing the same event for each observer. There was excellent inter-rater agreement within the confines of the study. Rating of video recordings of anaesthetist performance in a simulation setting can be used for scoring of performance. The validity of the scenarios and the scoring system for assessing clinician performance have yet to be determined.

Performance of the vascular physical examination by residents and medical students

PURPOSE

This study uses an objective structured clinical examination to evaluate the performance and interpretation of the vascular physical examination by interns and medical students.

METHODS

A patient with lower extremity arterial occlusive disease findings was examined by 20 third-year students (M3), 23 physicians entering surgical internship (PGY1), and 7 individuals completing internship (PGY2). The test consisted of two sections: part A evaluated the individuals' ability to perform a pulse examination (data gathering); part B evaluated the interpretation of the physical examination findings (data interpretation). National Board of Medical Examiners Part II examination results were obtained for 84% of participants.

RESULTS

All groups performed poorly, with overall correct percent scores being 43% (M3), 39% (PGY1), and 62% (PGY2). PGY2s performed significantly better than M3s or PGY1s (p = 0.0002). No statistical difference was noted between M3 and PGY1 scores. Overall, data gathering skills were significantly better than data interpretation skills (51% vs 37%, p = 0.0001). National Board of Medical Examiners Part II scores did not vary substantially among groups.

CONCLUSIONS

Interns and medical students demonstrated considerable inaccuracy in both data gathering and data interpretation. A modest improvement was observed in individuals tested at the end of the internship year. This study suggests that increased attention should be directed toward instructing surgical residents and students how to perform an accurate peripheral vascular physical examination and how to interpret its significance.

The oral examination: a study of academic and non-academic factors

The oral examination in psychiatry for final-year medical students at Wellington and Dunedin School of Medicine, University of Otago, was studied. Between December 1989 and April 1990, 40 medical students were videorecorded during such an examination. The transcripts of the recording of each oral, and at a later date the videorecordings, were individually scored by a panel of six research psychiatrists who were experienced examiners. In addition verbal and non-verbal behaviour was rated using visual analogue scales and the students completed personality and anxiety questionnaires. There was a low level of agreement between research psychiatrists in the allocation of oral marks. The oral score was positively associated with the level of confidence of the student and negatively with anxiety in men.

The assessment of clinical competence in obstetrics and gynaecology in two medical schools by an objective structured clinical examination

OBJECTIVE

To compare and assess clinical competence among final year students in two British medical schools using a standardised objective structured clinical examination (OSCE) in obstetrics and gynaecology.

DESIGN

A comparative study of an OSCE with stations designed to test student abilities in history-taking, physical examination, interpretation of data or results, interpersonal skills, practical procedures, as well as factual knowledge.

SUBJECTS

Two groups of final year medical students from two British medical schools.

METHODS

An OSCE of 26 stations.

RESULTS

The standardised OSCE was simple to organise and conduct once the content had been decided. Analysis of the results revealed significant differences in mean marks between the two sets of students in six stations, and in eight stations the mean score was less than five for one or other sets of students. The reasons for the differences between the medical schools and the poor performance of the students at some stations were investigated and possible causative factors identified.

CONCLUSIONS

We have shown that an OSCE is suitable for testing clinical competence of students within and across medical schools, is able to highlight differences in standards between institutions, and can identify areas where teaching methods and/or course content are deficient.

The structured oral examination as a method for assessing surgical residents

The purpose of this study was to determine the overall reliability, inter-rater reliability, and criterion validity of the structured oral examination (SOE) for assessing surgical residents. An SOE consisting of four predetermined clinically oriented scenarios was administered to 23 second postgraduate year surgical residents. Each scenario had five to six questions, each with a specific marking scheme. Candidates were assessed by two examiners and scores were derived independently. Overall reliability (Cronbach's alpha) was 0.75. Inter-rater reliability was significant for each pair of examiners and each question (r = 0.78 to 0.91: p less than 0.0001). Criterion validity was measured by correlating SOE scores with multiple-choice examination (MCQ) and objective structured clinical examination (OSCE) scores. Correlations between the SOE and MCQ and OSCE were significant and fell into the moderate range (0.48 to 0.51). The results of this study show that the SOE is useful in the assessment of clinical knowledge and problem-solving abilities of the surgical resident. Overall and inter-rater reliabilities achieved exceed those of traditional oral examination formats.

A comparative study of the traditional long case with the objective structured clinical examination in Lagos, Nigeria

An apparent difference in the results of the clinical examination of the final M.B., B.S. was observed following replacement of the traditional long case (TLC) with the Objective Structured Clinical Examination (OSCE) in 1979. This led to a study of the results of two consecutive years of each method, 1976 and 1978 (TLC), and 1979 and 1980 (OSCE). The OSCE pass rate of 61% was found to be significantly lower than TLC pass rate of 93% (P less than 0.05). Using the analysis of variance and the critical difference (CD) of the mean scores of the different types of examination, no significant difference was found to exist between the two TLC examinations or between the two OSCE examinations. However, significant difference exists between the TLC of 1978 and the OSCE of 1979, P less than 0.05. A comparison of the CD of MCQ to those of TLC and OSCE suggests that less differences exist between MCQ and OSCE scores compared to MCQ and TLC scores, and by 1980 no significant difference exists between MCQ and OSCE. OSCE, like MCQ, will therefore appear an acceptable method of examination and perhaps a more effective method of clinical examination than TLC.

Final examinations in surgery: support for the retention of the clinical examination

Students' results in the final M.B., B.S. (W.A.) examination in surgery between 1976 and 1979 inclusive (375 students) were analysed. Marks in all parts of the examination, namely clinical, multiple choice, essays, pathology, and fifth year multiple choice, were correlated with each other and the total mark in surgery. Multivariate analysis showed that the marks in the clinical examination accounted for most of the variation in a student's total mark, and this was only in part due to its heavy weighting in Western Australia. The authors of this paper strongly support the retention of the clinical examination in surgery and consider that the loading placed on this part of the final examination in Western Australia is justified.

The selection and training of examiners for clinical examinations

The inconsistency of the marking in clinical examinations is a well documented problem. This project identified some of the factors responsible for this inconsistency. A standardized rating situation was devised. Five students were videotaped as they performed part of a physical examination on simulated patients. Eighteen experienced medical and surgical examiners rated their performances using an objective checklist type of rating form. No differences were evident between physicians and surgeons. The group of examiners was divided into three subgroups, one receiving no training, one limited training and one more extensive training. Examiners re-rated the same students 2 months after the first rating. Inter-rater reliability was satisfactory for the first ratings and training produced no significant improvement. A substantial improvement was achieved by identifying the most inconsistent raters and removing them from the analysis. Training was shown to be unnecessary for consistent examiners and ineffective for examiners who were less consistent. On the basis of these results, only consistent examiners were selected to take part in the interactive component of the objective structured final year examinations. The ratings in these examinations achieved high levels of inter-rater reliability. It was concluded that the combination of an objective check-list rating form, a controlled test situation and the selection of inherently consistent examiners could solve the problem of inconsistent marking in clinical examinations.

Assess clinical competence-an overview

Assessment of a doctor's or student's clinical competence is of key importance. The examiner should ask what competencies he is attempting to assess and how he can best assess them; whether, for example, by using an objective structured clinical examination (OSCE), patient management problems (PMP), supervisors' reports, project work or assignments, or by auditing case records. He should consider the functions of the clinical examination, in particular its role in providing feedback to the student and teacher. With regard to timing there is a place for both in-course and end-of-course assessment of clinical competence. Finally, in considering who should undertake the assessment, the role of the student or doctor in assessing his own competence should not be overlooked.

Assessment of clinical competence using objective structured examination

To avoid many of the disadvantages of the traditional clinical examination we have introduced the structured clinical examination. In this students rotate round a series of stations in the hospital ward. At one station they are asked to carry out a procedure, such as take a history, undertake one aspect of physical examination, or interpret laboratory investigations in the light of a patient's problem, and at the next station they have to answer questions on the findings at the previous station and their interpretation. As they cannot go back to check on omissions multiple-choice questions have a minimal cueing effect. The students may be observed and scored at some stations by examiners using a check list. In the structured clinical examination the variables and complexity of the examination are more easily controlled, its aims can be more clearly defined, and more of the student's knowledge can be tested. The examination is more objective and a marking strategy can be decided in advance. The examination results in improved feed-back to students and staff.