The influence of task formats on the accuracy of medical students' self-assessments

Systematic Review of Computer Based Assessments in Medical Education

Medical schools, postgraduate training institutes, licensing and certification bodies have developed and implemented many new methods for accurate, reliable, and timely assessments of the competence of medical professionals and practicing physicians. The underlying objective of all these assessments is to not only evaluate the students' learning and educational goals but also to establish the graduating individual's skills and professionalism. Computer based assessment (CBA) has emerged in recent years as a viable alternative to traditional assessment techniques. It has also infiltrated and influenced the medical curriculum where it has been employed for assessment tasks. This study presents how CBA offers pedagogical opportunities and analyzes its usage pattern over the past three decades. We examined 47 CBAs in medical education and analyzed several assessment components, including application area, assessment purpose, assessment type, assessment format, student level, and emphasized the interplay among these components. Our analysis determined that formative assessment is the most frequently used type and 75% of all assessment types employed used the multiple choice questions format.

A pilot study of orthopaedic resident self-assessment using a milestones' survey just prior to milestones implementation

OBJECTIVE

To pilot test if Orthopaedic Surgery residents could self-assess their performance using newly created milestones, as defined by the Accreditation Council on Graduate Medical Education.

METHODS

In June 2012, an email was sent to Program Directors and administrative coordinators of the 154 accredited Orthopaedic Surgery Programs, asking them to send their residents a link to an online survey. The survey was adapted from the Orthopaedic Surgery Milestone Project. Completed surveys were aggregated in an anonymous, confidential database. SAS 9.3 was used to perform the analyses.

RESULTS

Responses from 71 residents were analyzed. First and second year residents indicated through self-assessment that they had substantially achieved Level 1 and Level 2 milestones. Third year residents reported they had substantially achieved 30/41, and fourth year residents, all Level 3 milestones. Fifth year, graduating residents, reported they had substantially achieved 17 Level 4 milestones, and were extremely close on another 15. No milestone was rated at Level 5, the maximum possible. Earlier in training, Patient Care and Medical Knowledge milestones were rated lower than the milestones reflecting the other four competencies of Practice Based Learning and Improvement, Systems Based Practice, Professionalism, and Interpersonal Communication. The gap was closed by the fourth year.

CONCLUSIONS

Residents were able to successfully self-assess using the 41 Orthopaedic Surgery milestones. Respondents' rate improved proficiency over time. Graduating residents report they have substantially, or close to substantially, achieved all Level 4 milestones. Milestone self-assessment may be a useful tool as one component of a program's overall performance assessment strategy.

Evaluation of the validity of medical students' self-assessments of proficiency in clinical simulations

BACKGROUND

The accuracy of self-assessments has not been well supported in the literature. This study was undertaken to examine the validity of medical students' ratings of their proficiency during encounters with simulated patients and simulation devices.

METHODS

Confidential self-assessments for 10 skills were collected from 195 students during a formal clinical skills assessment related to 3 cases at the end of a surgery clerkship. The cases required students to gather data from simulated patients and perform procedures such as rectal examinations, nasogastric tube insertions, and suturing on bench simulation models. The patients were trained to assess student performance.

RESULTS

There were significant differences between student self-assessments and simulated patient scores for general clinical skills as opposed to procedural skills. Students' mean self-assessments in the data gathering and interpersonal skills were 2-6 % points higher than ratings of their proficiency by simulated patients. However, self-assessments on procedures were 5-8 points lower than patient ratings. The median correlation between self-assessments and patient ratings for general clinical skills such as data gathering and interpersonal skills was 0.08 (not significant), whereas the median correlation between student and patient ratings in procedures was 0.22 (P < 0.01).

CONCLUSIONS

Third-year medical students' self-assessments for specific procedures are more valid than self-assessments of general clinical skills. Students are less confident in their procedural skills compared with general clinical skills. Although self-assessments should not be used as the sole measure of performance in clinical simulations, self-assessments for specific procedures can provide supplemental information on proficiency.

Can endoscopists accurately self-assess performance during simulated colonoscopic polypectomy? A prospective, cross-sectional study

BACKGROUND

The aim of this study was to establish if endoscopists can reliably self-assess their ability to perform simulated colonic polypectomy.

METHODS

Novices, intermediates, advanced, and experts performed a video-recorded polypectomy task using the Welsh Institute for Minimal Access Therapy (WIMAT) colonoscopy suitcase simulator. This involved removal of a simple polyp (A) and a complex polyp (B). Participants self-assessed themselves using a Direct Observation of Polypectomy Skills (DOPyS) assessment form. Two blinded, independent, Joint Advisory Group on Gastrointestinal Endoscopy (JAG) accredited assessors graded each performance using the same DOPyS scoring. The Spearman coefficient was used to determine the correlation between self and assessors' scores.

RESULTS

Eighty participants completed the task. There was a weak correlation between assessors' scores and self-assessment scores for all groups (novices: ρ = -.44, P = .85; intermediates: ρ = -.16, P = .51; advanced: ρ = .16, P = .50; and experts: ρ = .07, P = .76). There was a strong correlation between scores from assessor 1 and 2 for polyp A (ρ = .80, P ≤ .01) and polyp B (ρ = .80, P ≤ .01).

CONCLUSIONS

The correlation between self-assessment and assessors' scores is weak. Novices and intermediates underestimate performance, whereas advanced and experts overestimate performance. Regular feedback may improve accuracy.

Medical student self-reported confidence in obstetrics and gynaecology: development of a core clinical competencies document

BACKGROUND

Clinical competencies in obstetrics and gynaecology have not been clearly defined for Australian medical students, the growing numbers of which may impact clinical teaching. Our aim was to administer and validate a competencies list, for self-evaluation by medical students of their confidence to manage common clinical tasks in obstetrics and gynaecology; to evaluate students' views on course changes that may result from increasing class sizes.

METHODS

A draft list of competencies was peer-reviewed, and discussed at two student focus groups. The resultant list was administered as part of an 81 item online survey.

RESULTS

Sixty-eight percent (N = 172) of those eligible completed the survey. Most respondents (75.8%) agreed or strongly agreed that they felt confident and well equipped to recognise and manage most common and important obstetric and gynaecological conditions. Confidence was greater for women, and for those who received a higher assessment grade. Free-text data highlight reasons for lack of clinical experience that may impact perceived confidence.

CONCLUSIONS

The document listing competencies for medical students and educators is useful for discussions around a national curriculum in obstetrics and gynaecology in medical schools, including the best methods of delivery, particularly in the context of increasing student numbers.

Telehomecare communication and self-care in chronic conditions: moving toward a shared understanding

BACKGROUND

Remote telemonitoring of patients' vital signs is a rapidly increasing practice. Although methods of communication in remote electronic monitoring differ from those in traditional home health care, the understanding shared by the nurse, patient, and family members remains the same: patients' self-care behaviors affect exacerbations of chronic health conditions. The purpose of this paper is to examine the relationship between communication and information integration into the daily lives of patients with chronic illnesses and offer best practice recommendations for telehomecare nurses (THN).

METHODS

The original study used the Social Relations Model to examine relationships within 43 triads composed of patients with chronic conditions, home helpers, and their nurse involved in telehomecare at three Veterans Health Administration sites. This secondary descriptive and correlational analysis compared 43 patients' and nine THNs' ratings of themselves and each other on communication (frequency, timeliness, and understanding) and the use of patients' daily telemonitored information.

RESULTS

There was almost no correlation between patients' perception of THNs' communication (frequency [r=0.05], timeliness [r=0.09], and understandability [r=0.03]) and patients' integration of information into daily health practices. However, significant correlations were found between the THNs' perception of patients' communication frequency and timeliness, and integration, (p=0.02; p<0.001) respectively.

CONCLUSIONS

This study suggests that frequent phone communication may lead the remote THN to believe patients are integrating blood pressure, weight, and other information into daily self-care behaviors, when in fact the patient reports that they are not. The influence of a halo effect on the THN may cloud an accurate perception of what is actually occurring. Remote communication may require more attention to THNs educating patients about shared understandings when using telemonitoring. Best practices for THN should include explicit goals and intentions for telemonitored home care with individualized instructions about how to use the information for self-care.

Medical students' self-assessment of performance: results from three meta-analyses

OBJECTIVE

Self-assessment is an important component of medical education. Meta-analyses were conducted to better understand accuracy of self-assessment and direction of inaccuracy.

METHODS

Three meta-analyses were conducted on results from 35 published articles on medical student self-assessment, one for each of the theoretically distinct ways of measuring accuracy of self-reported ability (correlational, paired comparison, and independent means comparison). Characteristics that potentially influence self-assessment accuracy, including gender, year in medical school, and type of self-assessment, were examined.

RESULTS

Students are moderately able to self-assess performance and are more accurate later in medical school. Students as a whole do not significantly over- or underestimate, but are more likely to overestimate on communication-based, standardized patient encounters than objective, knowledge-based performance measures. Female students underestimate their performance more than male students, but gender analyses are often unreported.

CONCLUSION

A deeper understanding of the causes and consequences of over- and underestimation is impossible without measurement and reporting of the direction of inaccuracy.

PRACTICE IMPLICATIONS

To improve our understanding of self-assessment and increase its effectiveness as a teaching tool, research should report self-assessment as both a correlation and a paired comparison, and conduct analyses of important moderators that can influence self-assessment accuracy.

The analysis of self and tutor assessment in the skill of basic life support (BLS) and endotracheal intubation: focused on the discrepancy in assessment

INTRODUCTION

This paper reports the results of a study of fourth year medical students that assessed whether assessments of basic life support (BLS) and intubation performance differ when assessed by the students themselves or by tutors. This information should be helpful for designing the contents of a complementary education core.

METHODS

Tutor assessments and student-assessments were conducted using a checklist and a fivepoint rating scale, and then compared. For the two skill performance tests, Resusci(®) Anne SkillGuide™ and Laerdal(®) Airway Management Trainer (Laerdal, Norway) devices were used. The check-lists used to evaluate students were based on International Liasion Committee on Resuscitation (ILCOR) guidelines and Korean Emergency Airway Management Society (KEAMS) tutor guidelines.

RESULTS

A total of 83 medical students participated in the study, intra-class correlation coefficient between tutor and student assessment were 0.542 (95% CI 0.371-0.678) in BLS and 0.693 (0.538-0.802). There were also no significant differences between self-assessments and tutor assessments based on the five-point. In BLS skill session, we found out that "maintenance of airway" and "palpating a carotid pulse" were the mostly missed parts. In the intubation skill, omitting the parts of 'securing the airway' while preparing for intubation, proper positioning of blade tip in the valleculae, and appropriate insertion of endotracheal tube were demonstrated.

CONCLUSION

We observed correlations between student self-assessments and tutor assessments for both BLS and intubation. Analyzing the discrepancies between self-assessment and tutor assessment will be helpful in focusing training on the steps that were omitted by students or during which students demonstrated incompetence.

Influences on self-evaluation during a clinical skills programme for nurses

Education has moved from teacher to student-centred practices. Increasing emphasis is placed on 'life-long' learning in the context of a rapidly changing knowledge base. Self-evaluation is seen as one strategy to facilitate student-centred continuous professional development. The literature, however, suggests that learners' ability to self-assess is mixed, and little is known regarding how students perform self-assessment. This study focussed on senior nurses undertaking a scenario-based clinical skills course. Learners were asked to self-evaluate several times during the course. This research explored the influences on using the self-evaluation exercise. The study drew upon grounded theory methodology and was influenced by constructionist and postmodernist theories. Three methods of data collection were used: semi-structured interviews, observation of supervision sessions and recording of the numerical self-evaluation ratings. Multiple interviews with students (n = 14) and the educational supervisor (n = 1) were conducted. Thematic analysis and data collection were conducted iteratively. The study found that feeling confident and stating that confidence were not necessarily the same. Feeling confident was complex, influenced by changing perceptions of clinical skills and credibility. Changing frames of reference were used to judge feelings of confidence. Stating confidence appeared to be socially negotiated, influenced by social acceptability considerations such as modesty and the need to show progress over time. The discourses of empowerment and surveillance were influential and self-evaluation is discussed using Foucault's theory of governmentality, illustrating how learners can be both empowered and controlled through self-evaluation. Further consideration of the socially constructed nature of self-evaluations would benefit both educational practice and future research.

Remediating students' failed OSCE performances at one school: the effects of self-assessment, reflection, and feedback

PURPOSE

To investigate whether and how use of an online remediation system requiring reflective review of performance and self-assessment influenced students' performance on objective structured clinical examination (OSCE) station repeats (subsequent to failure on the first attempt) and their self-assessments of their performance (between the first and second attempts).

METHOD

Fourth-year medical students' performances on seven OSCE stations were videotaped at University of Michigan Medical School in 2006. Failing students took the exam again; remediation included self-assessment and review, plus faculty guidance for failures that were greater than one standard error of measurement of the distribution. A total of 1,171 possible observations of students' actual performance and performance self-assessments were analyzed using independent and dependent t tests and within-subjects ANOVA.

RESULTS

Results indicate statistically significant changes in students' performance between first and second attempts and statistically significant improvements in self-assessment between first and second attempts. No significant changes were found between self-assessed and faculty-guided remediation.

CONCLUSIONS

This study provides evidence that OSCE remediation combining review, reflection, and self-assessment has a salutary effect on (subsequent) performance and self-assessment of performance.

Medical students' assessment of education and training in women's health and in sex and gender differences

BACKGROUND

The authors surveyed U.S. medical students to learn their perceptions of the adequacy of women's health and sex/gender-specific teaching and of their preparedness to care for female patients.

METHODS

Between September 2004 and June 2005, third and fourth year students at the 125 allopathic medical schools received an online survey conducted by the American Medical Women's Association (AMWA). Students rated the extent to which 44 topics were included in curricula from 1 to 4 (1 = no coverage, 4 = in-depth coverage) and their preparedness to perform 27 clinical skills (1 = no preparation, 4 = thorough preparation).

RESULTS

From 101 of the 125 schools, 1267 students responded (mean number of respondents/school = 13, SD 12). The mean curriculum rating (2.53, SD 0.52) indicated brief to moderate coverage of topics. The mean preparedness rating was higher (3.09, SD 0.44), indicating moderate preparedness. In a regression model, female student sex and site of an AMWA chapter were associated with lower mean combined curriculum and preparedness ratings (female 2.76, male 3.01, p < 0.001; AMWA 2.77, non-AMWA 2.89, p < 0.001), whereas other school characteristics (female dean, federally funded women's health program, and proportion of tenured women faculty) had no association.

The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10

BACKGROUND

Health professionals are increasingly expected to identify their own learning needs through a process of ongoing self-assessment. Self-assessment is integral to many appraisal systems and has been espoused as an important aspect of personal professional behaviour by several regulatory bodies and those developing learning outcomes for clinical students. In this review we considered the evidence base on self-assessment since Gordon's comprehensive review in 1991. The overall aim of the present review was to determine whether specific methods of self-assessment lead to change in learning behaviour or clinical practice. Specific objectives sought evidence for effectiveness of self-assessment interventions to: a. improve perception of learning needs; b. promote change in learning activity; c. improve clinical practice; d. improve patient outcomes.

METHODS

The methods for this review were developed and refined in a series of workshops with input from an expert BEME systematic reviewer, and followed BEME guidance. Databases searched included Medline, CINAHL, BNI, Embase, EBM Collection, Psychlit, HMIC, ERIC, BEI, TIMElit and RDRB. Papers addressing self-assessment in all professions in clinical practice were included, covering under- and post-graduate education, with outcomes classified using an extended version of Kirkpatrick's hierarchy. In addition we included outcome measures of accuracy of self-assessment and factors influencing it. 5,798 papers were retrieved, 194 abstracts were identified as potentially relevant and 103 papers coded independently by pairs using an electronic coding sheet adapted from the standard BEME form. This total included 12 papers identified by hand-searches, grey literature, cited references and updating. The identification of a further 12 papers during the writing-up process resulted in a total of 77 papers for final analysis.

RESULTS

Although a large number of papers resulted from our original search only a small proportion of these were of sufficient academic rigour to be included in our review. The majority of these focused on judging the accuracy of self-assessment against some external standard, which raises questions about assumed reliability and validity of this 'gold standard'. No papers were found which satisfied Kirkpatrick's hierarchy above level 2, or which looked at the association between self-assessment and resulting changes in either clinical practice or patient outcomes. Thus our review was largely unable to answer the specific research questions and provide a solid evidence base for effective self-assessment. Despite this, there was some evidence that the accuracy of self-assessment can be enhanced by feedback, particularly video and verbal, and by providing explicit assessment criteria and benchmarking guidance. There was also some evidence that the least competent are also the least able to self-assess accurately. Our review recommends that these areas merit future systematic research to further our understanding of self-assessment.

CONCLUSION

As in other BEME reviews, the methodological issues emerging from this review indicate a need for more rigorous study designs. In addition, it highlights the need to consider the potential for combining qualitative and quantitative data to further our understanding of how self-assessment can improve learning and professional clinical practice.

Critical care recognition, management and communication skills during an emergency medicine clerkship

BACKGROUND

Medical students need to learn how to recognize and manage critically ill patients; to communicate in critical situations with patients, families, and the healthcare team; and finally, to integrate technical knowledge with communication skills in caring for these patients. Meeting their needs will help prepare them to demonstrate, as physicians, the ability to synthesize information while simultaneously caring for patients, that the American Medical Association recently characterized as vital.

AIMS

Responding to these needs, we developed and implemented a curriculum to enable students in a required emergency medicine clerkship to recognize, manage, and simultaneously communicate with critically ill patients.

METHODS

The curriculum consisted of lectures and exercises on caring for the critically ill including: an introduction to the systematic approach; an interactive lecture on comprehensive communication; observation and discussion of real patients in the emergency department; participation in a single standardized patient encounter while peers and a faculty member observed them; assessment of students' own videotaped performance of the examination by using critical care and communication/interpersonal skills checklists; and receipt of private feedback based on the checklists from the faculty and the standardized patient. Students evaluated the curriculum at the end of the clerkship.

RESULTS

Complete performance data for 46 students and curriculum evaluation data from 42 students were available. According to faculty assessment, students as a group performed 79.6% (SD 0.15) of the critical care and 70.9% (SD 11.5%) of the communication skills. Students most often demonstrated Basic Interpersonal Skills (97.9%, SD 0.056) and least often demonstrated Empathy skills (41.7%, SD 0.235). Students rated the curriculum positively.

CONCLUSIONS

It is feasible to integrate the teaching of communication skills with the recognition and management of critically ill patients. The next step will be to revise the curriculum to address student deficiencies and to evaluate its effectiveness more rigorously.

Growth of self-perceived clinical competence in postgraduate training for general practice and its relation to potentially influencing factors

OBJECTIVE

To examine the increase in self-perceived clinical competence during a three-year postgraduate training in general practice and to explore the relation between the growth of self-perceived competence and several background variables.

DESIGN

Cohort, 1995-1998.

SETTING

Three-year Postgraduate Training for General practice in the Netherlands

PARTICIPANTS

All Dutch trainees who followed postgraduate training from September 1995 to September 1998 (N=191).

INTERVENTION

We asked the trainees at the start and at the end of their postgraduate training to complete a questionnaire, which assessed their self-perceived knowledge, clinical skills and consultations skills. We collected information about potentially influencing background variables. Amongst these were variables such as: age, gender, prior medical experience, the effort someone has spent upon her/his education, insight in weak and strong areas of clinical competence and knowledge and skills levels.

MAIN OUTCOME MEASURE

Self-perceived competence.

RESULTS

A total of 127 trainees completed both questionnaires (190 at the first administration and 128 at the second one). We found statistically significant growth of self-perceived clinical competence. Self-perceived consultation skills increased more than self-perceived knowledge and clinical skills. The afore mentioned background variables did not relate in any way with the growth of self-perceived clinical competence.

CONCLUSION

This study shows that a 3-year postgraduate training in general practice enhances self-perceived clinical competence. However, we still do not know how to explain this improvement. Further study into the theoretical concept of self-assessment in medical education and into the factors contributing to the feeling of being competent, is required.

Self and peer assessment of pediatricians, psychiatrists and medicine specialists: implications for self-directed learning

Self-regulation in medicine depends on accurate self-assessment. The purpose of the present study was to examine the discrepancy between self and peer assessments for a group of specialist physicians from internal medicine (IM), pediatrics, and psychiatry clinical domains (i.e., patient management, clinical assessment, professional development, and communication). Data from 304 psychiatrists, pediatricians and internal medicine specialists were used. Each physician had data from an identical self and 8 peer (38 item/4 clinical domains assessment). A total of 2306 peer assessments were available. Physicians were classified into quartiles based on mean assessment peer data and compared with self-assessment data. The analyses showed that self and peer assessment profiles were consistent across specialties and domains. Physicians assessed in the lowest and highest quartiles (i.e., <25th and >75th) by colleagues tended to rate themselves 30-40 percentile ranks higher and lower than peers, respectively. This study suggests that practicing physicians are inaccurate in assessing their own performance. These data suggest that systems to provide practicing physicians with regular and routine feedback may be appropriate if we are to ensure physicians are able to accurately assess themselves in a profession in which self-regulation is predicated upon the assumption that physicians know their capabilities and limitations.

Relationships between self-assessment skills, test performance, and demographic variables in psychiatry residents

Some researchers have seen the capacity for self-assessment in trainees as a special skill, and some reports have concluded that this skill is positively and crucially correlated with academic competence. Thus, it is believed that those trainees who are most deficient in knowledge are least likely to be aware of their limitations. Other researchers have emphasized the impact of statistical regression and other technical considerations in the studies, which have led to these conclusions. Our study used a relative-ranking design to measure the accuracy of self-assessments of both strengths and weaknesses in psychiatry residents. We analyzed the relationships between indices of self-assessment accuracy and other resident characteristics, particularly current academic strength as measured by a standard test of psychiatric knowledge. A total of 56 residents in two general psychiatry programs evaluated their performance on the Psychiatry Resident in Training Examination by estimating the rank order of their scores in the 11 psychiatry subject areas. For each resident, actual examination results were then used to generate measures of the accuracy of the identification of strengths and weaknesses. Residents' identifications of their strengths and weaknesses were significantly more accurate than chance levels. Strengths and weaknesses were identified with roughly equal proficiency, and accuracy in these assessments was not correlated to any of the following variables: academic competence as measured by examination raw scores, postgraduate year, gender, international vs. American medical education, program membership, or age. Our results do not support the hypothesis that trainees who show the least academic mastery also make the most inaccurate self-assessments. In addition, we found no resident characteristics that accounted for variation in self-assessment accuracy.

Enhancing self-awareness in medical students: an overview of teaching approaches

Self-awareness is an individual's tendency to pay attention to his or her own emotions, attitudes, and behavior in response to specific situations. In the case of physicians, self-awareness is their insight into how their emotional makeup influences patient care. Conceivably, such insight may improve doctors' professional performance. The authors review published approaches aimed at enhancing the self-awareness of medical students and draw attention to some problems in these approaches that call for further research. Published teaching programs of self-awareness may be classified as direct or indirect. The primary objective of direct programs is to promote students' insight into their own feelings and attitudes by classroom instruction or small-group discussions, during which students share with their peers their emotional responses to various clinical experiences. The primary objective of indirect approaches is to teach clinical skills, such as patient interviewing, patient counseling, and self-assessment. It has been claimed that these programs also enhance self-awareness by drawing students' attention to differences between students' assessment of their own performance and the assessments of their instructors and patients. Both types of programs should be given consideration for inclusion into the medical curriculum. However, since presently available evidence does not allow educators to identify an optimal teaching program, more study is needed concerning the effectiveness of the various approaches to teaching self-awareness. Specifically, an effort should be made to ascertain that the benefit of the direct approaches exceeds their cost in terms of time, teacher training, and-possibly-student embarrassment.

A longitudinal study of self-assessment accuracy

AIM

Although studies have examined medical students' ability to self-assess their performance, there are few longitudinal studies that document the stability of self-assessment accuracy over time. This study compares actual and estimated examination performance for three classes during their first 3 years of medical school.

METHODS

Students assessed their performance on classroom examinations and objective structured clinical examination (OSCE) stations. Each self-assessment was then contrasted with their actual performance using idiographic (within-subject) methods to define three measures of self-assessment accuracy: bias (arithmetic differences of actual and estimated scores), deviation (absolute differences of actual and estimated scores), and covariation (correlation of actual and estimated scores). These measures were computed for four intervals over the course of 3 years. Multivariate analyses of variance and correlational analyses were used to evaluate the stability of these measures.

RESULTS

Self-assessment accuracy measures were relatively stable over the first 2 years of medical school with a decease occurring in the third year. However, the correlational analyses indicated that the stability of self-assessment accuracy was comparable to the stability of actual performance over this same period.

CONCLUSION

The apparent decline in accuracy in the third year may reflect the transition from familiar classroom-based examinations to the substantially different clinical examination tasks of the third year OSCE. However, the stability of self-assessment accuracy compares favorably with the stability of actual performance over this period. These results suggest that self-assessment accuracy is a relatively stable individual characteristic that may be influenced by task familiarity.

Resident self-assessment of operative performance

BACKGROUND

In medicine, the development of expertise requires the recognition of one's capabilities and limitations. This study aimed to verify the accuracy of self-assessment for the performance of a surgical task, and to determine whether self-assessment may be improved through self-observation or exposure to relevant standards of performance.

METHODS

Twenty-six senior surgical residents were videotaped performing a laparoscopic Nissen fundoplication in a pig. Experts rated the videos using two scoring systems. Subjects evaluated their performances after performance of the Nissen, after self-observation of their videotaped performance, and after review of four videotaped "benchmark" performances.

RESULTS

Expert interrater reliability was 0.66 (intraclass correlation coefficient). The correlation between experts' and residents' self-evaluations was initially moderate (r = 0.50, P <0.01), increasing significantly after the residents reviewed their own videotaped performance to r = 0.63 (Deltar = 0.13, P <0.01), yet did not change after review of the benchmarks.

CONCLUSIONS

Self-observation of videotaped performance improved the residents' ability to self-evaluate.