The relationship between competence and performance: implications for assessing practice performance

Should essays and other "open-ended"-type questions retain a place in written summative assessment in clinical medicine?

BACKGROUND

Written assessments fall into two classes: constructed-response or open-ended questions, such as the essay and a number of variants of the short-answer question, and selected-response or closed-ended questions; typically in the form of multiple-choice. It is widely believed that constructed response written questions test higher order cognitive processes in a manner that multiple-choice questions cannot, and consequently have higher validity.

DISCUSSION

An extensive review of the literature suggests that in summative assessment neither premise is evidence-based. Well-structured open-ended and multiple-choice questions appear equivalent in their ability to assess higher cognitive functions, and performance in multiple-choice assessments may correlate more highly than the open-ended format with competence demonstrated in clinical practice following graduation. Studies of construct validity suggest that both formats measure essentially the same dimension, at least in mathematics, the physical sciences, biology and medicine. The persistence of the open-ended format in summative assessment may be due to the intuitive appeal of the belief that synthesising an answer to an open-ended question must be both more cognitively taxing and similar to actual experience than is selecting a correct response. I suggest that cognitive-constructivist learning theory would predict that a well-constructed context-rich multiple-choice item represents a complex problem-solving exercise which activates a sequence of cognitive processes which closely parallel those required in clinical practice, hence explaining the high validity of the multiple-choice format.

SUMMARY

The evidence does not support the proposition that the open-ended assessment format is superior to the multiple-choice format, at least in exit-level summative assessment, in terms of either its ability to test higher-order cognitive functioning or its validity. This is explicable using a theory of mental models, which might predict that the multiple-choice format will have higher validity, a statement for which some empiric support exists. Given the superior reliability and cost-effectiveness of the multiple-choice format consideration should be given to phasing out open-ended format questions in summative assessment. Whether the same applies to non-exit-level assessment and formative assessment is a question which remains to be answered; particularly in terms of the educational effect of testing, an area which deserves intensive study.

Assessment of competence in pediatric gastrointestinal endoscopy

Gastrointestinal endoscopy is an invaluable tool for the diagnosis and treatment of upper and lower gastrointestinal diseases in children. Pediatric and adult endoscopy differ in several respects including differences in procedural indications, sedation practices, pre-procedure preparation, equipment, and the importance of routine tissue sampling and terminal ileum intubation. In the same way that performance of endoscopy in children requires pediatric-specific training, assessment of pediatric endoscopists requires an approach that is tailored to pediatric practice and the use of assessment methods and measures that have been developed and validated specifically within the pediatric context.

Development of a behaviour change intervention: a case study on the practical application of theory

BACKGROUND

Use of theory in implementation of complex interventions is widely recommended. A complex trial intervention, to enhance self-management support for people with osteoarthritis (OA) in primary care, needed to be implemented in the Managing Osteoarthritis in Consultations (MOSAICS) trial. One component of the trial intervention was delivery by general practitioners (GPs) of an enhanced consultation for patients with OA. The aim of our case study is to describe the systematic selection and use of theory to develop a behaviour change intervention to implement GP delivery of the enhanced consultation.

METHODS

The development of the behaviour change intervention was guided by four theoretical models/frameworks: i) an implementation of change model to guide overall approach, ii) the Theoretical Domains Framework (TDF) to identify relevant determinants of change, iii) a model for the selection of behaviour change techniques to address identified determinants of behaviour change, and iv) the principles of adult learning. Methods and measures to evaluate impact of the behaviour change intervention were identified.

RESULTS

The behaviour change intervention presented the GPs with a well-defined proposal for change; addressed seven of the TDF domains (e.g., knowledge, skills, motivation and goals); incorporated ten behaviour change techniques (e.g., information provision, skills rehearsal, persuasive communication); and was delivered in workshops that valued the expertise and professional values of GPs. The workshops used a mixture of interactive and didactic sessions, were facilitated by opinion leaders, and utilised 'context-bound communication skills training.' Methods and measures selected to evaluate the behaviour change intervention included: appraisal of satisfaction with workshops, GP report of intention to practise and an assessment of video-recorded consultations of GPs with patients with OA.

CONCLUSIONS

A stepped approach to the development of a behaviour change intervention, with the utilisation of theoretical frameworks to identify determinants of change matched with behaviour change techniques, has enabled a systematic and theory-driven development of an intervention designed to enhance consultations by GPs for patients with OA. The success of the behaviour change intervention in practice will be evaluated in the context of the MOSAICS trial as a whole, and will inform understanding of practice level and patient outcomes in the trial.

Assessing communication quality of consultations in primary care: initial reliability of the Global Consultation Rating Scale, based on the Calgary-Cambridge Guide to the Medical Interview

OBJECTIVES

To investigate initial reliability of the Global Consultation Rating Scale (GCRS: an instrument to assess the effectiveness of communication across an entire doctor-patient consultation, based on the Calgary-Cambridge guide to the medical interview), in simulated patient consultations.

DESIGN

Multiple ratings of simulated general practitioner (GP)-patient consultations by trained GP evaluators.

SETTING

UK primary care.

PARTICIPANTS

21 GPs and six trained GP evaluators.

OUTCOME MEASURES

GCRS score.

METHODS

6 GP raters used GCRS to rate randomly assigned video recordings of GP consultations with simulated patients. Each of the 42 consultations was rated separately by four raters. We considered whether a fixed difference between scores had the same meaning at all levels of performance. We then examined the reliability of GCRS using mixed linear regression models. We augmented our regression model to also examine whether there were systematic biases between the scores given by different raters and to look for possible order effects.

RESULTS

Assessing the communication quality of individual consultations, GCRS achieved a reliability of 0.73 (95% CI 0.44 to 0.79) for two raters, 0.80 (0.54 to 0.85) for three and 0.85 (0.61 to 0.88) for four. We found an average difference of 1.65 (on a 0-10 scale) in the scores given by the least and most generous raters: adjusting for this evaluator bias increased reliability to 0.78 (0.53 to 0.83) for two raters; 0.85 (0.63 to 0.88) for three and 0.88 (0.69 to 0.91) for four. There were considerable order effects, with later consultations (after 15-20 ratings) receiving, on average, scores more than one point higher on a 0-10 scale.

CONCLUSIONS

GCRS shows good reliability with three raters assessing each consultation. We are currently developing the scale further by assessing a large sample of real-world consultations.

Validity and reliability of the Standardized Orthopedic Assessment Tool (SOAT): a variation of the traditional objective structured clinical examination

CONTEXT

Health care professions have replaced traditional multiple choice tests or essays with structured and practical, performance-based examinations with the hope of eliminating rater bias and measuring clinical competence.

OBJECTIVE

To establish the validity and reliability of the Standardized Orthopedic Assessment Tool (SOAT) as a measure of clinical competence of orthopaedic injury evaluation.

DESIGN

Descriptive laboratory study.

SETTING

University.

PATIENTS OR OTHER PARTICIPANTS

A total of 60 undergraduate students and 11 raters from 3 Canadian universities and 1 standardized patient.

INTERVENTION(S)

Students were required to complete a 30-minute musculoskeletal evaluation in 1 of 2 randomly assigned mock scenarios involving the knee (second-degree medial collateral ligament sprain) or the shoulder (third-degree supraspinatus muscle strain).

MAIN OUTCOME MEASURE(S)

We measured interreliability with an intraclass correlation coefficient (ICC) (2,k) and stability of the tool with standard error of measurement and confidence intervals. Agreement was measured using Bland-Altman plots. Concurrent validity was measured using a Pearson product moment correlation coefficient whereby the raters' global rating of a student was matched to the cumulative mean grade score.

RESULTS

The ICCs were 0.75 and 0.82 for the shoulder and knee cases, respectively. Bland-Altman plots indicated no systematic bias between raters. In addition, Pearson product moment correlation analysis demonstrated a strong relationship between the overall cumulative mean grade score and the global rating score of the examinees' performances.

CONCLUSIONS

This study demonstrated good interrater reliability of the SOAT with a standard error of measurement that indicated very modest stability, strong agreement between raters, and correlation indicative of concurrent validity.

How to attain expertise in clinical communication?

Several factors complicate the attainment of expertise in clinical communication. Medical curricula and postgraduate training insufficiently provide the required learning conditions of deliberate practice to overcome these obstacles. In this paper we provide recommendations for learning objectives and teaching methods for the attainment of professional expertise in patient education. Firstly, we propose to use functional learning objectives derived from the goals and strategies of clinical communication. Secondly, we recommend using teaching and assessment methods which: (1) contain stimulating learning tasks with opportunities for immediate feedback, reflection and corrections, and (2) give ample opportunity for repetition, gradual refinements and practice in challenging situations. Video-on-the-job fits these requirements and can be used to improve the competency in patient education of residents and medical staff in clinical practice. However, video-on-the-job can only be successful if the working environment supports the teaching and learning of communication and if medical staff which supervises the residents, is motivated to improve their own communication and didactic skills.

The national portfolio of learning for postgraduate family medicine training in South Africa: experiences of registrars and supervisors in clinical practice

BACKGROUND

In South Africa the submission of a portfolio of learning has become a national requirement for assessment of family medicine training. A national portfolio has been developed, validated and implemented. The aim of this study was to explore registrars' and supervisors' experience regarding the portfolio's educational impact, acceptability, and perceived usefulness for assessment of competence.

METHODS

Semi-structured interviews were conducted with 17 purposively selected registrars and supervisors from all eight South African training programmes.

RESULTS

The portfolio primarily had an educational impact through making explicit the expectations of registrars and supervisors in the workplace. This impact was tempered by a lack of engagement in the process by registrars and supervisors who also lacked essential skills in reflection, feedback and assessment. The acceptability of the portfolio was limited by service delivery demands, incongruence between the clinical context and educational requirements, design of the logbook and easy availability of the associated tools. The use of the portfolio for formative assessment was strongly supported and appreciated, but was not always happening and in some cases registrars had even organised peer assessment. Respondents were unclear as to how the portfolio would be used for summative assessment.

CONCLUSIONS

The learning portfolio had a significant educational impact in shaping work-place based supervision and training and providing formative assessment. Its acceptability and usefulness as a learning tool should increase over time as supervisors and registrars become more competent in its use. There is a need to clarify how it will be used in summative assessment.

Reflections on current methods for evaluating skills during joint replacement surgery: a scoping review

Valid and reliable techniques for assessing performance are essential to surgical education, especially with the emergence of competency-based frameworks. Despite this, there is a paucity of adequate tools for the evaluation of skills required during joint replacement surgery. In this scoping review, we examine current methods for assessing surgeons' competency in joint replacement procedures in both simulated and clinical environments. The ability of many of the tools currently in use to make valid, reliable and comprehensive assessments of performance is unclear. Furthermore, many simulation-based assessments have been criticised for a lack of transferability to the clinical setting. It is imperative that more effective methods of assessment are developed and implemented in order to improve our ability to evaluate the performance of skills relating to total joint replacement. This will enable educators to provide formative feedback to learners throughout the training process to ensure that they have attained core competencies upon completion of their training. This should help ensure positive patient outcomes as the surgical trainees enter independent practice.

A randomized controlled pilot trial comparing the impact of access to clinical endocrinology video demonstrations with access to usual revision resources on medical student performance of clinical endocrinology skills

BACKGROUND

Demonstrating competence in clinical skills is key to course completion for medical students. Methods of providing clinical instruction that foster immediate learning and potentially serve as longer-term repositories for on-demand revision, such as online videos demonstrating competent performance of clinical skills, are increasingly being used. However, their impact on learning has been little studied. The aim of this study was to determine the value of adjunctive on-demand video-based training for clinical skills acquisition by medical students in endocrinology.

METHODS

Following an endocrinology clinical tutorial program, 2nd year medical students in the pre-assessment revision period were recruited and randomized to either a set of bespoke on-line clinical skills training videos (TV), or to revision as usual (RAU). The skills demonstrated on video were history taking in diabetes mellitus (DMH), examination for diabetes lower limb complications (LLE), and examination for signs of thyroid disease (TE). Students were assessed on these clinical skills in an observed structured clinical examination two weeks after randomization. Assessors were blinded to student randomization status.

RESULTS

For both diabetes related clinical skills assessment tasks, students in the TV group performed significantly better than those in the RAU group. There were no between group differences in thyroid examination performance. For the LLE, 91.7% (n = 11/12) of students randomized to the video were rated globally as competent at the skill compared with 40% (n = 4/10) of students not randomized to the video (p = 0.024). For the DMH, 83.3% (n = 10/12) of students randomized to the video were rated globally as competent at the skill compared with 20% (n = 2/10) of students not randomized to the video (p = 0.007).

CONCLUSION

Exposure to high quality videos demonstrating clinical skills can significantly improve medical student skill performance in an observed structured clinical examination of these skills, when used as an adjunct to clinical skills face-to-face tutorials and deliberate practice of skills in a blended learning format. Video demonstrations can provide an enduring, on-demand, portable resource for revision, which can even be used at the bedside by learners. Such resources are cost-effectively scalable for large numbers of learners.

"Best practice" skills lab training vs. a "see one, do one" approach in undergraduate medical education: an RCT on students' long-term ability to perform procedural clinical skills

Background

Benefits of skills lab training are widely accepted, but there is sparse research on its long-term effectiveness. We therefore conducted a prospective, randomised controlled-trial to investigate whether in a simulated setting students trained according to a "best practice" model (BPSL) perform two skills of different complexity (nasogastral tube insertion, NGT; intravenous cannulation, IVC) better than students trained with a traditional "see one, do one" teaching approach (TRAD), at follow-up of 3 or 6 months.

Methodology and Principal Findings

94 first-year medical students were randomly assigned to one of four groups: BPSL training or TRAD teaching with follow-up at 3 (3M) or 6 (6M) months. BPSL included structured feedback, practice on manikins, and Peyton’s "Four-Step-Approach", while TRAD was only based on the "see one - do one" principle. At follow-up, manikins were used to assess students’ performance by two independent blinded video-assessors using binary checklists and a single-item global assessment scale. BPSL students scored significantly higher immediately after training (NGT: BPSL3M 94.8%±0.2 and BPSL6M 95.4%±0.3 percentage of maximal score ± SEM; TRAD3M 86.1%±0.5 and TRAD6M 84.7%±0.4. IVC: BPSL3M 86.4%±0.5 and BPSL6M 88.0%±0.5; TRAD3M 73.2%±0.7 and TRAD6M 72.5%±0.7) and lost significantly less of their performance ability at each follow-up (NGT: BPSL3M 86.3%±0.3 and TRAD3M 70.3%±0.6; BPSL6M 89.0%±0.3 and TRAD6M 65.4%±0.6; IVC: BPSL3M 79.5%±0.5 and TRAD3M 56.5%±0.5; BPSL6M 73.2%±0.4 and TRAD6M 51.5%±0.8). In addition, BPSL students were more often rated clinically competent at all assessment times. The superiority at assessment after training was higher for the more complex skill (IVC), whereas NGT with its lower complexity profited more with regard to long-term retention.

Conclusions

This study shows that within a simulated setting BPSL is significantly more effective than TRAD for skills of different complexity assessed immediately after training and at follow-up. The advantages of BPSL training are seen especially in long-term retention.

"How Do You Feel about Antibiotics for This?" A Qualitative Study of Physician Attitudes towards a Context-Rich Communication Skills Method

To explore experiences with and views of general practitioners (GPs) on a physician communication training method in primary care and its applicability and implementation in daily practice, we performed a semi-structured qualitative study of GPs’ experience of training in and implementing a communication skills training program for managing lower respiratory tract infection (LRTI) which included a seminar, simulated patient consultation together with providing and receiving feedback on ones own transcript, and a seminar in a structured approach to the LRTI consultation. Seventeen out of 20 eligible GPs who had participated in the IMPAC³T trial and were allocated to receiving enhanced physician communication training for managing lower respiratory tract infection participated. GPs’ experiences with the physician communication training method and its specific components were positive. The method gave GPs additional tools for managing LRTI consultations and increased their sense of providing evidence-based management. During the study, GPs reported using almost all communication items covered in the training, but some GPs stated that the communication skills diluted over time, and that they continued to use a selected set of the skills. The general communication items were most regularly used. Implementation of the method in daily practice helped GPs to prescribe fewer antibiotics in LRTI with the only perceived disadvantage being time-pressure. This study suggests that GPs felt positive about the physician communication training method for enhanced management of LRTI in primary care. GPs continued to use some of the communication items, of which general communication items were the most common. Furthermore, GPs believed that implementation of the communication skills in daily practice helped them to prescribe fewer antibiotics. The context-rich communication method could have wider application in common conditions in primary care.

CORD-AEUS: consensus document for the emergency ultrasound milestone project

In 2012, the Accreditation Council for Graduate Medical Education (ACGME) designated ultrasound (US) as one of 23 milestone competencies for emergency medicine (EM) residency graduates. With increasing scrutiny of medical educational programs and their effect on patient safety and health care delivery, it is imperative to ensure that US training and competency assessment is standardized. In 2011, a multiorganizational committee composed of representatives from the Council of Emergency Medicine Residency Directors (CORD), the Academy of Emergency Ultrasound of the Society for Academic Emergency Medicine (SAEM), the Ultrasound Section of the American College of Emergency Physicians (ACEM), and the Emergency Medicine Residents' Association was formed to suggest standards for resident emergency ultrasound (EUS) competency assessment and to write a document that addresses the ACGME milestones. This article contains a historical perspective on resident training in EUS and a table of core skills deemed to be a minimum standard for the graduating EM resident. A survey summary of focused EUS education in EM residencies is described, as well as a suggestion for structuring education in residency. Finally, adjuncts to a quantitative measurement of resident competency for EUS are offered.

Effectiveness of oncogenetics training on general practitioners' consultation skills: a randomized controlled trial

PURPOSE

General practitioners are increasingly called upon to deliver genetic services and could play a key role in translating potentially life-saving advancements in oncogenetic technologies to patient care. If general practitioners are to make an effective contribution in this area, their genetics competencies need to be upgraded. The aim of this study was to investigate whether oncogenetics training for general practitioners improves their genetic consultation skills.

METHODS

In this pragmatic, blinded, randomized controlled trial, the intervention consisted of a 4-h training (December 2011 and April 2012), covering oncogenetic consultation skills (family history, familial risk assessment, and efficient referral), attitude (medical ethical issues), and clinical knowledge required in primary-care consultations. Outcomes were measured using observation checklists by unannounced standardized patients and self-reported questionnaires.

RESULTS

Of 88 randomized general practitioners who initially agreed to participate, 56 completed all measurements. Key consultation skills significantly and substantially improved; regression coefficients after intervention were equivalent to 0.34 and 0.28 at 3-month follow-up, indicating a moderate effect size. Satisfaction and perceived applicability of newly learned skills were highly scored.

CONCLUSION

The general practitioner-specific training proved to be a feasible, satisfactory, and clinically applicable method to improve oncogenetics consultation skills and could be used as an educational framework to inform future training activities with the ultimate aim of improving medical care.

Details acquired from medical history and patients' experience of empathy--two sides of the same coin

BACKGROUND

History taking and empathetic communication are two important aspects in successful physician-patient interaction. Gathering important information from the patient's medical history is needed for effective clinical decision making while empathy is relevant for patient satisfaction. We wanted to investigate whether medical students near graduation are able to combine both skills as required in daily medical practice.

METHODS

Thirty near graduates from Hamburg Medical School participated in an assessment for clinical competences including a consultation hour with five standardized patients. Each patient interview was videotaped and standardized patients rated participants with the CARE questionnaire for consultation and relational empathy. All videotaped interviews were rated with a checklist based on the number of important medical aspects for each case. Data were analysed with the linear mixed model to correct for random effects. Regression analysis was performed to look for correlations between the number of questions asked by a participant and their respective empathy rating.

RESULTS

Of the 123 aspects that could have been gathered in total, students only requested 56.4% (95% CI 53.5-59.3). While no difference between male and female participants was found, a significant difference (p<.001) was observed between the two parts of the checklist with 61.1% (95% CI 57.9-64.3) of aspects asked for in part 1 (patient's symptoms) versus 52.0 (95 47.4-56.7) in part 2 (further history). All female standardized patients combined rated female participants (mean score 14.2, 95% CI 12.3-16.3) to be significantly (p<.01) more empathetic than male participants (mean score 19.2, 95% CI 16.3-22.6). Regression analysis revealed no correlation between the number of medical aspects gathered by a participant and his or her respective empathy score given by the standardized patient in the CARE questionnaire.

CONCLUSION

Gathering sufficient medical data from a patient's history and empathetic communication are two completely separate sides of the coin of history taking. While both skills have to be acquired during medical school training with particular focus on their respective learning objectives, medical students need to be provided with additional learning and feedback opportunities where they can be observed exercising both skills combined as required in physicians' daily practice.

The current state of sleep medicine education in US neurology residency training programs: where do we go from here?

OBJECTIVE

To evaluate the current state of sleep medicine educational resources and training offered by US neurology residency programs.

METHODS

In 2010, a 20-item peer reviewed Sleep Education Survey (SES) was sent to neurology residency program directors surveying them about sleep medicine educational resources used in teaching residents. Pearson product momentum correlation was used to determine correlation of program attributes with resident interest in pursuing a career in sleep medicine.

RESULTS

Of the programs completing the survey, 81% listed a formal sleep rotation and 24% included a forum for sleep research. A variety of innovative approaches for teaching sleep medicine were noted. Program directors noted that 5.7% residents entered sleep medicine fellowship training programs in the preceding 5 years. Programs that had a more substantial investment in sleep medicine teaching resources were more likely to report residents entering a sleep medicine training program.

CONCLUSION

This is the first report providing an analysis of the current state of sleep medicine training in US Neurology Residency Programs. Our data provide evidence that investment by the residency program in sleep education may enhance the ultimate decision by the neurology trainee to pursue a career in sleep medicine.

Education in patient-physician communication: how to improve effectiveness?

OBJECTIVE

Despite educational efforts expertise in communication as required by the CanMEDS competency framework is not achieved by medical students and residents. Several factors complicate the learning of professional communication.

METHODS

We adapted the reflective-impulsive model of social behaviour to explain the complexities of learning professional communication behaviour. We formulated recommendations for the learning objectives and teaching methods of communication education. Our recommendations are based on the reflective-impulsive model and on the model of deliberate practice which complements the reflective-impulsive model. Our recommendations are substantiated by those we found in the literature.

RESULTS

The reflective-impulsive model explains why the results of communication education fall below expectations and how expertise in communication can be attained by deliberate practice. The model of deliberate practice specifies learning conditions which are insufficiently fulfilled in current communication programmes.

CONCLUSION

The implementation of our recommendations would require a great deal of effort. Therefore we doubt whether expertise in professional communication can be fully attained during medical training.

PRACTICE IMPLICATIONS

We propose that the CanMEDS communication competencies not be regarded as endpoints in medical education but as guidelines to improve communication competency through deliberate practice throughout a professional career.

Doctors' and nurses' views on patient care for type 2 diabetes: an interview study in primary health care in Oman

Aim

This study aimed at exploring the experiences of primary health-care providers of their encounters with patients with type 2 diabetes, and their preferences and suggestions for future improvement of diabetes care.

Background

Barriers to good diabetes care could be related to problems from health-care providers’ side, patients’ side or the health-care system of the country. Treatment of patients with type 2 diabetes has become a huge challenge in Oman, where the prevalence has increased to high levels.

Method

Semi-structured interviews were conducted with 26 health-care professionals, 19 doctors and seven nurses, who worked in primary health care in Oman. Qualitative content analysis was applied.

Findings

Organizational barriers and barriers related to patients and health-care providers were identified. These included workload and lack of teamwork approach. Poor patients’ management adherence and influence of culture on their attitudes towards illness were identified. From the providers’ side, language barriers, providers’ frustration and aggressive attitudes towards the patients were reflected. Decreasing the workload, availability of competent teams with diabetes specialist nurses and continuity of care were suggested. Furthermore, changing professional behaviours towards a more patient-centred approach and need for health education to the patients, especially on self-management, were addressed. Appropriate training for health-care providers in communication skills with emphasis on self-care education and individualization of care according to each patient's needs are important for improvement of diabetes care in Oman.

Outcome-based medical education: implications, opportunities, and challenges

Outcome-based education (OBE) is a major reframing of how medical educators think about teaching, learning, and assessment. There are many alternative versions of OBE and the implications of this framework are not always well-understood. A review of the literature on OBE and an analysis of the educational implications suggest seven areas of contrast with traditional educational frameworks. Key contrasts center around how educational outcomes are defined, the emphasis of learning over teaching, the centrality of rigorous assessment, the need for flexibility and individualization in the curriculum, and shifting roles and responsibilities of teachers and learners. OBE has the potential for dramatic and even revolutionary changes in medical education. However, it carries with it significant challenges that include the expenses of additional assessment, uncertainty among students and faculty about their responsibilities, and complexities in planning and organizing the educational process. Instead, of whole-hearted adoption of the OBE model, most medical schools and residency programsare exploring OBE in small-scale "experiments" that will inform the field about the best ways to incorporate the critical features of OBE into medical education. Such experiments are invaluable for helping us better understand the promise and possibilities of OBE.

Exploratory study to identify the process used by pharmacy staff to verify the accuracy of dispensed medicines

Objectives

To determine the common stages and strategies involved in the dispensing accuracy-checking process used by pharmacy staff and to determine the training activities used by these staff to gain the knowledge and skills for accuracy checking.

Method

Face-to-face tape-recorded ethnographic interviews (n = 28) were undertaken in 2009–2010 at two large teaching hospitals with a purposive sample of pharmacists and accredited checking technicians qualified to undertake the final accuracy check on dispensed medicines. Participants described their accuracy-checking process, strategies used to aid checking using anonymised prescriptions and accurate dispensing of medicines to aid discussion. The range of training activities undertaken to develop this skill were discussed. Qualitative data were analysed in accordance with the principles of grounded theory to identify themes.

Key findings

The accuracy-checking process was described as a cognitive and systematic process. The order in which accuracy checking was executed was found to follow two pathways, with all participants checking the prescription first before verifying either the label or dispensed product. Various physical and sensory aids were used to assist in this verification process. There were inconsistencies in the level of accuracy-checking training received by pharmacists and accredited checking technicians, with many pharmacists reporting no training.

Conclusion

Although an important medication-error prevention strategy, until this study little was known about the process used by pharmacy staff when verifying the accuracy of dispensed medicines. Accuracy checking is a complex cognitive task involving verification of the product and label with the prescription. Strategies obtained during past experience and in training were used to aid checking. The study highlighted that pharmacy staff training to undertake this task was variable. Application of strategies identified in this study may allow individuals to adopt further safeguards to improve patient safety.

The promise of competency-based education in the health professions for improving global health

Competency-based education (CBE) provides a useful alternative to time-based models for preparing health professionals and constructing educational programs. We describe the concept of 'competence' and 'competencies' as well as the critical curricular implications that derive from a focus on 'competence' rather than 'time'. These implications include: defining educational outcomes, developing individualized learning pathways, setting standards, and the centrality of valid assessment so as to reflect stakeholder priorities. We also highlight four challenges to implementing CBE: identifying the health needs of the community, defining competencies, developing self-regulated and flexible learning options, and assessing learners for competence. While CBE has been a prominent focus of educational reform in resource-rich countries, we believe it has even more potential to align educational programs with health system priorities in more resource-limited settings. Because CBE begins with a careful consideration of the competencies desired in the health professional workforce to address health care priorities, it provides a vehicle for integrating the health needs of the country with the values of the profession.

A Delphi study to construct a CanMEDS competence based inventory applicable for workplace assessment

BACKGROUND

During workplace based learning students develop professional competences and an appropriate performance. To gain insight in the learning process and to evaluate competences and performance, assessment tools are essential and need to be of good quality. We aimed to construct a competence inventory applicable as an instrument to measure the content validity of workplace based assessment tools, such as portfolio.

METHODS

A Delphi study was carried out based on the CanMEDS Roles Framework. In three rounds, experts (N = 25-30) were invited to score the key competences per CanMEDS role on relevance (6-point Likert-scale), and to comment on the content and formulation bearing in mind its use in workplace based assessment. A descriptive analysis of relevances and comments was performed.

RESULTS

Although all competences were scored as relevant, many comments pointed at a lack of concrete, transparent and applicable descriptions of the key competences for the purpose of assessment. Therefore, the CanMEDS roles were reformulated in this Delphi procedure as concrete learning outcomes, observable and suitable for workplace based assessment.

CONCLUSIONS

A competence based inventory, ready for validating workplace based assessment tools, was constructed using a Delphi procedure and based on a clarification and concretisation of the CanMEDS roles.

Inter-examiner agreement in clinical evaluation

BACKGROUND

The reliability of assessment is an important issue in the evaluation of competence in medical and allied health practice, particularly when assessments are conducted by multiple examiners. The purpose of this study was to examine the agreement between multiple examiners in the assessment of a postgraduate physiotherapy student using a specifically designed performance evaluation system.

METHODS

Seven examiners simultaneously watched a recording of a postgraduate student's examination and treatment of one patient. The Postgraduate Physiotherapy Performance Assessment (PPPA) form was used to guide the assessment of performance in key areas of patient examination and management. Each examiner independently recorded a grade for each of five performance categories, and these scores were used to guide the global performance grade and mark.

RESULTS

Five examiners agreed on the global performance grade and four of the performance categories. The level of pass grade awarded was more variable, with scores in the performance categories spanning two grades, and in one case, three grades. The two examiners who were not in agreement with the majority consistently awarded higher grades across most performance categories.

DISCUSSION

This preliminary study has demonstrated majority agreement in global performance between multiple examiners when physiotherapy clinical practice is assessed against specific performance standards. Not all examiners awarded global grades consistent with the majority, and there was greater variability between examiners when grading performance in specific aspects of practice. These findings highlight the importance of examiner training and review sessions to improve inter-examiner agreement in assessments of clinical performance that require multiple examiners.

Practical skills teaching in contemporary surgical education: how can educational theory be applied to promote effective learning?

BACKGROUND

Teaching practical skills is a core component of undergraduate and postgraduate surgical education. It is crucial to optimize our current learning and teaching models, particularly in a climate of decreased clinical exposure. This review explores the role of educational theory in promoting effective learning in practical skills teaching.

METHODS

Peer-reviewed publications, books, and online resources from national bodies (eg, the UK General Medical Council) were reviewed.

RESULTS

This review highlights several aspects of surgical education, modeling them on current educational theory. These include the following: (1) acquisition and retention of motor skills (Miller's triangle; Fitts' and Posner's theory), (2) development of expertise after repeated practice and regular reinforcement (Ericsson's theory), (3) importance of the availability of expert assistance (Vygotsky's theory), (4) learning within communities of practice (Lave and Wenger's theory), (5) importance of feedback in learning practical skills (Boud, Schon, and Endes' theories), and (6) affective component of learning.

CONCLUSIONS

It is hoped that new approaches to practical skills teaching are designed in light of our understanding of educational theory.

The Assessment of Physiotherapy Practice (APP) is a valid measure of professional competence of physiotherapy students: a cross-sectional study with Rasch analysis

QUESTION

Is the Assessment of Physiotherapy Practice (APP) a valid instrument for the assessment of entry-level competence in physiotherapy students?

DESIGN

Cross-sectional study with Rasch analysis of initial (n=326) and validation samples (n=318). Students were assessed on completion of 4, 5, or 6-week clinical placements across one university semester.

PARTICIPANTS

298 clinical educators and 456 physiotherapy students at nine universities in Australia and New Zealand provided 644 completed APP instruments.

RESULTS

APP data in both samples showed overall fit to a Rasch model of expected item functioning for interval scale measurement. Item 6 (Written communication) exhibited misfit in both samples, but was retained as an important element of competence. The hierarchy of item difficulty was the same in both samples with items related to professional behaviour and communication the easiest to achieve and items related to clinical reasoning the most difficult. Item difficulty was well targeted to person ability. No Differential Item Functioning was identified, indicating that the scale performed in a comparable way regardless of the student's age, gender or amount of prior clinical experience, and the educator's age, gender, or experience as an educator, or the type of facility, university, or clinical area. The instrument demonstrated unidimensionality confirming the appropriateness of summing the scale scores on each item to provide an overall score of clinical competence and was able to discriminate four levels of professional competence (Person Separation Index=0.96). Person ability and raw APP scores had a linear relationship (r(2)=0.99).

CONCLUSION

Rasch analysis supports the interpretation that a student's APP score is an indication of their underlying level of professional competence in workplace practice.

Conceptual framework for performance assessment: competency, competence and performance in the context of assessments in healthcare--deciphering the terminology

BACKGROUND

The definitions of performance, competence and competency are not very clear in the literature. The assessment of performance and the selection of tools for this purpose depend upon a deep understanding of each of the above terms and the factors influencing performance.

AIM

In this article, we distinguish between competence and competency and explain the relationship of competence and performance in the light of the Dreyfus model of skills acquisition. We briefly critique the application of the principles described by Miller to the modern assessment tools and distinguish between assessment of actual performance in workplace settings and the observed performance, demonstrated by the candidates in the workplace or simulated settings.

RESULTS

We describe a modification of the Dreyfus model applicable to assessments in healthcare and propose a new model for the assessment of performance and performance rating scale (PRS) based on this model.

CONCLUSION

We propose that the use of adapted versions of this PRS will result in benchmarking of performance and allowing the candidates to track their progression of skills in various areas of clinical practice.

Workplace-based assessment instruments in the health sciences

A historical overview of the development of assessment instruments in the health sciences is presented here, with specific attention paid to workplace-based assessment instruments. Three instruments are reviewed in detail: the mini clinical evaluation exercise (mCEX), direct observation of procedural skills (DOPS), and multi-source feedback (MSF). Features common to these instruments include their authenticity, their use in assessing professional skills, and the opportunities they afford for the provision of feedback. Although almost exclusively used in graduate medical training, they are likely to play an increasingly important role in the assessment of veterinary undergraduate students in preparation for professional practice. However, the time and cost associated with implementing these instruments raises questions about their feasibility. The continued search for the holy grail of assessment instruments and the challenges relating to the need for trained assessors leads us to conclude that ultimately, the competence of health professionals should continue to be measured using several complementary instruments.

Workplace-based assessments of junior doctors: do scores predict training difficulties?

OBJECTIVES

Workplace-based assessment (WPBA) is an increasingly important part of postgraduate medical training and its results may be used as evidence of professional competence. This study evaluates the ability of WPBA to distinguish UK Foundation Programme (FP) doctors with training difficulties and its effectiveness as a surrogate marker for deficiencies in professional competence.

METHODS

We conducted a retrospective observational study using anonymised records for 1646 trainees in a single UK postgraduate deanery. Data for WPBAs conducted from August 2005 to April 2009 were extracted from the e-portfolio database. These data included all scores submitted by trainees in FP years 1 and 2 on mini-clinical evaluation exercise (mini-CEX), case-based discussion (CbD), direct observation of procedural skills (DOPS) and mini-peer assessment tool (mini-PAT) assessments. Records of trainees in difficulty, as identified by their educational supervisors, were tagged as index cases. Main outcome measures were odds ratios (ORs) for associations between mean WPBA scores and training difficulties. Further analyses by the reported aetiology of the training difficulty (health-, conduct- or performance-related) were performed.

RESULTS

Of the 1646 trainees, 92 had been identified as being in difficulty. Mean CbD and mini-CEX scores were lower for trainees in difficulty and an association was found between identified training difficulties and average scores on the mini-CEX (OR = 0.54; p = 0.034) and CbD (OR = 0.39; p = 0.002). A receiver operator characteristic curve analysis of mean WPBA scores for diagnosing 'in difficulty' status yielded an area under the curve of 0.64, indicating weak predictive value. There was no statistical evidence that mean scores on DOPS and mini-PAT assessments differed between the two groups.

CONCLUSIONS

Analysis of a large dataset of WPBA scores revealed significant associations between training difficulties and lower mean scores on both the mini-CEX and CbD. Models show that using WPBA scores is, however, not a valid way of screening for trainees in difficulty. Workplace-based assessments have value as formative assessments that prompt supervision, feedback and reflection. They should not be relied upon to certify competence and their use for such ends may reduce their effectiveness in training. Their results should be interpreted in the context of multiple other methods of assessment, with the aim of achieving a genuinely holistic and representative assessment of professional competence.

Objective Structured Clinical Examinations (OSCEs) compared with traditional assessment methods

OBJECTIVES

To compare objective structured clinical examinations (OSCEs) and traditional assessment methods among recent pharmacy graduates.

METHODS

Individual student performance in OSCEs was compared with performance on traditional pharmacy-practice examinations at the same level of program study.

RESULTS

A moderate correlation was found between individual attainment in OSCE examinations and on traditional pharmacy practice examinations at the same level.

CONCLUSIONS

OSCEs add value to traditional methods of assessment because the 2 evaluation methods measure different competencies.

Factors influencing pharmacist performance: a review of the peer-reviewed literature

OBJECTIVES

To undertake a review of peer-reviewed literature to explore factors affecting pharmacists' performance.

METHODS

The following databases were searched: Medline, Embase, Scopus, ISI Web of Knowledge and PsychInfo. Inclusion criteria were: English language only, published between 1990 and 2010 and published in the United Kingdom (UK), United States of America (USA), Canada, Australia, New Zealand or Europe.

RESULTS

The search strategy identified 37 items. The review found that there was some evidence to suggest that pharmacists with certain characteristics (e.g. being male, being of ethnic minority origin, working in community pharmacy and having trained overseas) were more likely to experience performance problems. Factors relating to workload and work environment were associated with performance problems, particularly in relation to errors. There was some limited evidence to suggest that experiencing problems with alcohol or drugs could negatively impact on pharmacists' performance.

CONCLUSION

The findings suggest that pharmacist performance may be affected by multiple factors, including personal characteristics such as age, gender, ethnicity, place of primary qualification, factors associated with the workplace and mental and physical health problems. The evidence is not unequivocal and gaps in the literature exist, suggesting that pharmacist performance is an under-researched area.

The role of assessment in the student learning process

Assessment is a powerful learning tool that can enhance learning and education. The process of student assessment should align with curricular goals and educational objectives. Identifying the assessment strategies necessary for the proper evaluation of students' progress within individual programs is as important as establishing curricular content and delivery methods. The purpose of this paper is to discuss elements to be considered in assessment design and implementation as well as common challenges encountered during this process. Elements to be considered during assessment design include purpose of assessment, domains to be tested, and characteristics of the assessment tools to be employed. Assessment tools are evaluated according to four main characteristics: relevance, feasibility, validity, and reliability. Based on the evidence presented in the literature, the use of a variety of assessment tools is recommended to match diverse domains and learning styles. The assessment cycle concludes with the evaluation of the results and, based on these, the institution, program, or course can make changes to improve the quality of education. If assessment design aligns with educational outcomes and instructional methods, it improves the quality of education and supports student learning.

Understanding vs. competency: the case of accuracy checking dispensed medicines in pharmacy

Ensuring the competence of healthcare professionals' is core to undergraduate and post-graduate education. Undergraduate pharmacy students and pre-registration graduates are required to demonstrate competence at dispensing and accuracy checking medicines. However, competence differs from understanding. This study determined the competence and understanding of undergraduate students and pharmacists at accuracy checking dispensed medicines. Third year undergraduate pharmacy students and first year post-graduate diploma pharmacists participated in the study, which involved an accuracy checking task and concept mapping exercise. Participants accuracy checked eight medicines which contained 13 dispensing errors and then constructed a concept map illustrating their understanding of the accuracy checking process. The error detection rates and types of dispensing errors detected by undergraduates and pharmacists were compared using Mann-Whitney and chi-square, respectively. Statistical significance was p ≤ 0.05. Concept maps were qualitatively analysed to identify structural typologies. Forty-one undergraduates and 78 pharmacists participated in the study. Pharmacists detected significantly more dispensing errors (85%) compared to the undergraduates (77%, p ≤ 0.001). Only one undergraduate and seven pharmacists detected all dispensing errors. The majority of concept maps were chains (undergraduates = 46%, n = 19; pharmacists = 45%, n = 35) and spokes (undergraduates = 54%, n = 22; pharmacists = 54%, n = 42) indicating surface learning. One pharmacist, who detected all dispensing errors in the accuracy checking exercise, created a networked map characteristic of deep learning. Undergraduate students and pharmacists demonstrated a degree of operational competence at detecting dispensing errors without fully understanding the accuracy checking process. Accuracy checking training should be improved at undergraduate and post-graduate level so that pharmacists are equipped with the knowledge and understanding to accurately check medicines and detect dispensing errors, thereby safeguarding patient safety.

Impact of workplace based assessment on doctors' education and performance: a systematic review

OBJECTIVE

To investigate the literature for evidence that workplace based assessment affects doctors' education and performance.

DESIGN

Systematic review.

DATA SOURCES

The primary data sources were the databases Journals@Ovid, Medline, Embase, CINAHL, PsycINFO, and ERIC. Evidence based reviews (Bandolier, Cochrane Library, DARE, HTA Database, and NHS EED) were accessed and searched via the Health Information Resources website. Reference lists of relevant studies and bibliographies of review articles were also searched. Review methods Studies of any design that attempted to evaluate either the educational impact of workplace based assessment, or the effect of workplace based assessment on doctors' performance, were included. Studies were excluded if the sampled population was non-medical or the study was performed with medical students. Review articles, commentaries, and letters were also excluded. The final exclusion criterion was the use of simulated patients or models rather than real life clinical encounters.

RESULTS

Sixteen studies were included. Fifteen of these were non-comparative descriptive or observational studies; the other was a randomised controlled trial. Study quality was mixed. Eight studies examined multisource feedback with mixed results; most doctors felt that multisource feedback had educational value, although the evidence for practice change was conflicting. Some junior doctors and surgeons displayed little willingness to change in response to multisource feedback, whereas family physicians might be more prepared to initiate change. Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses. Four studies examined the mini-clinical evaluation exercise, one looked at direct observation of procedural skills, and three were concerned with multiple assessment methods: all these studies reported positive results for the educational impact of workplace based assessment tools. However, there was no objective evidence of improved performance with these tools.

CONCLUSIONS

Considering the emphasis placed on workplace based assessment as a method of formative performance assessment, there are few published articles exploring its impact on doctors' education and performance. This review shows that multisource feedback can lead to performance improvement, although individual factors, the context of the feedback, and the presence of facilitation have a profound effect on the response. There is no evidence that alternative workplace based assessment tools (mini-clinical evaluation exercise, direct observation of procedural skills, and case based discussion) lead to improvement in performance, although subjective reports on their educational impact are positive.

Psychometric properties of a peer-assessment program to assess continuing competence in physical therapy

BACKGROUND

The College of Physiotherapists of Ontario implemented an Onsite Assessment to evaluate the continuing competence of physical therapists.

OBJECTIVE

This study was undertaken to examine the reliability of the various tools used in the Onsite Assessment and to consider the relationship between the final decision and demographic factors.

DESIGN

This was a psychometric study.

METHODS

Trained peer assessors (n=63) visited randomly selected physical therapists (n=106) in their workplace. Fifty-three physical therapists were examined by 2 assessors simultaneously. The assessment included a review of practice issues, record keeping, billing practices, the physical therapist's professional portfolio, and a chart-stimulated recall process. The Quality Management Committee made the final decision regarding the physical therapist's performance using the assessor's summary report. Generalizability theory was used to examine the interrater reliability of the tools. Correlation coefficients and regression analyses were used to examine the relationships between demographic factors and performance.

RESULTS

The majority of the physical therapists (88%) completed the program successfully, 11% required remediation, and 1% required further assessment. The interrater reliability of the components was above .70 for 2 raters' evaluations, with the exception of billing practices. There was no relationship between the final decision and age or years since graduation (r<.05). Limitations Limitations include a small sample and a lack of data on system-related factors that might influence performance.

CONCLUSIONS

The vast majority of the physical therapists met the College of Physiotherapists of Ontario's professional standards. Reliability analysis indicated that the number of charts reviewed could be reduced. Strategies to improve the reliability of the various components must take into account feasibility issues related to financial and human resources. Further research to examine factors associated with failure to adhere to professional standards should be considered. These results can provide valuable information to regulatory agencies or managers considering similar continuing competence assessment programs.

Workplace-based assessment in radiology-where to now?

Assessment of doctors is in a state of flux. Traditional methods of assessment have been critically examined and found inherently limited. The wholesale shift towards outcome-orientated education in the last 10 years has led to the relatively rapid development of a radically different method of assessment. This method focuses on assessing what doctors do in everyday practice rather than written or practical simulations. Known collectively as "workplace-based assessment" tools, these have been embraced in North America, whereas they have been more cautiously adopted in the UK. However, many of these assessment tools have not been rigorously studied and, moreover, few have been specifically developed for assessing radiologists. However, they are likely to be incorporated into radiology training in the near future. This paper critically analyses both the underpinning assumptions behind this method and the evidence behind existing tools, and looks at the work that is required to develop, adopt or adapt such tools for use in radiology.