Commentary: Assessment is an educational tool

Remediation and reassessment methods in pharmacy education: A systematic review

BACKGROUND

Colleges of pharmacy are currently required to implement a remediation program within their curricula, but no specifications are provided on the ideal methodology. While the need for successful remediation strategies continues to grow, literature describing positive or negative outcomes of different approaches is significantly lacking. The objective of this literature review was to describe and evaluate remediation methodologies in pharmacy education.

METHODS

This literature review was completed following PRISMA criteria. A search of the PubMed, Cochrane Library, Cumulative Index of Nursing and Allied Health, Academic Search Complete, PsycInfo, Scopus, and ProQuest Central databases was conducted in July 2019. Studies were included if they involved pharmacy student education and described either remediation or reassessment.

RESULTS

The evaluated studies discussed a range of course types being remediated, a large variety of remediation strategies and timeframes, and differing overall outcomes. No studies provided comparison of remediation techniques or provided details on the implementation of their chosen approaches. A consistent finding within the evaluated studies was the inclusion of prevention strategies to attempt to avoid the need for remediation preemptively. Overall outcomes for each remedial program were inconsistent and no clear patterns were evident other than an improvement in student performance following remediation.

IMPLICATIONS

Remediation strategies included course repetition, summer restudy, reassessment, and individualized plans. Outcomes varied significantly between studies, making methodology comparisons difficult. Future studies that include more detail and consistency in the reported outcomes would be beneficial to students and help clarify remediation for colleges of pharmacy.

Validation of time to task performance assessment method in simulation: A comparative design study

BACKGROUND

There is a lack of objective and valid measures for assessing nursing clinical competence which could adversely impact patient safety. Therefore, we evaluated an objective assessment of clinical competence, Time to Task (ability to perform specific, critical nursing care activities within 5 min), and compared it to two subjective measures, (Lasater Clinical Judgement Rubric [LCJR] and common "pass/fail" assessment).

DESIGN/METHODS

Using a prospective, "Known Groups" (Expert vs. Novice nurses) comparative design, Expert nurses (ICU nurses with >5 years of ICU experience) and Novice nurses (senior prelicensure nursing students) participated individually in a simulation of a patient in decompensated heart failure. Fourteen nursing instructors or preceptors, blinded to group assignment, reviewed 28 simulation videos (15 Expert and 13 Novice) and scored them using the LCJR and pass/fail assessments. Time to Task assessment was scored based on time thresholds for specific nursing actions prospectively set by an expert clinical panel. Statistical analysis consisted of Medians Test and sensitivity and specificity analyses.

RESULTS

The LCJR total score was significantly different between Experts and Novices (p < 0.01) and revealed adequate sensitivity (ability to correctly identify "Expert" nurses; 0.72) but had a low specificity (ability to correctly identify "Novice" nurses; 0.40). For the subjective measure 'pass/fail', sensitivity was high (0.90) but specificity was low (0.47). The Time to Task measure had statistical significance between Expert and Novice groups (p < 0.01) and sensitivity (0.80) and specificity (0.85) were good.

CONCLUSION

Commonly used subjective measures of clinical nursing competence have difficulties with achieving acceptable specificity. However, an objective measure, Time to Task, had good sensitivity and specificity in differentiating between groups. While more than one assessment instrument should be used to determine nurse competency, an objective measure, such as Time to Task, warrants further study.

Utilising an innovative digital software to grade pre-clinical crown preparation exercise

BACKGROUND

Accurate assessment of dental students' pre-clinical work is the most critical component of the dental education process. Thus, this study came to investigate the effectiveness of using technology in students' pre-clinical work evaluation; by comparing grades generated from a digital assessment software of a prepared tooth and a traditional visual inspection carried out by four calibrated faculty members.

METHODS

Ninety-six teeth were prepared for a ceramo-metal crown by fourth year dental students. The four examiners and the digital grading software evaluated independently each preparation once. A random sample of 20 preparations were graded twice to assess intra-rater reliability. Inter-class correlation (ICC) was used to measure agreement among the four examiners, and between the examiners and the digital grading software. Paired student t-test was used to assess the accuracy of grades generated from visual inspection when compared to the digital grading system.

RESULTS

Intra-rater reliability for examiners 1 and 2 were 0.73 and 0.78 and for the digital grading system was 0.99. The inter-rater reliability among the four examiners was very good, ICC of 0.76. However, the agreement between scores produced by the examiners and the digital system were mostly in the low to moderate range. The paired t-test demonstrated statistically significant differences between each examiner and the digital grading by 6-25 grades.

CONCLUSIONS

This study demonstrates that the digital grading system used in this study can reliably scan and compare students' tooth preparations to a known gold standard. Results of this study suggests that using digital grading will preclude the variability and the subjectivity that usually result from the traditional visual inspection grading.

Examiner effect on the objective structured clinical exam - a study at five medical schools

BACKGROUND

The Objective Structured Clinical Examination (OSCE) is increasingly used at medical schools to assess practical competencies. To compare the outcomes of students at different medical schools, we introduced standardized OSCE stations with identical checklists.

METHODS

We investigated examiner bias at standardized OSCE stations for knee- and shoulder-joint examinations, which were implemented into the surgical OSCE at five different medical schools. The checklists for the assessment consisted of part A for knowledge and performance of the skill and part B for communication and interaction with the patient. At each medical faculty, one reference examiner also scored independently to the local examiner. The scores from both examiners were compared and analysed for inter-rater reliability and correlation with the level of clinical experience. Possible gender bias was also evaluated.

RESULTS

In part A of the checklist, local examiners graded students higher compared to the reference examiner; in part B of the checklist, there was no trend to the findings. The inter-rater reliability was weak, and the scoring correlated only weakly with the examiner's level of experience. Female examiners rated generally higher, but male examiners scored significantly higher if the examinee was female.

CONCLUSIONS

These findings of examiner effects, even in standardized situations, may influence outcome even when students perform equally well. Examiners need to be made aware of these biases prior to examining.

Human structure in six and one-half weeks: one approach to providing foundational anatomical competency in an era of compressed medical school anatomy curricula

The University of Oklahoma College of Medicine reduced gross anatomy from a full semester, 130-hour course to a six and one-half week, 105-hour course as part of a new integrated systems-based pre-clinical curriculum. In addition to the reduction in contact hours, content from embryology, histology, and radiology were added into the course. The new curriculum incorporated best practices in the area of regular assessments, feedback, clinical application, multiple teaching modalities, and professionalism. A comparison of the components of the traditional and integrated curriculum, along with end of course evaluations and student performance revealed that the new curriculum was just as effective, if not more effective. This article also provides important lessons learned.

Students' self-assessment: a learning tool and its comparison with the faculty assessments

OBJECTIVE

This study compared the student's self-grades versus the examiners grades, inter examiner grades and grades of anterior with posterior teeth in a preclinical prosthodontic course.

METHODS

75 students and 2 examiners participated in the study. The students prepared one anterior (upper central incisor) and one posterior (lower first molar) teeth for full veneer crowns in allocated time of 2 hours and 30 minutes. After the preparations, the students self-graded their preparations based on criteria-based evaluation forms. The examiners also completed the grading for the prepared teeth. All the grades were recorded, comparisons were made using SPSS version 21 and results tabulated.

RESULTS

The means of grades (8.32) by the students themselves were found to be higher compared to the examiners grades (7.3) for the anterior as well as posterior teeth. Comparison of the grades for the anterior/posterior teeth and the overall grades showed a statistically significant difference (p=0.000). A moderate correlation (0.399) and a strong correlation (0.601) were found between the grades of the faculty and the students for the anterior and posterior teeth respectively. The overall grading for the anterior and posterior teeth by the two faculty members showed no statistically significant difference (p=0.053) and a very strong correlation (0.784). The results of the test showed a significant difference (p=0.001) between the overall grading for anterior and posterior teeth.

CONCLUSION

Students tended to grade their teeth preparations higher compared to the examiner grades, inter examiner variation in the grades existed and the grades of the anterior teeth were higher compared to the posterior teeth.

Expertise in performance assessment: assessors' perspectives

The recent rise of interest among the medical education community in individual faculty making subjective judgments about medical trainee performance appears to be directly related to the introduction of notions of integrated competency-based education and assessment for learning. Although it is known that assessor expertise plays an important role in performance assessment, the roles played by different factors remain to be unraveled. We therefore conducted an exploratory study with the aim of building a preliminary model to gain a better understanding of assessor expertise. Using a grounded theory approach, we conducted seventeen semi-structured interviews with individual faculty members who differed in professional background and assessment experience. The interviews focused on participants' perceptions of how they arrived at judgments about student performance. The analysis resulted in three categories and three recurring themes within these categories: the categories assessor characteristics, assessors' perceptions of the assessment tasks, and the assessment context, and the themes perceived challenges, coping strategies, and personal development. Central to understanding the key processes in performance assessment appear to be the dynamic interrelatedness of the different factors and the developmental nature of the processes. The results are supported by literature from the field of expertise development and in line with findings from social cognition research. The conceptual framework has implications for faculty development and the design of programs of assessment.

Summative assessments are more powerful drivers of student learning than resource intensive teaching formats

BACKGROUND

Electrocardiogram (ECG) interpretation is a core clinical skill that needs to be acquired during undergraduate medical education. Intensive teaching is generally assumed to produce more favorable learning outcomes, but recent research suggests that examinations are more powerful drivers of student learning than instructional format. This study assessed the differential contribution of teaching format and examination consequences to learning outcome regarding ECG interpretation skills in undergraduate medical students.

METHODS

A total of 534 fourth-year medical students participated in a six-group (two sets of three), partially randomized trial. Students received three levels of teaching intensity: self-directed learning (two groups), lectures (two groups) or small-group peer teaching facilitated by more advanced students (two groups). One of the two groups on each level of teaching intensity was assessed in a formative, the other in a summative written ECG examination, which provided a maximum of 1% credit points of the total curriculum. The formative examination provided individual feedback without credit points. Main outcome was the correct identification of ≥3 out of 5 diagnoses in original ECG tracings. Secondary outcome measures were time spent on independent study and use of additional study material.

RESULTS

Compared with formative assessments, summative assessments increased the odds of correctly identifying at least three out of five ECG diagnoses (OR 5.14; 95% CI 3.26 to 8.09), of spending at least 2 h/week extra on ECG self-study (OR 4.02; 95% CI 2.65 to 6.12) and of using additional learning material (OR 2.86; 95% CI 1.92 to 4.24). Lectures and peer teaching were associated with increased learning effort only, but did not augment examination performance.

CONCLUSIONS

Medical educators need to be aware of the paramount role of summative assessments in promoting student learning. Consequently, examinations within medical schools need to be closely matched to the desired learning outcomes. Shifting resources from implementing innovative and costly teaching formats to designing more high-quality summative examinations warrants further investigation.

Explicit feedback to enhance the effect of an interim assessment: a cross-over study on learning effect and gender difference

In a previous study we demonstrated by a prospective controlled design that an interim assessment during an ongoing small group work (SGW) session resulted in a higher score in the course examination. As this reflects the so-called testing effect, which is supposed to be enhanced by feedback, we investigated whether feedback following an interim assessment would have an effect on the score of the course exam, and whether the effect is influenced by the gender of the student. During a General Pathology bachelor course all 386 (bio) medical students took an interim assessment on the topics cell damage (first week) and tumour pathology (fourth week). The intervention consisted of immediate detailed oral feedback on the content of the questions of the interim assessment by the tutor, including the rationale of the correct and incorrect answers. It concerned a prospective randomized study using a cross-over design. Outcome measures were: (1) the difference in the normalized scores (1-10) of the course examination multiple choice questions related to the two topics, (2) effect of gender, and (3) gender-specific scores on formal examination. The effect of feedback was estimated as half the difference in the outcome between the two conditions. Mixed-model analysis was used whereby the SGW group was taken as the study target. The scores of the questions on cell damage amounted to 7.70 (SD 1.59) in the group without and 7.78 (SD 1.39) in the group with feedback, and 6.73 (SD 1.51) and 6.77 (SD 1.60), respectively, for those on tumour pathology. No statistically significant effect of feedback was found: 0.02 on a scale of 1-10 (95 % CI: -0.20; 0.25). There were no significant interactions of feedback with gender. Female students scored 0.43 points higher on the formal examination in comparison with their male colleagues. No additional effect of immediate explicit feedback following an interim assessment during an SGW session in an ongoing bachelor course could be demonstrated in this prospective randomized controlled study. Gender analysis revealed a higher performance of female students on the formal examination, which could not be explained by the effect of feedback in the current study. In this particular learning environment, SGW, explicit feedback may have little added value to the interactive learning that includes implicit feedback.

How to achieve synergy between medical education and cognitive neuroscience? An exercise on prior knowledge in understanding

A major challenge in contemporary research is how to connect medical education and cognitive neuroscience and achieve synergy between these domains. Based on this starting point we discuss how this may result in a common language about learning, more educationally focused scientific inquiry, and multidisciplinary research projects. As the topic of prior knowledge in understanding plays a strategic role in both medical education and cognitive neuroscience it is used as a central element in our discussion. A critical condition for the acquisition of new knowledge is the existence of prior knowledge, which can be built in a mental model or schema. Formation of schemas is a central event in student-centered active learning, by which mental models are constructed and reconstructed. These theoretical considerations from cognitive psychology foster scientific discussions that may lead to salient issues and questions for research with cognitive neuroscience. Cognitive neuroscience attempts to understand how knowledge, insight and experience are established in the brain and to clarify their neural correlates. Recently, evidence has been obtained that new information processed by the hippocampus can be consolidated into a stable, neocortical network more rapidly if this new information fits readily into a schema. Opportunities for medical education and medical education research can be created in a fruitful dialogue within an educational multidisciplinary platform. In this synergetic setting many questions can be raised by educational scholars interested in evidence-based education that may be highly relevant for integrative research and the further development of medical education.

Toward meaningful evaluation of medical trainees: the influence of participants' perceptions of the process

An essential goal of evaluation is to foster learning. Across the medical education spectrum, evaluation of clinical performance is dominated by subjective feedback to learners based on observation by expert supervisors. Research in non-medical settings has suggested that participants' perceptions of evaluation processes exert considerable influence over whether the feedback they receive actually facilitates learning, but similar research on perceptions of feedback in the medical setting has been limited. In this review, we examine the literature on recipient perceptions of feedback and how those perceptions influence the contribution that feedback makes to their learning. A focused exploration of relevant work on this subject in higher education and industrial psychology settings is followed by a detailed examination of available research on perceptions of evaluation processes in medical settings, encompassing both trainee and evaluator perspectives. We conclude that recipients' and evaluators' perceptions of an evaluation process profoundly affect the usefulness of the evaluation and the extent to which it achieves its goals. Attempts to improve evaluation processes cannot, therefore, be limited to assessment tool modification driven by reliability and validity concerns, but must also take account of the critical issue of feedback reception and the factors that influence it. Given the unique context of clinical performance evaluation in medicine, a research agenda is required that seeks to more fully understand the complexity of the processes of giving, receiving, interpreting, and using feedback as a basis for real progress toward meaningful evaluation.

Challenging medical students with an interim assessment: a positive effect on formal examination score in a randomized controlled study

Until now, positive effects of assessment at a medical curriculum level have not been demonstrated. This study was performed to determine whether an interim assessment, taken during a small group work session of an ongoing biomedical course, results in students' increased performance at the formal course examination. A randomized controlled trial was set up, with an interim assessment without explicit feedback as intervention. It was performed during a regular biomedical Bachelor course of 4 weeks on General Pathology at the Radboud University Nijmegen Medical Centre. Participants were 326 medical and 91 biomedical science students divided into three study arms: arm Intervention-1 (I-1) receiving one interim assessment; arm I-2 receiving two interim assessments, and control arm C, receiving no interim assessment. The study arms were stratified for gender and study discipline. The interim assessment consisted of seven multiple-choice questions on tumour pathology. Main outcome measures were overall score of the formal examination (scale 1-10), and the subscore of the questions on tumour pathology (scale 1-10). We found that students who underwent an interim assessment (arm I) had a 0.29-point (scale 1-10) higher score on the formal examination than the control group (p = 0.037). For the questions in the formal examination on tumour pathology the score amounted to 0.47 points higher (p = 0.007), whereas it was 0.17 points higher for the questions on topics related to the previous 3 weeks. No differences in formal examination score were found between arms I-1 and I-2 (p = 0.817). These findings suggest that an interim assessment during a small group work session in a randomized study setting stimulates students to increase their formal examination score.

Centralized assessment in graduate medical education: cents and sensibilities

RATIONALE

The start of residency represents an "educational handoff." Accreditation and credentialing organizations have called for better assessments of learner and patient outcomes and improved patient safety and quality of care.

INTERVENTION

We describe the development of centralized assessments of baseline, core residency competencies at 2 institutions, and summarize principles and lessons learned for other institutions interested in developing similar interventions.

RESULTS

At one institution, 70% of 1 083 new residents assessed via the Objective Standardized Clinical Examination stated they learned a new skill; 80% believe it was a useful way to spend orientation; 78% felt better prepared for aspects of internship; and 80% would recommend it for next year's interns. High levels of satisfaction are expressed by participants at the other institution, especially with the immediate provision of feedback after each station. At this institution, average new resident performance in the communication skills domain approached 90%, but patient care domain scores showed wide variability. The lowest scores were related to performing the psychomotor skills of aseptic technique.

DISCUSSION

From a patient safety perspective, results suggest a need to improve the preparation of new residents, along with careful supervision of their early clinical work. The presence of skill deficits likely adds to the highly stressful transition into residency. Teaching institutions may use centralized assessment to enhance education and patient safety and to promote accountability to accrediting bodies, residents, and patients. The approach may identify gaps in the undergraduate curriculum. The addition of hand hygiene and aseptic technique teaching and assessment modules are currently being piloted at each of the institutions.