Peer-assessment of medical communication skills: the impact of students' personality, academic and social reputation on behavioural assessment

Peer assessment in medical communication skills training in programmatic assessment: A qualitative study examining faculty and student perceptions

INTRODUCTION

Current literature recommends assessment of communication skills in medical education combining different settings and multiple observers. There is still a gap in understanding about whether and how peers assessment facilitates learning in communication skills training.

METHODS

We designed a qualitative study using focus group interviews and thematic analysis, in a medical course in the Netherlands. We aimed to explore medical students' and teachers' experiences, perceptions, and perspectives about challenges and facilitating factors in PACST (Peer assessment in medical communication skills training).

RESULTS

Most of the participants reported that peer feedback was a valuable experience when learning communication skills. The major challenges for the quality and credibility of PACST reported by the participants are the question whether peer feedback is critical enough for learning and the difficulty of actually engaging students in the assessment process.

CONCLUSION

Teachers reviewing students' peer assessments may improve the quality and their credibility and the reviewed assessments can best be used for learning purposes. We suggest to pay sufficient attention to teachers' roles in PACST, ensuring a safe and trustworthy environment and additionally helping students to internalize the value of being vulnerable during the evaluation process.

Peer feedback as a medium to facilitate reflective practice among pharmacy students in a case-based learning environment

INTRODUCTION

The ability to reflect is a key element in preparing pharmacy professionals to meet the challenges of a dynamic health care environment. This mixed-methods study explored the pedagogical benefits of peer feedback by designing, developing, and implementing a peer feedback activity to facilitate reflective practice among pharmacy students.

METHODS

Twenty second-year doctor of pharmacy (PharmD) students in a required pharmacotherapy course participated in a systematic peer feedback activity and five of these students volunteered for semi-structured interviews.

RESULTS

No significant correlation was found between perceived effectiveness of peer feedback and students' reflective thinking skills. Qualitative interview data revealed three major themes regarding PharmD students' perception of peer feedback as an instructional strategy to promote reflective practice: (1) the cognitive process of providing feedback, (2) the cognitive process after receiving peer feedback, and (3) perceptions of peer feedback as a tool to exercise reflective practice.

CONCLUSIONS

Although limited in sample size, important lessons were learned on how to design, develop, and implement a peer feedback activity.

Peer overmarking and insufficient diagnosticity: the impact of the rating method for peer assessment

The present study explores two rating methods for peer assessment (analytical rating using criteria and comparative judgement) in light of concurrent validity, reliability and insufficient diagnosticity (i.e. the degree to which substandard work is recognised by the peer raters). During a second-year undergraduate course, students wrote a one-page essay on an air pollutant. A first cohort (N = 260) relied on analytical rating using criteria to assess their peers' essays. A total of 1297 evaluations were made, and each essay received at least four peer ratings. Results indicate a small correlation between peer and teacher marks, and three essays of substandard quality were not recognised by the group of peer raters. A second cohort (N = 230) used comparative judgement. They completed 1289 comparisons, from which a rank order was calculated. Results suggest a large correlation between the university teacher marks and the peer scores and acceptable reliability of the rank order. In addition, the three essays of substandard quality were discerned as such by the group of peer raters. Although replication research is warranted, the results provide the first evidence that, when peer raters overmark and fail to identify substandard work using analytical rating with criteria, university teachers may consider changing the rating method of the peer assessment to comparative judgement.

Peer evaluations of group work in different years of medical school and academic achievement: how are they related?

BACKGROUND

To develop the skills needed in health care teams, training communication and teamwork skills are important in medical education. Small group collaborative learning is one of the methods utilized in such trainings, and peer evaluation is suggested to be useful in reinforcing the effectiveness of group learning activities. In Mie University Faculty of Medicine, group work consisting of book review sessions of liberal arts education in the first grade and problem-based learning (PBL) sessions in preclinical years were conducted using the same peer evaluation system that included three domains: degree of prior learning, contribution to group discussion, and cooperative attitude. This study was conducted to determine the relationships among behaviors during group work and the academic achievement of medical students.

METHODS

With the data from a cohort of medical students in three consecutive academic years (n = 340), peer evaluation scores in groupworks of book review sessions, those in PBL sessions and paper test scores of preclinical years were analyzed. The correlations were analyzed with Spearman's correlation coefficient, and the respective scores were compared by using the Wilcoxon signed-ranked test.

RESULTS

Significant correlations were observed among the evaluation scores of respective domains in group work and paper test scores. The degree of prior learning had the strongest relationship among the three domains (r_s = 0.355, p < 0.001 between book review sessions and PBL; r_s = 0.338, p < 0.001 between book review sessions and paper test score; r_s = 0.551, p < 0.001 between PBL and paper test score). Peer evaluation scores of respective domains were found to be significantly higher in PBL.

CONCLUSION

Medical students maintained their groupwork behaviors to some extent from early school to preclinical years. Those behaviors were positively related to their academic achievement in the later years of the medical education curriculum. Our study highlighted the importance of the early introduction of group work. The results will be useful to motivate medical students to put more effort into group work.

Breaking bad news: A randomized controlled trial to test a novel interactive course for medical students using blended learning

OBJECTIVE

Breaking bad news (BBN) is challenging for physicians and patients and specific communication strategies aim to improve these situations. This study evaluates whether an E-learning assignment could improve medical students' accurate recognition of BBN communication techniques.

METHODS

This randomized controlled trial was conducted at the University of Basel. After a lecture on BBN, 4th year medical students were randomized to an intervention receiving an E-learning assignment on BBN or to a control group. Both groups then worked on an examination video and identified previously taught BBN elements shown in a physician-patient interaction. The number of correctly, misclassified and incorrectly identified BBN communication elements as well as missed opportunities were assessed in the examination video.

RESULTS

We included 160 medical students (55% female). The number of correctly identified BBN elements did not differ between control and intervention group (mean [SD] 3.51 [2.50] versus 3.72 [2.34], p = 0.58). However, the mean number of inappropriate BBN elements was significantly lower in the intervention than in the control group (2.33 [2.57] versus 3.33 [3.39], p = 0.037).

CONCLUSIONS

Use of an E-learning tool reduced inappropriate annotations regarding BBN communication techniques.

PRACTICE IMPLICATIONS

This E-learning might help to further advance communication skills in medical students.

Assessment of medical students' clinical performance using high-fidelity simulation: comparison of peer and instructor assessment

BACKGROUND

High-fidelity simulators are highly useful in assessing clinical competency; they enable reliable and valid evaluation. Recently, the importance of peer assessment has been highlighted in healthcare education, and studies using peer assessment in healthcare, such as medicine, nursing, dentistry, and pharmacy, have examined the value of peer assessment. This study aimed to analyze inter-rater reliability between peers and instructors and examine differences in scores between peers and instructors in the assessment of high-fidelity-simulation-based clinical performance by medical students.

METHODS

This study analyzed the results of two clinical performance assessments of 34 groups of fifth-year students at Ajou University School of Medicine in 2020. This study utilized a modified Queen's Simulation Assessment Tool to measure four categories: primary assessment, diagnostic actions, therapeutic actions, and communication. In order to estimate inter-rater reliability, this study calculated the intraclass correlation coefficient and used the Bland and Altman method to analyze agreement between raters. A t-test was conducted to analyze the differences in evaluation scores between colleagues and faculty members. Group differences in assessment scores between peers and instructors were analyzed using the independent t-test.

RESULTS

Overall inter-rater reliability of clinical performance assessments was high. In addition, there were no significant differences in overall assessment scores between peers and instructors in the areas of primary assessment, diagnostic actions, therapeutic actions, and communication.

CONCLUSIONS

The results indicated that peer assessment can be used as a reliable assessment method compared to instructor assessment when evaluating clinical competency using high-fidelity simulators. Efforts should be made to enable medical students to actively participate in the evaluation process as fellow assessors in high-fidelity-simulation-based assessment of clinical performance in situations similar to real clinical settings.

A Structured Peer Assessment Method with Regular Reinforcement Promotes Longitudinal Self-Perceived Development of Medical Students' Feedback Skills

BACKGROUND

Given that training is integral to providing constructive peer feedback, we examined the impact of a regularly reinforced, structured peer assessment method on student-reported feedback abilities throughout a two-year preclinical Communication Skills course.

METHODS

Three consecutive 32-student medical school classes were introduced to the Observation-Reaction-Feedback method for providing verbal assessment during Year 1 Communication Skills orientation. In biweekly small-group sessions, students received worksheets reiterating the method and practiced giving verbal feedback to peers. Periodic questionnaires evaluated student perceptions of feedback delivery and the Observation-Reaction-Feedback method.

RESULTS

Biweekly reinforcement of the Observation-Reaction-Feedback method encouraged its uptake, which correlated with reports of more constructive, specific feedback. Compared to non-users, students who used the method noted greater improvement in comfort with assessing peers in Year 1 and continued growth of feedback abilities in Year 2. Comfort with providing modifying feedback and verbal feedback increased over the two-year course, while comfort with providing reinforcing feedback and written feedback remained similarly high. Concurrently, student preference for feedback anonymity decreased.

CONCLUSIONS

Regular reinforcement of a peer assessment framework can increase student usage of the method, which promotes the expansion of self-reported peer feedback skills over time. These findings support investigation of analogous strategies in other medical education settings.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-021-01242-w.

Peer assessment of professionalism in undergraduate medical education

BACKGROUND

Fostering professional behaviour has become increasingly important in medical education and non-traditional approaches to assessment of professionalism may offer a more holistic representation of students' professional behaviour development. Emerging evidence suggests peer assessment may offer potential as an alternative method of professionalism assessment. We introduced peer assessment of professionalism in pre-clerkship phases of undergraduate medical education curriculum at our institution and evaluated suitability of adopting a professional behaviour scale for longitudinal tracking of student development, and student comfort and acceptance of peer assessment.

METHODS

Peer assessment was introduced using a validated professional behaviours scale. Students conducted repeated, longitudinal assessments of their peers from small-group, clinical skills learning activities. An electronic assessment system was used to collect peer assessments, collate and provide reports to students. Student opinions of peer assessment were initially surveyed before introducing the process, confirmatory analyses were conducted of the adopted scale, and students were surveyed to explore satisfaction with the peer assessment process.

RESULTS

Students across all phases of the curriculum were initially supportive of anonymous peer assessment using small-group learning sessions. Peer scores showed improvement over time, however the magnitude of increase was limited by ceiling effects attributed to the adopted scale. Students agreed that the professional behaviours scale was easy to use and understand, however a majority disagreed that peer assessment improved their understanding of professionalism or was a useful learning experience.

CONCLUSIONS

Peer assessment of professional behaviours does expose students to the process of assessing one's peers, however the value of such processes at early stages of medical education may not be fully recognized nor appreciated by students. Electronic means for administering peer assessment is feasible for collecting and reporting peer feedback. Improvement in peer assessed scores was observed over time, however student opinions of the educational value were mixed and indeterminate.

Validation of a competence-based assessment of medical students' performance in the physician's role

BACKGROUND

Assessing competence of advanced undergraduate medical students based on performance in the clinical context is the ultimate, yet challenging goal for medical educators to provide constructive alignment between undergraduate medical training and professional work of physicians. Therefore, we designed and validated a performance-based 360-degree assessment for competences of advanced undergraduate medical students.

METHODS

This study was conducted in three steps: 1) Ten facets of competence considered to be most important for beginning residents were determined by a ranking study with 102 internists and 100 surgeons. 2) Based on these facets of competence we developed a 360-degree assessment simulating a first day of residency. Advanced undergraduate medical students (year 5 and 6) participated in the physician's role. Additionally knowledge was assessed by a multiple-choice test. The assessment was performed twice (t1 and t2) and included three phases: a consultation hour, a patient management phase, and a patient handover. Sixty-seven (t1) and eighty-nine (t2) undergraduate medical students participated. 3) The participants completed the Group Assessment of Performance (GAP)-test for flight school applicants to assess medical students' facets of competence in a non-medical context for validation purposes. We aimed to provide a validity argument for our newly designed assessment based on Messick's six aspects of validation: (1) content validity, (2) substantive/cognitive validity, (3) structural validity, (4) generalizability, (5) external validity, and (6) consequential validity.

RESULTS

Our assessment proved to be well operationalised to enable undergraduate medical students to show their competences in performance on the higher levels of Bloom's taxonomy. Its generalisability was underscored by its authenticity in respect of workplace reality and its underlying facets of competence relevant for beginning residents. The moderate concordance with facets of competence of the validated GAP-test provides arguments of convergent validity for our assessment. Since five aspects of Messick's validation approach could be defended, our competence-based 360-degree assessment format shows good arguments for its validity.

CONCLUSION

According to these validation arguments, our assessment instrument seems to be a good option to assess competence in advanced undergraduate medical students in a summative or formative way. Developments towards assessment of postgraduate medical trainees should be explored.

Peer assessment to improve medical student's contributions to team-based projects: randomised controlled trial and qualitative follow-up

BACKGROUND

Medical schools increasingly incorporate teamwork in their curricula but medical students often have a negative perception of team projects, in particular when there is unequal participation. The purpose of this study is to evaluate whether a novel peer evaluation system improves teamwork contributions and reduces the risk of students "free loading".

METHODS

A cluster randomised controlled trial (RCT) with qualitative follow up enrolled 37 teams (n = 223 students). Participating teams were randomised to intervention group (19 teams) or control group (18 teams). The validated Comprehensive Assessment Team Member Effectiveness (CATME) tool was used as the outcome measure, and was completed at baseline (week 2) and at the end of the project (week 10). The team contribution subscale was the primary outcome, with other subscales as secondary outcomes. Six focus group discussions were held with students to capture the team's experiences and perceptions of peer assessment and its effects on team work.

RESULTS

The results of the RCT showed that there was no difference in team contribution, and other forms of team effectiveness, between intervention and control teams. The focus group discussions highlighted students' negative attitudes, and lack of implementation of this transparent, points-based peer assessment system, out of fear of future consequences for relationships with peers. The need to assess peers in a transparent way to stimulate open discussion was perceived as threatening by participants. Teams suggested that other peer assessment systems could work such as rewarding additional or floating marks to high performing team members.

CONCLUSIONS

Other models of peer assessment need to be developed and tested that are non-threatening and that facilitate early acceptance of this mode of assessment.

Reliability of peer assessment of patient education simulations

BACKGROUND

At Belmont University College of Pharmacy, the final introductory pharmacy practice experience (IPPE) course in the IPPE series, IPPE V, is designed to assess readiness for advanced pharmacy practice experiences and includes three patient counseling simulations. These simulations have required greater resources. The objective of our study was to determine if student performance on patient counseling simulations can be accurately assessed by peers.

EDUCATIONAL ACTIVITY

Students were required to participate in patient counseling simulations throughout the semester. For each simulation, students were assigned one role: pharmacist, patient, or peer-evaluator. Each pharmacist counseled the patient on a specific product while the peer-evaluator assessed the accuracy and completeness of the counseling using a detailed checklist. The patient used a checklist to assess the pharmacist's communication skills. Faculty assessed the student evaluators and the patients by counting the number of discrepancies between the student evaluator's and the live faculty checklists. Students were surveyed at the end of the semester regarding their beliefs and perceptions of peer assessment for the communication simulations.

CRITICAL ANALYSIS OF THE EDUCATIONAL ACTIVITY

Of 65 students enrolled in the spring 2018 course, complete recordings and checklists were available for 54 simulations (83.1%). Interrater reliability was high with all correlation coefficients exceeding 0.86. Students agreed that they were comfortable assessing patient education content (82.14%) and communication skills (82.14%). Our results indicate that peer evaluation during patient education simulation is reliable and acceptable to students.

General principles to consider when designing a clinical communication assessment program

OBJECTIVES

Assessment of clinical communication helps teachers in healthcare education determine whether their learners have acquired sufficient skills to meet the demands of clinical practice. The aim of this paper is to give input to educators when planning how to incorporate assessment into clinical communication teaching by building on the authors' experience and current literature.

METHODS

A summary of the relevant literature within healthcare education is discussed, focusing on what and where to assess, how to implement assessment and how to choose appropriate methodology.

RESULTS

Establishing a coherent approach to teaching, training, and assessment, including assessing communication in the clinical context, is discussed. Key features of how to implement assessment are presented including: establishing a system with both formative and summative approaches, providing feedback that enhances learning and establishing a multi-source and longitudinal assessment program.

CONCLUSIONS

The implementation of a reliable, valid, credible, feasible assessment method with specific educational relevance is essential for clinical communication teaching.

PRACTICE IMPLICATIONS

All assessment methods have strengths and limitations. Since assessment drives learning, assessment should be aligned with the purpose of the teaching program. Combining the use of different assessment formats, multiple observations, and independent measurements in different settings is advised.

Objectivity in subjectivity: do students' self and peer assessments correlate with examiners' subjective and objective assessment in clinical skills? A prospective study

OBJECTIVES

The qualitative subjective assessment has been exercised either by self-reflection (self-assessment (SA)) or by an observer (peer assessment (PA)) and is considered to play an important role in students' development. The objectivity of PA and SA by students as well as those by faculty examiners has remained debated. This matters most when it comes to a high-stakes examination. We explored the degree of objectivity in PA, SA, as well as the global rating by examiners being Examiners' Subjective Assessment (ESA) compared with Objective Structured Clinical Examinations (OSCE).

DESIGN

Prospective cohort study.

SETTING

Undergraduate medical students at Alfaisal University, Riyadh.

PARTICIPANTS

All second-year medical students (n=164) of genders, taking a course to learn clinical history taking and general physical examination.

MAIN OUTCOME MEASURES

A Likert scale questionnaire was distributed among the participants during selected clinical skills sessions. Each student was evaluated randomly by peers (PA) as well as by himself/herself (SA). Two OSCEs were conducted where students were assessed by an examiner objectively as well as subjectively (ESA) for a global rating of confidence and well-preparedness. OSCE-1 had fewer topics and stations, whereas OSCE-2 was terminal and full scale.

RESULTS

OSCE-1 (B=0.10) and ESA (B=8.16) predicted OSCE-2 scores. 'No nervousness' in PA (r=0.185, p=0.018) and 'confidence' in SA (r=0.207, p=0.008) correlated with 'confidence' in ESA. In 'well-preparedness', SA correlated with ESA (r=0.234, p=0.003).

CONCLUSIONS

OSCE-1 and ESA predicted students' performance in the OSCE-2, a high-stakes evaluation, indicating practical 'objectivity' in ESA, whereas SA and PA had minimal predictive role. Certain components of SA and PA correlated with ESA, suggesting partial objectivity given the limited objectiveness of ESA. Such difference in 'qualitative' objectivity probably reflects experience. Thus, subjective assessment can be used with some degree of objectivity for continuous assessment.

Effects of personality traits on collaborative performance in problem-based learning tutorials

OBJECTIVES

To examine the relationship between students' collaborative performance in a problem-based learning (PBL) environment and their personality traits. Methods:This retrospective, cross-sectional study was conducted using student data of a PBL program between 2013 and 2014 at Sungkyunkwan University School of Medicine, Seoul, South Korea. Eighty students were included in the study. Student data from the Temperament and Character Inventory were used as a measure of their personality traits. Peer evaluation scores during PBL were used as a measure of students' collaborative performance. Results: Simple regression analyses indicated that participation was negatively related to harm avoidance and positively related to persistence, whereas preparedness for the group work was negatively related to reward dependence. On multiple regression analyses, low reward dependence remained a significant predictor of preparedness. Grade-point average (GPA)  was negatively associated with novelty seeking and cooperativeness and was positively associated with persistence.  Conclusion: Medical students who are less dependent on social reward are more likely to complete assigned independent work to prepare for the PBL tutorials. The findings of this study can help educators better understand and support medical students who are at risk of struggling in collaborative learning environments.

Tools to Assess Behavioral and Social Science Competencies in Medical Education: A Systematic Review

PURPOSE

Behavioral and social science (BSS) competencies are needed to provide quality health care, but psychometrically validated measures to assess these competencies are difficult to find. Moreover, they have not been mapped to existing frameworks, like those from the Liaison Committee on Medical Education (LCME) and Accreditation Council for Graduate Medical Education (ACGME). This systematic review aimed to identify and evaluate the quality of assessment tools used to measure BSS competencies.

METHOD

The authors searched the literature published between January 2002 and March 2014 for articles reporting psychometric or other validity/reliability testing, using OVID, CINAHL, PubMed, ERIC, Research and Development Resource Base, SOCIOFILE, and PsycINFO. They reviewed 5,104 potentially relevant titles and abstracts. To guide their review, they mapped BSS competencies to existing LCME and ACGME frameworks. The final included articles fell into three categories: instrument development, which were of the highest quality; educational research, which were of the second highest quality; and curriculum evaluation, which were of lower quality.

RESULTS

Of the 114 included articles, 33 (29%) yielded strong evidence supporting tools to assess communication skills, cultural competence, empathy/compassion, behavioral health counseling, professionalism, and teamwork. Sixty-two (54%) articles yielded moderate evidence and 19 (17%) weak evidence. Articles mapped to all LCME standards and ACGME core competencies; the most common was communication skills.

CONCLUSIONS

These findings serve as a valuable resource for medical educators and researchers. More rigorous measurement validation and testing and more robust study designs are needed to understand how educational strategies contribute to BSS competency development.

Self-evaluation and peer-feedback of medical students' communication skills using a web-based video annotation system. Exploring content and specificity

OBJECTIVE

Self-evaluation and peer-feedback are important strategies within the reflective practice paradigm for the development and maintenance of professional competencies like medical communication. Characteristics of the self-evaluation and peer-feedback annotations of medical students' video recorded communication skills were analyzed.

METHOD

Twenty-five year 4 medical students recorded history-taking consultations with a simulated patient, uploaded the video to a web-based platform, marked and annotated positive and negative events. Peers reviewed the video and self-evaluations and provided feedback. Analyzed were the number of marked positive and negative annotations and the amount of text entered. Topics and specificity of the annotations were coded and analyzed qualitatively.

RESULTS

Students annotated on average more negative than positive events. Additional peer-feedback was more often positive. Topics most often related to structuring the consultation. Students were most critical about their biomedical topics. Negative annotations were more specific than positive annotations. Self-evaluations were more specific than peer-feedback and both show a significant correlation. Four response patterns were detected that negatively bias specificity assessment ratings.

CONCLUSION

Teaching students to be more specific in their self-evaluations may be effective for receiving more specific peer-feedback.

PRACTICE IMPLICATIONS

Videofragmentrating is a convenient tool to implement reflective practice activities like self-evaluation and peer-feedback to the classroom in the teaching of clinical skills.