Validity, reliability and feasibility of a new observation rating tool and a post encounter rating tool for the assessment of clinical reasoning skills of medical students during their internal medicine clerkship: a pilot study

Development and psychometric testing of the Diagnostic Competency During Simulation-based (DCDS) learning tool

BACKGROUND

Despite diagnostic errors impacting an estimated 12 million people yearly in the United States, educational strategies that foster diagnostic performance among nurse practitioner (NP) students remain elusive. One possible solution is to focus explicitly on competencies fundamental for diagnostic excellence. Currently, no educational tools were found that comprehensively address individual diagnostic reasoning competencies during simulated-based learning experiences.

PURPOSE

Our research team developed and explored psychometric properties of the "Diagnostic Competency During Simulation-based (DCDS) Learning Tool."

METHOD

Items and domains were developed based on existing frameworks. Content validity was determined by a convenience sample of eight experts. Inter-rater reliability was determined by four faculty rating eight simulation scenarios.

RESULTS

Final individual competency domain scale content validity index (CVI) scores ranged between 0.9175 and 1.0; total scale CVI score was 0.98. The intra-class correlation coefficient (ICC) for the tool was 0.548 (p < 0.0001, 95 % confidence interval CI [0.482-0.612]).

CONCLUSIONS

Results suggest that the DCDS Learning Tool is relevant to diagnostic reasoning competencies and may be implemented with moderate reliability across varied simulation scenarios and performance levels. The DCDS tool expands the landscape of diagnostic reasoning assessment by providing NP educators with granular, actionable, competency-specific assessment measures to foster improvement.

Feasibility and Effectiveness of Mini-Clinical Evaluation Exercise (Mini-CEX) in an Undergraduate Medical Program: A Study From Pakistan

Background

In clinical settings, direct observation (DO) with feedback is an effective method to assess and improve learner performance. One tool used for DO is the mini-clinical evaluation exercise (Mini-CEX). We conducted a study to assess the effectiveness and feasibility of Mini-CEX for medical students at Aga Khan University, Karachi.

Methods

Utilizing a purposive sampling technique, a total of 199 students in six core clerkships of Years 3 and 4 were selected for this study. Participating faculty underwent training workshops for use of Mini-CEX and feedback strategies. Each student was assessed twice by one faculty, using a modified version of the Mini-CEX, which assessed four domains of clinical skills: Data Gathering, Communication, Diagnosis/Differential, and Management Plan and Organization. Feedback was given after each encounter. Faculty and students also provided detailed feedback regarding the process of DO.

Data were analyzed using Statistical Package for Social Sciences (SPSS) version 26 (IBM Corp., Armonk, NY, USA), with categorical variables arranged as frequencies and percentages. The Chi-squared test was used for further statistical analyses, and a P-value of < 0.05 was considered statistically significant. Effectiveness was assessed via a change in student performance between the first and second Mini-CEX, and feasibility was assessed via qualitative feedback.

Results

We obtained three sets of results: Mini-CEX forms (523), from which we included a total 350 evaluations for analysis, 216 from Year 3 and 134 from Year 4, and feedback on DO: student (70) and faculty (18). Year 3 students performed significantly better in all foci of the Mini-CEX between the first and second assessment (P ≤ 0.001), whereas in Year 4, significant improvement was limited to only two domains of the Mini-CEX [Communication of History/Physical Examination (P = 0.040) and Diagnosis/Differential and Management Plan (P < 0.001)]. Students (65.7%) and faculty (94.4%) felt this exercise improved their interaction. 83.3% faculty recommended its formal implementation compared to 27.1% of students, who reported challenges in implementation of the Mini-CEX such as time constraints, logistics, the subjectivity of assessment, and varying interest by faculty.

Conclusion

Direct observation using Mini-CEX is effective in improving the clinical and diagnostic skills of medical students and strengthens student-faculty interaction. While challenges exist in its implementation, the strategic placement of Mini-CEX may enhance its utility in measuring student competency.

Assessing clinical reasoning ability in fourth-year medical students via an integrative group history-taking with an individual reasoning activity

BACKGROUND

The most important factor in evaluating a physician's competence is strong clinical reasoning ability, leading to correct principal diagnoses. The process of clinical reasoning includes history taking, physical examinations, validating medical records, and determining a final diagnosis. In this study, we designed a teaching activity to evaluate the clinical reasoning competence of fourth-year medical students.

METHODS

We created five patient scenarios for our standardized patients, including hemoptysis, abdominal pain, fever, anemia, and chest pain. A group history-taking with individual reasoning principles was implemented to teach and evaluate students' abilities to take histories, document key information, and arrive at the most likely diagnosis. Residents were trained to act as teachers, and a post-study questionnaire was employed to evaluate the students' satisfaction with the training activity.

RESULTS

A total of 76 students, five teachers, and five standardized patients participated in this clinical reasoning training activity. The average history-taking score was 64%, the average key information number was 7, the average diagnosis number was 1.1, and the average correct diagnosis rate was 38%. Standardized patients presenting with abdominal pain (8.3%) and anemia (18.2%) had the lowest diagnosis rates. The scenario of anemia presented the most difficult challenge for students in history taking (3.5/5) and clinical reasoning (3.5/5). The abdominal pain scenario yielded even worse results (history taking: 2.9/5 and clinical reasoning 2.7/5). We found a correlation in the clinical reasoning process between the correct and incorrect most likely diagnosis groups (group history-taking score, p = 0.045; key information number, p = 0.009 and diagnosis number, p = 0.004). The post-study questionnaire results indicated significant satisfaction with the teaching program (4.7/5) and the quality of teacher feedback (4.9/5).

CONCLUSIONS

We concluded that the clinical reasoning skills of fourth-year medical students benefited from this training course, and the lower correction of the most likely diagnosis rate found with abdominal pain, anemia, and fever might be due to a system-based teaching modules in fourth-year medical students; cross-system remedial reasoning auxiliary training is recommended for fourth-year medical students in the future.