Multisource Assessment for Development Purposes: Revisiting the Methodology of Data Analysis

The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool's psychometric properties in different specialties

BACKGROUND

Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors' feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident's performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses' assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool's performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa.

METHODS

O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed.

RESULTS

179 O-RONs were completed on 30 residents. With four forms per resident, the ORON's reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001).

CONCLUSIONS

Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.

Family climate influences next-generation family business leader effectiveness and work engagement

Effective next-generation leadership is central to the multi-generational survival of family businesses. This study of 100 next-generation family business leaders found that business-owning families that openly express their opinions, take time to listen to each other, and squarely address difficult issues positively influence the development of the emotional and social intelligence competencies in next-generation family leaders that drive their leadership effectiveness. That kind of open and transparent communication in the family also makes it more likely next-generation leaders will be held accountable for their leadership performance by others, which increases the degree to which they are positively engaged with their work in the family firm. On the other hand, the results suggest that senior-generation family leaders who lead autocratically, a leadership style often observed in entrepreneurs who found family firms, make it less likely that next-generation family leaders will learn the emotional and social intelligence competencies that predict their leadership effectiveness. The study also found that autocratic senior-generation leaders negatively affect next-generation leader self-efficacy and make it less likely that others will hold them accountable, which limits their engagement with work in the family business. One of the study's most important findings is that next-generation leader acceptance of personal responsibility for their leadership behaviors and results serves as a mediator through which the nature of the family climate influences their leadership effectiveness and work engagement. This suggests that while the nature of family relationships may make it easier or more difficult, next-generation family leaders have ultimate control over the development of their leadership talent and the inspiration, enthusiasm, energy, and pride they feel when working in the family business.

A Training Intervention through a 360° Multisource Feedback Model

Physicians and other health sciences professionals need continuous training, not only in technical aspects of their activity but also in nontechnical, transversal competencies with a cost-efficient impact on the proper functioning of healthcare. The objective of this paper is to analyze the behavioral change among health professionals at a large public hospital following a training intervention on a set of core nontechnical competencies: Teamwork, Adaptability-Flexibility, Commitment-Engagement, Results Orientation, and Leadership Skills for Supervisors. The 360° Multisource Feedback (MSF) model was applied using three sources of information: supervisors, co-workers, and the workers themselves (self-assessment). A quasi-experimental pretest–post-test single-group design with two points in time was utilized. The training intervention improved the scores of only one of the trained competencies—the “Results Orientation” competency—although the scores were slightly inflated. Moreover, significant discrepancies were detected between the three sources, with supervisors awarding the highest scores. The magnitude of behavioral change was related to certain sociodemographic and organizational variables. The study was not immune to the ceiling effect, despite control measures aimed at avoiding it. The empirical evidence suggests that the 360° MSF model must be maintained over time to enhance and reinforce an evaluation culture for better patient care.