Measuring continuing medical education conference impact and attendee experience: a scoping review

Objectives

The aim was to comprehensively identify published research evaluating continuing medical education conferences, to search for validated tools and perform a content analysis to identify the relevant domains for conference evaluation.

Methods

We used scoping review methodology and searched MEDLINE® for relevant English or French literature published between 2008 and 2022 (last search June 3, 2022). Original research (including randomized controlled trials, non-randomized studies, cohort, mixed-methods, qualitative studies, and editorial pieces) where investigators described impact, experience, or motivations related to conference attendance were eligible. Citations were assessed in triplicate, and data extracted in duplicate.

Results

Eighty-three studies were included, 69 (83%) of which were surveys or interview based, with the majority conducted at the end of or following conference conclusion. Of the 74 tools identified, only one was validated and was narrowly focused on a specific conference component. A total of 620 items were extracted and categorized into 4 a priori suggested domains (engagement-networking, education-learning, impact, scholarship), and an additional 4 identified through content analysis (value-satisfaction, logistics, equity-diversity-inclusivity, career influences). Time trends were evident, including the absence of items related to equity-diversity-inclusivity prior to 2019, and a focus on logistics, particularly technology and virtual conferences, since 2020.

Conclusions

This study identified 8 major domains relevant for continuing medical education conference evaluation. This work is of immediate value to individuals and organizations seeking to either design or evaluate a conference and represents a critical step in the development of a standardized tool for conference evaluation.

The German version of the mHealth App Usability Questionnaire (GER-MAUQ): Translation and validation study in patients with cardiovascular disease

Objective

In Germany, only a few standardized evaluation tools for assessing the usability of mobile Health apps exist so far. This study aimed to translate and validate the English patient version for standalone apps of the mHealth App Usability Questionnaire (MAUQ) into a German version.

Methods

Following scientific guidelines for translation and cross-cultural adaptation, the patient version for standalone apps was forward and back-translated from English into German by an expert panel. In total, 53 participants who were recruited as part of the beta testing process of the recently developed mHealth app HerzFit, answered the questions of the German version of the MAUQ (GER-MAUQ) and the System Usability Scale. Subsequently, a descriptive as well as a psychometric analysis was performed to test validity and reliability.

Results

After conducting three cognitive interviews, five items were modified. The values for Cronbach alpha for the entire questionnaire and the three subscales (0.966, 0.814, 0.910, and 0.909) indicate strong internal consistency. The correlation analysis revealed that the scores of the GER-MAUQ, the subscales and the SUS were strongly correlated with each other. The correlation coefficient of the SUS and the GER-MAUQ overall score was r = 0.854, P < 0.001 and the coefficients of the subscales and the SUS were r = 0.642, P < 0.001; r = 0.866, P < 0.001 and r = 0.643, P < 0.001.

Conclusions

We have developed a novel German version of the MAUQ and demonstrated it as a reliable and valid measurement tool for assessing the usability of standalone mHealth apps from the patients' perspective. The GER-MAUQ allows a new form of standardized assessment of usability of mHealth apps for patients with cardiovascular disease in Germany. Further research with a larger sample and other samples is recommended.

Professionals Evaluating Clients' Suitability for Digital Health and Social Care: Scoping Review of Assessment Instruments

BACKGROUND

Increased digital health and social care services are generally considered to improve people's access to services. However, not everyone can equally access and use these resources. Health and social care professionals should assess clients' suitability for digital solutions, but to succeed, they need information about what to evaluate and how.

OBJECTIVE

This scoping review aimed to identify evaluation tools that professionals can use when assessing clients' suitability for digital health and social care. We summarized the dimensions and the practical usefulness of the instruments.

METHODS

The MEDLINE (Ovid), CINAHL, Web of Science, and ASSIA databases were searched in February 2023 following the Joanna Briggs Institute's Manual for Evidence Synthesis. Studies were included if they focused on health and social care clients and professionals, examined clients' suitability for using digital health or social care, and applied related assessment methods in the direct client work of professionals. Studies focusing primarily on instruments intended for research use without clear applicability to professionals' practical contexts were excluded. Details of the eligible studies were extracted, and qualitative content analysis according to the research objectives was performed.

RESULTS

A total of 19 articles introducing 12 different assessment instruments intended for the health care context were included in the review. No instruments were found for evaluating the suitability for digital social care. The instruments contained 60 dimensions of the client's suitability for digital health, which reflected four perspectives: (1) skill-based suitability, (2) suitability based on general ability to maintain health, (3) suitability based on attitude and experience, and (4) suitability based on practical matters. The described practical usefulness of the instruments included professionals' possibility to (1) identify clients most in need of education and support, (2) direct and recommend the right clients for the right digital services, (3) ensure that clients can use digital health, (4) improve effectiveness and maximize the provision of digital health, (5) develop and redesign services, and (6) empower clients.

CONCLUSIONS

Based on the diverse assessment instruments available and the dimensions they measure, there seems to be no comprehensive evaluation tool for assessing clients' prerequisites to use digital solutions. It is important to further develop comprehensive screening tools applicable to professionals' busy work (both in health and social care) with defined threshold values for suitability.

Development of TOEKAN, a 360° evaluation tool for the clinical learning environment in General Practice postgraduate training

BACKGROUND

The General Practice (GP) postgraduate program exists for 80% out of workplace learning. The quality of the clinical learning environment (CLE) has a direct effect on the quality of training and the professional development of GP trainees.

METHODOLOGY

Participatory research was used to involve all stakeholders in the development process of a 360° evaluation tool that should improve the average quality of GP training practices, guide GP trainees towards the best training practices and detect and remediate GP trainers of lower quality.

RESULTS

TOEKAN (Tool for Communication and Evaluation of Quality Standards) was developed, which consists of a 72-item questionnaire for GP trainees and GP trainers and an 18-item questionnaire for those who coach and remediate GP trainers. The outcomes of the TOEKAN questionnaires are visualized in an online dashboard.

DISCUSSION

TOEKAN is the first 360° evaluation tool for CLE in GP education. All stakeholders will complete the survey on a regular basis and have access to the results. By creating intrinsic and extrinsic motivation as well as mediation measures, the quality of CLE will improve. Continuous monitoring of the use and outcomes of TOEKAN will allow to critically review and improve this new evaluation tool as well as support the broader implementation.

Intersectoral Development of an Evaluation Tool for the Socio-Professional Rehabilitation Process Adapted to People with Autism

Since there was no French tool adapted to the reality of people with autism enrolled in a work integration program in Quebec, we have modified the Profile of Rehabilitation and Adaptation in a Work Context for People with Intellectual Disabilities according to the scientific literature on autism. Content validity (n=17) and applicability (n=5) were done with success. Result is an Excel compiler containing 3 informative tabs, 10 tabs on the dimensions of work rehabilitation (59 items) and 4 tabs used to compile the results. Socio-professional practicians are welcomed to use this new tool to pursue construct and internal validity.

Psychometric Values of a New Scale: The Rett Syndrome Fear of Movement Scale (RSFMS)

(1) Background: One of the characteristics associated with Rett syndrome (RTT) is a fear of movement (FOM). Despite the grave consequences on health, function, and the caregiver's burden associated with bradykinesia accompanying FOM, there is no specific FOM assessment tool for RTT. (2) Objective: To construct and assess the psychometric values of a scale evaluating FOM in RTT (Rett syndrome fear of movement scale-RSFMS). (3) Methods: Twenty-five girls aged 5-33, including a research group (N = 12 individuals with RTT) and control group (N = 13 typically developing girls at equivalent ages). The Pain and Discomfort Scale (PADS) and Facial Action Coding System (FACS) assessed the participants' behavior and facial expressions in rest and movement situations. (4) Results: Significant behavioral differences were recorded in these rest and movement situations within the research groups using the RSFMS (p = 0.003), FACS (p = 0.002) and PADS (p = 0.002). No differences in reactions were found within the control group. The new scale, RSFMS, was found to show a high inter- and intra-rater reliability (r = 0.993, p < 0.001; r = 0.958, p < 0.001; respectively), good internal consistency (α = 0.77), and high accuracy (94.4%). (5) Conclusions: The new scale for measuring FOM in RTT, the RSFMS, was validated using the FACS and PADS. The RSFMS was found to be a tool that holds excellent psychometric values. The new scale can help clinicians working with individuals with RTT to plan appropriate management strategies for this population.

Evaluation Methodology for Inclusive Schools Environments. A Comparative Analysis Towards Goals and Strategies for Urban Design

The paper deals with the issue of social inclusion in the scholastic environment where children begin to learn the set of rules that manage social life. The design of the spaces impact on people and becomes crucial to address a community behavioural change in terms of social inclusion. In line with Universal Design strategy, the paper presents a method developed for evaluating the degree of inclusion of primary schools' spaces, which consists of the definition of an evaluation matrix. The matrix makes possible to analyse quantitative-qualitative characteristics for each school and compare them objectively. It asses the level of accessibility and inclusion of the schools through four main categories (Outdoor space, Orientation, Movement, Spatial quality) and related criteria and indicators. The reliability of the evaluation matrix has been verified through its application in seven case studies (Italian and EU), and the analysis of one of them is described in the results. The present study proposes a basis to introduce a method able to support designing educational spaces that satisfy the needs of a wide range of users according to Universal Design strategy.

CalTAD is the Key Evaluation Tool for Measurement of Cephalic Fixation Position for Predicting Cut-Out in Geriatric Intertrochanteric Fracture Patients with Internal Fixations after Achieving Acceptable Reduction

Objective

To discover the key evaluation tool of the cephalic fixation position for predicting implant failures in geriatric intertrochanteric fracture (ITF) patients treated with internal fixations after achieving an acceptable reduction.

Methods

We measured the geriatric ITF patients undergoing single-screw cephalomedullary nailing (CMN) fixation surgery after obtaining the acceptable reduction (including anatomical reduction and positive medial cortex support reduction) in our treatment group between September 2016 and March 2020 by using four kinds of cephalic fixation position evaluation tools including Cleveland zone system, Parker’s ratio index, tip–apex distance (TAD), calcar-referenced TAD (CalTAD), and analyzed which were the key evaluation tools for measurement of cephalic fixation position for prediction of implant failures in geriatric ITF patients with internal fixations.

Results

Seventy-four ITF patients treated with single-screw CMN fixation after obtaining the acceptable reduction were enrolled in this study. Of the 74 patients, nine cases were observed with implant failures. There were six cases of cut-out and three of pending cut-out. We found that TAD (odds ratio (OR)=1.149; 95% confidence interval (CI), 1.00–1.32; P=.046) and CalTAD (OR=1.140; 95% CI, 1.00–1.30; P=.037) were risk factors for implant failures by univariate analysis, while only CalTAD (OR=1.200; 95% CI, 1.032–1.395; P=.018) was the independent risk factor for implant failures by multivariate analysis. The Kappa coefficient (κ) of CalTAD was .976 (95% CI, .966–.984) by ICC analysis. The ROC analysis showed that the best cut-off value of CalTAD was 23.76 mm with a sensitivity of 77.8% and specificity of 72.3% (area under the curve, AUC =.775; P = .001).

Conclusions

CalTAD is the key evaluation tool for measurement of cephalic fixation position for predicting implant failures in geriatric ITF patients treated with single-screw CMN after obtaining the acceptable reduction.

Validation of a clinical competence evaluation tool for community service nurses in North West province, South Africa

Background

Little has been done to evaluate clinical competence of community service nurses (CSNs) during the 12-month compulsory community service in South Africa. Evaluating clinical competence of CSNs would be of benefit as it might improve quality patient care and promote patient satisfaction. It therefore became of paramount importance for the researcher to establish some method of evaluating the CSNs’ clinical competence during their compulsory service in the North West province (NWP), South Africa.

Aim

To evaluate the clinical competence evaluation tool (CCET) for CSNs for reliability and validity.

Setting

A selected regional level 2 hospital.

Methods

Ten experts participated in the validation process. The tool was tested at one of the public hospitals in the NWP and 11 out of 13 CSNs participated in this process. Statistical Package for the Social Sciences version 25 was employed and the reliability of the tool was measured using Cronbach’s alpha.

Results

This tool’s content validity index has exceeded 0.80 and is indicated at 0.98, which reflects excellent content validity. The higher the content validity ratio score the greater the agreement amongst the experts. The Cronbach’s alpha coefficients in the six competencies are all greater than 0.7 implying that the tool developed in this study is reliable. All the experts indicated that the tool is clear, simple, general, accessible and important.

Conclusion

From the above-mentioned results, a CCET for CSNs was proven to be valid and reliable.

Contribution

This was the first tool to be developed in NWP of South Africa.

Design and Initial Validation of a Humanistic Care Evaluation Tool

Objective

This study aimed at developing and validating a humanistic care tool in Anhui province that could be used across Chinese public hospitals, and to reflect the humanistic care from patients’ perspective.

Participants

A cross-sectional survey was conducted in three public hospitals of Anhui Province, China by adopting simple random sampling, which included 312 outpatients and 323 inpatients.

Methods

The dimensions of the tool were set according to “Further Improve Medical Service Action Plan” in China and Patient-Doctor Relationship Questionnaire. Cronbach’s alpha values were calculated and used to evaluate the reliability of this tool. Construct validity was tested by the exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). The associations between characteristics and humanistic care were analyzed by binary logistic regression.

Results

These initial findings showed that about two-thirds of the respondents experienced humanistic care. Both the reliability and construct validity of the humanistic care evaluation tool were suitable Social aspects (location and yearly income), treatment style and having a regular doctor were significantly associated with better humanistic care (all P<0.05).

Conclusion

The humanistic care tool can directly reflect the humanistic care from patients’ perspective, and can be popularized and applied across Chinese public hospitals. These findings have important implications to further improve medical service in Chinese public hospitals.

One Health Evaluation of Antimicrobial Use and Resistance Surveillance: A Novel Tool for Evaluating Integrated, One Health Antimicrobial Resistance and Antimicrobial Use Surveillance Programs

We describe the development, application and utility of our novel, One Health Evaluation of Antimicrobial Use and Resistance Surveillance (OHE-AMURS) tool that we created to evaluate progress toward integrated, One Health surveillance of antimicrobial resistance (AMR) and antimicrobial use (AMU) as a complex system in Canada. We conducted a qualitative inquiry into the current state of policy and programs for integrated AMR/AMU surveillance using explicit and tacit knowledge. To assess the "messy" state of public health surveillance program development, we synthesized recommendations from previous reports by the National Collaborating Centre for Infectious Diseases and the Canadian Council of Chief Veterinary Officers; conducted an environmental scan to find all federal, provincial, and territorial AMR/AMU surveillance programs in Canada; and conducted semi-structured interviews with Canadian subject matter experts. To integrate evidence from these different sources we adapted two published tools to create a new evaluation matrix, deriving 36 components of the ideal integrated AMR/AMU surveillance system. Our two-way matrix tool allowed us to examine seven common, foundational elements of sustainable programs for each component, and assign a stage of development/sustainability ranking for each component according to the matrix definitions. Our adaptable novel tool allowed for granular and repeatable assessment of the many components of a complex surveillance system. The assessment proved robust and exacting to ensure transparency in our methods and results. The matrix allows flexible assignment of program components based on program principles, and stages can be adapted to evaluate any aspect of an AMR/AMU surveillance or other multi-faceted, multi-jurisdictional system. Future refinement should include an assessment of the scope of surveillance components.

Correlation between the Korean Version of the Trunk Control Measurement Scale and the Selective Control Assessment of the Lower Extremity Scores in Children with Cerebral Palsy

Background and Objectives: The purpose of this study was to investigate the correlation between the Korean version of the trunk control measurement scale (K-TCMS) and the selective control assessment of the lower extremity (SCALE). Through this, we tried to find out the effect of proximal stabilization on distal motor development. Materials and Methods: Fifty-one children with gross motor function classification system level I–III, diagnosed with cerebral palsy (CP), were studied. The K-TCMS was used to evaluate the body control ability of the children. SCALE was used to quantify selective voluntary motor control (SVMC). Results: Analysis of SCALE and K-TCMS showed a significant positive correlation in all items. Multiple regression analysis showed that the SCALE score decreased as age increased, and that it increased as the static sitting balance ability score and the dynamic sitting balance ability score of the K-TCMS increased significantly (p < 0.05). Conclusions: In children with cerebral palsy, there was a close correlation between trunk control and selective voluntary motor control of the lower extremities. Therefore, when trying to improve the lower extremity function of a child with cerebral palsy, a trunk control intervention should be considered.

Usability testing of radiotherapy systems as a medical device evaluation tool to inform hospital procurement decision-making

PURPOSE

Poor usability designs of radiotherapy systems can contribute to use errors and adverse events. Therefore, we evaluated the usability of two radiotherapy systems through radiation therapists' performance, workload, and experience that can inform hospital procurement decision-making about the selection of appropriate radiotherapy system for radiation therapist use.

METHODS

We performed a comparative usability study for two radiotherapy systems through user testing. Thirty radiation therapists participated in our study, in which four typical operational tasks were performed in two tested radiotherapy systems. User performance was measured by task completion time and completion difficulty level. User workloads were measured by perceived and physiological workload using NASA-TLX questionnaires and eye motion data. User experience was measured by the USE questionnaire.

RESULTS

Significantly less task completion time and an easier task completion difficulty level were shown with the Varian Trilogy than with the XHA600E. The study results suggest that higher perceived and physiological workloads were experienced with the XHA600E than with the Varian Trilogy. Radiation therapists reported better user experience with the Varian Trilogy than with the XHA600E. Five paired t-tests regarding user performance, user workload, and user experience between the Varian Trilogy and the XHA600E were performed, showing that the Varian Trilogy radiotherapy system has a better usability design than the XHA600E radiotherapy system.

CONCLUSIONS

Based on study results, we confirmed that the Varian Trilogy radiotherapy system has a better usability design than the XHA600E radiotherapy system. Furthermore, the study results provide valuable evidence for hospital procurement decision-making regarding the selection of a suitable radiotherapy system for radiation therapists to use.

The Application of a System of Eye Tracking in Laparoscopic Surgery: A New Didactic Tool to Visual Instructions

Introduction: Laparoscopic surgery is an increasingly used technique, but it requires a high degree of learning, and communication between the operating room crew is considerably difficult. The use of eye tracking has been proposed as a didactic and evaluation tool in several settings, including in laparoscopy in simulators.

Objectives: This study aimed to evaluate the usefulness of the use of eye tracking systems (Tobii glasses 2) in laparoscopic surgery as a didactic and assessment tool to improve communication in the operating room and improve patients' security.

Methodology: An anonymous survey was sent to the students and medical teachers of a faculty of medicine and practicing doctors and residents. The message contained an explanation about the use of the Tobii glasses, a link to watch the video showing its use in a laparoscopic surgery, and the survey to complete after watching the video.

Results: The survey was answered by 113 participants (51.3% medical students, 27.4% medical teachers, 18.6% practicing doctors, and 2.7% medicine residents). Eighty-three percent agreed with the usefulness of the “Tobii glasses” in the operating room for improving communication between the main surgeon and the assistant, for learning complex surgery techniques, for obtaining didactic videos, and for indicating anatomical structures. The item scored worst was the price of the glasses.

Conclusions: It is possible to record and project expert gaze patterns in the operating room in real time using the Tobii glasses. This device allows improving communication among the surgical crew and the learning of residents and also improving the security of surgical patients.

Assessment of Good Practices in Community-Based Interventions for Physical Activity Promotion: Development of a User-Friendly Tool

Tools to identify good practices in the design, implementation, and evaluation of physical activity community-based interventions (PACIs) are key to address the physical inactivity pandemic. Existing tools tend to be extensive and with limited applicability to assess small-scale PACIs. This work aimed to report the development and preliminary validity results of a simple, practical, and user-friendly tool to evaluate PACIs in local/municipal contexts. Eighty-six good practice characteristics defined by the World Health Organization (WHO), the Joint Action Framework on Chronic Diseases (CHRODIS), and an umbrella review of good practice characteristics of diet and physical activity interventions were initially extracted and refined in four rounds of revision from an expert panel using a Delphi-type methodology and rated on their relative importance. A pilot application was conducted, and data on the tool usability and applicability were collected through three semi-structured interviews with specialists and coordinators of local/municipal PACIs. For preliminary validation, the refined tool was applied to five community-based programs mostly aimed at an elderly population. The final tool included thirty-four selected characteristics, with a brief explanation and practical examples for each, under three main sections: design, evaluation, and implementation. Each characteristic has a rating (i.e., somewhat important, highly important, mandatory) and a percentage weight. Preliminary validation of this tool pointed to an adequate evaluation of good practice characteristics of municipal PACIs in a reliable, practical, and user-friendly way. Given its adequacy, this tool can support the definition of quality standards for PACIs, encouraging their dissemination and adoption at a regional or national level.

Prevention of Nausea and Vomiting After Laparoscopic Sleeve Gastrectomy: Are We Doing Enough?

The worldwide obesity epidemic has led to an increase in the number of bariatric procedures for treatment. Currently, the most commonly performed bariatric surgical operation in the United States is the laparoscopic sleeve gastrectomy (LSG). The risk of postoperative nausea and vomiting (PONV) is exceptionally high following the LSG: as much as 65%. Anesthesia providers should be mindful of this elevated risk and stay up to date on evidence-based techniques that help to mitigate the incidence of PONV related to this specific procedure. The purposes of this clinical review are to increase provider awareness of the PONV risk following LSG and to examine methods for improving PONV outcomes for this population.

Test-retest reliability of the Italian version of the M-BACK questionnaire to assess the barriers, attitudes, confidence, and knowledge of mental health staff regarding metabolic health of psychiatric patients

OBJECTIVES

The Metabolic-Barriers, Attitudes, Confidence, and Knowledge Questionnaire (M-BACK) was developed to determine the barriers, attitudes, confidence, and knowledge of mental health practitioners regarding the metabolic health of patients in order to determine the efficacy of targeted training interventions. This study aimed to validate the Italian version of M-BACK questionnaire (M-BACK-IT) and to determine the test-retest reliability.

METHODS

The M-BACK questionnaire was translated into Italian and back-translated using an established protocol. In order to determine the test-retest reliability of the instrument, mental health professionals were recruited from a private psychiatric hospital located in northeast Italy and completed the questionnaire on two separate occasions, seven days apart. Intraclass correlation coefficients (ICC) were calculated for the total score, as well as each of the four M-BACK domains.

RESULTS

Thirty mental health professionals (4 psychiatrists, 9 psychologists, 12 nurses, and 5 exercise specialists) completed the M-BACK-IT. ICCs ranged from 0.58 to 0.94.

CONCLUSIONS

The test-retest reliability of the M-BACK-IT demonstrated comparable results to the English version. The M-BACK-IT is a reliable measure to assess key elements of practitioners' perceptions of the barriers, their knowledge, attitudes, and confidence regarding metabolic monitoring and intervention in people with mental illness.

Development of a Common Clinical Assessment Tool for Evaluation in Nurse Anesthesia Education

There is a lack of standardization among evaluations completed by clinical educators of student registered nurse anesthetists (SRNAs) during their clinical education as reported by nurse anesthesia program administrators and students. To address this issue, the Council on Accreditation of Nurse Anesthesia Educational Programs (COA) Board created the Common Clinical Assessment Tool (CCAT) Special Interest Group to develop a standardized clinical evaluation instrument. The goal was to improve the consistency of clinical evaluation across nurse anesthesia programs while assisting program administrators to make programmatic changes to ensure compliance with COA standards. In May 2016, the CCAT Special Interest Group began to create an evaluation instrument that was competency based and reflective of the COA's Practice Doctorate Standards. After a review of literature, input from the communities of interest, results from the American Association of Nurse Anesthetists professional practice survey, and analysis of the National Certification Examination for Nurse Anesthetists content outline and information from other sources, a draft CCAT was completed. A Delphi study was conducted, and expert opinions from program administrators, academic and clinical faculty, and students were collected to ascertain consensus on competencies, competency descriptors, and progression indicators.

Validation transculturelle de contenu de la version franco-canadienne de l’échelle COREQ

Purpose: The main purpose of this article is to produce a French-Canadian translation of the Consolidated criteria for reporting qualitative studies (COREQ) scale under the proposed name "échelle COREQ" and to assess the transcultural validity of its content. The secondary purpose is to examine the inter-rater reliability of the French-Canadian version of the COREQ scale. Method: A modified approach to Vallerand's methodology (1989) for cross-cultural validation was used. First, a parallel back-translation of the COREQ scale was performed, by both professionals and clinicians. Next, a first committee of experts(P1) examined the translations to create a first draft of the French-Canadian version of the COREQ scale. This draft was then evaluated and modified by a second committee of experts (P2). Finally, 28future professionals (master's students in physiotherapy) rated this second draft of the tool for clarity using a seven-point scale (1:very clear; 7:very ambiguous). The principal co-investigators then reviewed the problematic elements and proposed final changes. Two independent raters used this French-Canadian version of the COREQ scale to assess 13qualitative studies that were published in French after the year2007. The kappa coefficient was used to examine inter-rate reliability. Results: The different elements of the final version of the COREQ scale received an average ambiguity rating between 1.04 and 2.56. These low values show a high level of clarity for the French-Canadian version of the COREQ scale. In relation to the total score of the COREQ scale, inter-rater reliability (n = 2) is considered to be average to excellent for 62.5% of individual elements, according to the kappa values obtained. Conclusions: A valid French-Canadian version of the COREQ scale was created using this rigorous five-step process.

Implementation of Mock Acute Care Advance Pharmacy Practice Experience Simulations and an Assessment Rubric

Objective. To implement a mock acute care advanced pharmacy practice experience series into the didactic training of second-year pharmacy students and validate an accompanying assessment rubric. Methods. Three 90-minute acute care patient simulation laboratory sessions were developed with input from clinical specialists, preceptors, students, and faculty members. An accompanying student evaluation rubric was also developed. The assessment rubric was validated using pairs of preceptor raters to determine inter-rater reliability, along with predictive validity on advanced pharmacy practice experience (APPE) acute care scores. A student survey was also conducted. Results. The mock acute care APPEs were successfully implemented into the didactic curriculum. The assessment rubric had good inter-rater reliability and good predictive validity with acute care APPEs. Survey results indicated that students found the mock acute care APPE simulation laboratories useful. Conclusion. Other schools seeking to enhance their students' preparedness for and performance in acute care APPEs should consider implementing acute care APPE simulations in the didactic curriculum.

Dementia Friendly Communities in England: A scoping study

OBJECTIVES

To describe the characteristics of Dementia Friendly Communities (DFCs) across England in order to inform a national evaluation of their impact on the lives of those affected by dementia.

METHODS

DFCs in England were identified through online searches and Alzheimer's Society records. A subsample (n = 100) were purposively selected for in-depth study based on online searches and, where necessary, follow-up telephone calls. Data collection and analysis were guided by a pilot evaluation tool for DFCs that addressed how DFCs are organised and resourced and how their impact is assessed. The evidence was predominantly qualitative, in addition to some descriptive quantitative information.

RESULTS

Of 284 DFCs identified, 251 were defined by geographical location, while 33 were communities of interest. Among 100 sampled DFCs, 89 had been set up or started activities following policy endorsement of DFCs in 2012. In the resourcing of DFCs, statutory agencies and charities played an important role. Among DFC activities, awareness raising was cited most commonly. There was some evidence of involvement of people living with dementia in organisational and operational aspects of DFCs. Approaches to evaluation varied, with little evidence of findings having effected change.

CONCLUSIONS

DFCs are characterised by variation in type, resourcing, and activities. England has policy endorsement and a recognition system for DFCs. These can be important catalysts for initiation and growth. A systematic approach to evaluation is lacking. This would enable DFCs to be consistent in how they demonstrate progress and how they enable people living with dementia to live well.

La version franco-canadienne du « STrengthening the Reporting of OBservational studies in Epidemiology » (STROBE) Statement : L’outil STROBE

Purpose: The main purpose is to produce a French-Canadian translation of the "STrengthening the Reporting of OBservational studies in Epidemiology" (STROBE) Statement under the proposed name of "outil STROBE" and to assess the cross-cultural validity of its content. The secondary purpose is to examine its preliminary interrater reliability. Method: A modified approach to Vallerand's cross-cultural validation methodology was used. First, professional translators and clinical practitioners produced a parallel reverse translation of the "STROBE Statement." Then, a committee of experts (P1) examined the translated versions and created a first experimental draft of the "outil STROBE." This draft was assessed and modified by a second committee of experts (P2). Finally, 32 future professionals doing their master's degree in physiotherapy and occupational therapy assessed this second experimental version of the tool using an ambiguity scale of 8 points (0 meaning "not at all ambiguous" and 7 meaning "extremely ambiguous"). The main co-researchers examined the problematic elements and proposed final modifications. Ten observational studies published in French after 1980 were assessed by two independent raters using the French-Canadian version of the "outil STROBE." The kappa coefficient was used to examine interrater reliability. Results: For the different elements of the final version of the "outil STROBE," the averages on the ambiguity scale varied from 0.0 to 2.4. No element received an average below 2.4, which showed a high level of clarity. The interrater reliability (n = 2) for the "outil STROBE" is thought to be good for 74% of individual elements, according to the kappa coefficient values obtained. Conclusions: The process's five rigorous steps enabled the production of a valid French-Canadian version of the "STROBE Statement."

How We Should Assess the Delivery of End-Of-Life Care in General Practice? A Systematic Review

BACKGROUND

The majority of end-of-life (EOL) care occurs in general practice. However, we still have little knowledge about how this care is delivered or how it can be assessed and supported.

AIM

(i) To review the existing evaluation tools used for assessment of the delivery of EOL care from the perspective of general practice; (ii) To describe how EOL care is provided in general practice; (iii) To identify major areas of concern in providing EOL care in this context.

DESIGN

A systematic review.

DATA SOURCES

Systematic searches of major electronic databases (Medline, EMBASE, PsycINFO, and CINAHL) from inception to 2017 were used to identify evaluation tools focusing on organizational structures/systems and process of end-of-life care from a general practice perspective.

RESULTS

A total of 43 studies representing nine evaluation tools were included. A relatively restricted focus and lack of validation were common limitations. Key general practitioner (GP) activities assessed by the evaluation tools were summarized and the main issues in current GP EOL care practice were identified.

CONCLUSIONS

The review of evaluation tools revealed that GPs are highly involved in management of patients at the EOL, but there are a range of issues relating to the delivery of care. An EOL care registration system integrated with electronic health records could provide an optimal approach to address the concerns about recall bias and time demands in retrospective analyses. Such a system should ideally capture the core GP activities and any major issues in care provision on a case-by-case basis.

The Validity and Reliability Characteristics of the M-BACK Questionnaire to Assess the Barriers, Attitudes, Confidence, and Knowledge of Mental Health Staff Regarding Metabolic Health of Mental Health Service Users

Background

Addressing the burden of poor physical health and the subsequent gap in life expectancy experienced by people with mental illness is a major priority in mental health services. To equip mental health staff with the competence to deliver evidence-based interventions, targeted staff training regarding metabolic health is required. In order to evaluate the effectiveness of staff training regarding metabolic health, we aimed to develop a succinct measure to determine the barriers, attitudes, confidence, and knowledge of health practitioners through the development and test–retest reliability of the Metabolic-Barriers, Attitudes, Confidence, and Knowledge Questionnaire (M-BACK).

Methods

The M-BACK questionnaire was developed to evaluate the impact of specialized training in metabolic health care for mental health nurses. Content of the M-BACK was developed from a literature review and refined by an expert review panel and validated via a piloting process. To determine the test–retest reliability of the M-BACK, 31 nursing students recruited from the University of Notre Dame, Sydney completed the questionnaire on two separate occasions, 7 days apart. Intraclass correlation coefficients (ICCs) were calculated for the total score, as well as each of the four domains.

Results

Pilot testing was undertaken with a sample of 106 mental health nurses with a mean age 48.2, ranging from 24 to 63 years of age, who participated in six training courses. Questionnaire development resulted in a 16-item instrument, with each item is scored on a five-point Likert scale ranging from “strongly disagree” to “strongly agree.” Test–retest reliability of the M-BACK was completed by 30 of 31 nursing students recruited, ICCs ranged from 0.62 to 0.96.

Conclusion

The M-BACK is a reliable measure of the key elements of practitioner perceptions of barriers, and their knowledge, attitudes, and confidence regarding metabolic monitoring in people with mental illness. It can be used to assess the effectiveness of interventions aimed at increasing uptake of metabolic monitoring, a key component of programs to reduce the life expectancy gap in people living with severe mental illness.

Objective Assessment of Technical Skills in Otorhinolaryngology-Head and Neck Surgery Residents: A Systematic Review

Objectives The primary goal is the indexation of validated methods used to assess surgical competency in otorhinolaryngology-head and neck surgery (ORL-HNS) residents. Secondary goals include assessment of the reliability and validity of these tools, as well as the documentation of specific procedures in ORL-HNS involved. Data Sources MEDBASE, OVID, Medline, CINAHL, and EBM, as well as the printed references, available through the Université de Montréal library. Review Methods The PRISMA method was used to review digital and printed databases. Publications were reviewed by 2 independent reviewers, and selected articles were fully analyzed to classify evaluation methods and categorize them by procedure and subspecialty of ORL-HNS involved. Reliability and validity were assessed and scored for each assessment tool. Results Through the review of 30 studies, 5 evaluation methods were described and validated to assess the surgical competency of ORL-HNS residents. The evaluation method most often described was the combined Global Rating Scale and Task-Specific Checklist tool. Reliability and validity for this tool were overall high; however, considerable data were unavailable. Eleven distinctive surgical procedures were studied, encompassing many subspecialties of ORL-HNS: facial plastics, general ear-nose-throat, laryngology, otology, pediatrics, and rhinology. Conclusions Although assessment tools have been developed for an array of surgical procedures, involving most ORL-HNS subspecialties, the use of combined checklists has been repeatedly validated in the literature and shown to be easily applicable in practice. It has been applied to many ORL-HNS procedures but not in oncologic surgery to date.

Developing an Assessment Tool for Enhancing Interprofessional Education of Patient Safety

The increasing recognition of the value of effective interprofessional collaboration has promoted interprofessional education (IPE) among healthcare professionals. The IPE pilot program initiative at UTHealth matriculated students from the disciplines of biomedical informatics, biomedical sciences, dentistry, medicine, nursing and public health. There has been a pressing need for developing an assessment tool in reflecting how IPE participants recognize and understand the other disciplines including clinical informatics in nowadays' practice. This paper reports our development process of the assessment tool, which is still under an iterative, in-depth refinement and aiming at greater collaborations.

How Effective are New Milestones Assessments at Demonstrating Resident Growth? 1 Year of Data

OBJECTIVE

Assessment tools that accrue data for the Accreditation Council for Graduate Medical Education Milestones must evaluate residents across multiple dimensions, including medical knowledge, procedural skills, teaching, and professionalism. Our objectives were to: (1) develop an assessment tool to evaluate resident performance in accordance with the Milestones and (2) review trends in resident achievements during the inaugural year of Milestone implementation.

DESIGN

A novel venue and postgraduate year (PGY) specific assessment tool was built, tested, and implemented for both operating room and labor and delivery "venues." Resident development of competence and independence was captured over time. To account for variable rotation schedules, the year was divided into thirds and compared using two-tailed Fisher's exact test.

SETTING

Brigham and Women's and Massachusetts General Hospitals, Boston MA.

PARTICIPANTS

Faculty evaluators and obstetrics and gynecology residents.

RESULTS

A total of 822 assessments of 44 residents were completed between 9/2014 and 6/2015. The percentage of labor and delivery tasks completed "independently" increased monotonically across the start of all years: 8.4% for PGY-1, 60.3% for PGY-2, 73.7% for PGY-3, and 87.5% for PGY-4. Assessments of PGY-1 residents demonstrated a significant shift toward "with minimal supervision" and "independent" for the management of normal labor (p = 0.03). PGY-3 residents demonstrated an increase in "able to be primary surgeon" in the operating room, from 36% of the time in the first 2/3 of the year, to 62.3% in the last 1/3 (p < 0.01).

CONCLUSION

Assessment tools developed to assist with Milestone assignments capture the growth of residents over time and demonstrate quantifiable differences in achievements between PGY classes. These tools will allow for targeted teaching opportunities for both individual residents and residency programs.

Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool

BACKGROUND

Medical Image segmentation is an important image processing step. Comparing images to evaluate the quality of segmentation is an essential part of measuring progress in this research area. Some of the challenges in evaluating medical segmentation are: metric selection, the use in the literature of multiple definitions for certain metrics, inefficiency of the metric calculation implementations leading to difficulties with large volumes, and lack of support for fuzzy segmentation by existing metrics.

RESULT

First we present an overview of 20 evaluation metrics selected based on a comprehensive literature review. For fuzzy segmentation, which shows the level of membership of each voxel to multiple classes, fuzzy definitions of all metrics are provided. We present a discussion about metric properties to provide a guide for selecting evaluation metrics. Finally, we propose an efficient evaluation tool implementing the 20 selected metrics. The tool is optimized to perform efficiently in terms of speed and required memory, also if the image size is extremely large as in the case of whole body MRI or CT volume segmentation. An implementation of this tool is available as an open source project.

CONCLUSION

We propose an efficient evaluation tool for 3D medical image segmentation using 20 evaluation metrics and provide guidelines for selecting a subset of these metrics that is suitable for the data and the segmentation task.

[Not Available]

PURPOSE

To prepare a Canadian French translation of the PEDro Scale under the proposed name l'Échelle PEDro, and to examine the validity of its content.

METHODS

A modified approach of Vallerand's cross-cultural validation methodology was used, beginning with a parallel back-translation of the PEDro scale by both professional translators and clinical researchers. These versions were reviewed by an initial panel of experts (P1), who then created the first experimental version of l'Échelle PEDro. This version was evaluated by a second panel of experts (P2). Finally, 32 clinical researchers evaluated the second experimental version of l'Échelle PEDro, using a 5-point clarity scale, and suggested final modifications.

RESULTS

The various items on the final version of l'Échelle PEDro show a high degree of clarity (from 4.0 to 4.7 on the 5-point scale).

CONCLUSION

The four rigorous steps of the translation process have produced a valid Canadian French version of the PEDro scale.

Development and Evaluation of an Evaluation Tool for Healthcare Smartphone Applications

INTRODUCTION

Various types of healthcare smartphone applications (apps) have been released in recent years, making it possible for people to manage their health anytime and anywhere. As a healthcare provider, who has the responsibility to provide guidance as to which apps can be used? The purpose of this study was to develop and evaluate an evaluation tool for the various aspects of healthcare smartphone apps.

MATERIALS AND METHODS

In the first phase, a provisional version of an evaluation tool for healthcare smartphone apps was developed from a review of previous studies. In the second phase, the provisional tool was modified and edited after verification by five experts with regard to its content validity. In the third phase, from September 25 to October 4, 2013, 200 responses were collected to verify the construct validity and reliability of the tool.

RESULTS

The edited tool had 23 evaluating items with three evaluating factors along with seven subevaluating factors as a result of confirmatory factor analysis. The reliability was found to be high (0.905).

CONCLUSIONS

This study is meaningful because it demonstrates a healthcare smartphone app evaluation tool that is proven in terms of its validity and reliability. The evaluation tool developed and tested in this study is an appropriate and widely applicable tool with which to evaluate healthcare smartphone apps to determine if they are reliable and useful. However, this evaluation tool represents the beginning of the research in this area.

Iranian Effective Clinical Nurse Instructor evaluation tool: Development and psychometric testing

BACKGROUND

Clinical education is the heart of the nursing education program. Effective nursing clinical instructors are needed for graduating the future qualified nurses. There is a well-developed body of knowledge about the effectiveness of clinical teaching and the instructors. However, translating this knowledge into a context-based evaluation tool for measuring the effectiveness of Iranian clinical nursing instructors remains a deficiency. The purpose of this study is to describe the development and psychometric testing process of an instrument to evaluate the characteristics of Iranian effective clinical nurse instructor.

MATERIALS AND METHODS

Following a precise review of Iranian literatures and expert consultation, 83 statements about the characteristics that make clinical nurse instructors effective were extracted. In the next phase, the psychometric properties of the instrument were established by looking at the content validity, face validity, and internal consistency. Content validity of the instrument was assessed based on the comments of an expert panel including 10 nursing faculty members. During this phase, 30 items of the instrument were omitted or merged. Face validity of the instrument was assured based on the advices of 10 nursing students and 10 nursing faculty members. Finally, in the pilot test, the data of 168 filled questionnaires were gathered and analyzed by an exploratory factor analysis to reduce the items and identify the factor structure of the instrument.

RESULTS

Through subsequent analyses, of the 83 items, 31 items were merged or omitted. At last, 52 retained items were divided into four subscales including student-centric behaviors, clinical performances, planning ability, and personality traits. The Cronbach's alpha level of the inventory was 0.96, with the value for each domain ranging from 0.87 to 0.94.

CONCLUSIONS

Iranian Effective Clinical Nurse Instructor evaluation tool has acceptable psychometric properties and can be used in evaluating the effectiveness of clinical nursing instructors.

A tool to assess student performance in a Clostridium difficile infection simulation scenario

OBJECTIVE

To develop and validate an evaluation tool to assess student pharmacists' performance in a simulation scenario involving a patient with Clostridium difficile infection (CDI).

METHODS

The authors used an expert panel review process to establish content validity of the tool. Four faculty members used the tool to evaluate student pharmacist groups during 2011 and tested a modified version of the tool in 2012. The authors analyzed the results for each year to determine internal consistency and inter-rater reliability.

RESULTS

The 2011 tool demonstrated sound internal consistency, but several items had poor inter-rater agreement. The revised 2012 tool demonstrated acceptable internal consistency and good to excellent inter-rater agreement for all items except one.

CONCLUSIONS

The tool facilitated reliable assessment of student pharmacists' clinical decision-making during simulation performance involving a patient with CDI.

The Placement History Chart: A Tool for Understanding the Longitudinal Pattern of Foster Children's Placements

Despite growing concerns about foster placement instability, little information is available regarding the longitudinal patterns of placement histories among foster children. The purpose of the present study was to develop a charting system using child welfare records to facilitate a better understanding of longitudinal patterns of placement history for 117 foster children. The resulting Placement History Chart included all placements that occurred during the observed time period and accounted for various dimensions: number, length, type, and sequence of placements; timing of transitions; and total time in out-of-home care. The Placement History Chart is an effective tool for placing foster care experiences within a broader developmental context. As such, the Placement History Chart can be a valuable research tool for understanding various dimensions and variations of placement transitions among foster children by capturing sequences and cumulative risks over time. Furthermore, this chart can facilitate the development of intervention programs that are developmentally sensitive and effectively address particularly vulnerable subpopulations of foster children.

A tool for evaluating the potential for cost-effective outcomes measurement

Cost related to higher-level outcomes measurement is often very high. However, the cost burden is felt even more by smaller, less well-funded continuing medical education (CME) programs. It is possible to overcome financial and participant-related barriers to measuring Level 6 outcomes, which are patient health outcomes. The Temple University School of Medicine's Office for Continuing Medical Education developed a sequential tool for attaining cost-effective outcomes measurement for determining the likelihood of a CME intervention to produce significant changes in physician performance. The appropriate selection of the CME topic and specific practice change indictors drive this tool. This tool walks providers through a simple YES or NO decision-making list that guides them toward an accurate prediction of potential programmatic outcomes. Factors considered during the decision-making process include whether: (a) the intended change(s) will have a substantial impact on current practice; (b) the intended practice change(s) are well supported by clinical data, specialty organization/government recommendations, expert opinion, etc; (c) the potential change(s) affects a large population; (d) external factors, such as system pressures, media pressures, financial pressures, patient pressures, safety pressures, etc, are driving this intended change in performance; (e) there is a strong motivation on the part of physicians to implement the intended change(s); and (f) the intended change(s) is relatively easy to implement within any system of practice. If each of these questions can be responded to positively, there is a higher likelihood that the intended practice-related change(s) will occur. Such change can be measured using a simpler and less costly methodology.

[Not Available]

INTRODUCTION

The Lower Extremity Functional Scale was translated into Canadian French in 2006; the translated version was titled Échelle fonctionnelle des membres inférieurs (EFMI). The construct validity and reliability of the EFMI have not been examined.

OBJECTIVE

To examine the test-retest reliability, internal consistency, and construct validity of the EFMI.

METHODOLOGY

Thirty-four participants who had undergone surgery or had an injury to the lower limb were asked to fill out a demographic questionnaire and complete the EFMI after their initial assessment while hospitalized. They completed the EFMI a second time within 72 hours. The intra-class correlation coefficient (ICC) and the kappa coefficient were chosen to examine the test-retest reliability of the EFMI, and Cronbach's alpha coefficient was calculated to assess internal consistency. A factor analysis was conducted to examine construct validity by determining the number of constructs and their meanings.

RESULTS

The ICC value was 0.92 (95% CI: 0.88-0.96), and Cronbach's alpha was 0.95 (95% CI: 0.91-0.99). The factor analysis revealed that all elements of the EFMI share one principal construct.

CONCLUSION

The results demonstrate that the French-Canadian version of the EFMI is a unidimensional tool with excellent test-retest reliability.

[Not Available]

INTRODUCTION

The Rehabilitation Network of Ottawa-Carleton (RENOC) recommends that its institutions use the Lower Extremity Functional Scale (LEFS). Since RENOC serves both Francophone and Anglophone clients, the LEFS needed to be translated to assess the Francophone clientele.

OBJECTIVE

To create a French-Canadian version of the LEFS, with the proposed title "Échelle fonctionnelle des membres inférieurs" (ÉFMI), and to examine its content validity.

METHODOLOGY

A modified version of Vallerand's (1989) cross-validation technique for questionnaires was used to validate the equivalence of the French version of the LEFS. First, a parallel reverse translation of the LEFS was completed. Second, a committee of experts examined the two drafts and created a first experimental version of the ÉFMI. Third, this version was evaluated by a second committee of experts. Following the comments received, a second experimental version was created. Fourth, 35 rehabilitation professionals evaluated the second experimental version of the ÉFMI on a scale of ambiguity and proposed final changes.

RESULTS

On the final version of the ÉFMI, the statements demonstrated a relatively low average level of ambiguity, ranging between 1 and 2.54.

CONCLUSION

The rigorous four-step process followed in this study produced a French-Canadian version of the LEFS with demonstrated content validity.

Identifying key components for an effective case report poster: an observational study

BACKGROUND

Residents demonstrate scholarly activity by presenting posters at academic meetings. Although recommendations from national organizations are available, evidence identifying which components are most important is not.

OBJECTIVE

To develop and test an evaluation tool to measure the quality of case report posters and identify the specific components most in need of improvement.

DESIGN

Faculty evaluators reviewed case report posters and provided on-site feedback to presenters at poster sessions of four annual academic general internal medicine meetings. A newly developed ten-item evaluation form measured poster quality for specific components of content, discussion, and format (5-point Likert scale, 1 = lowest, 5 = highest).

MAIN OUTCOME MEASURE(S)

Evaluation tool performance, including Cronbach alpha and inter-rater reliability, overall poster scores, differences across meetings and evaluators and specific components of the posters most in need of improvement.

RESULTS

Forty-five evaluators from 20 medical institutions reviewed 347 posters. Cronbach's alpha of the evaluation form was 0.84 and inter-rater reliability, Spearman's rho 0.49 (p < 0.001). The median score was 4.1 (Q1 -Q3, 3.7-4.6)(Q1 = 25th, Q3 = 75th percentile). The national meeting median score was higher than the regional meetings (4.4 vs, 4.0, P < 0.001). We found no difference in faculty scores. The following areas were identified as most needing improvement: clearly state learning objectives, tie conclusions to learning objectives, and use appropriate amount of words.

CONCLUSIONS

Our evaluation tool provides empirical data to guide trainees as they prepare posters for presentation which may improve poster quality and enhance their scholarly productivity.