The development of a theory-based instrument to evaluate the effectiveness of continuing medical education

From Two Dimensions to Multidimensions: A Mechanistic Model to Support Deliberate CPD Development, Coordination, and Evaluation

INTRODUCTION

The effectiveness of continuing professional development as an intervention to improve health professional behavior and patient health is variable and contentious. To clarify the causal relationships underlying program outcomes and facilitate a necessary shift from outcomes-only-based approaches to outcome-based and theory-based approaches in program development and evaluation, we developed a model of mechanisms mapped to relevant outcomes.

METHODS

Mechanisms identified in a prior realist synthesis of opioid agonist therapy continuing professional development programs were iteratively tested and refined using purposive and opportunistic sampling and realist approaches against two systematic reviews of programs in analgesic prescribing and palliative care. Further testing involved practical application within programs in sustainable health care and pain management.

RESULTS

Ninety reports on 75 programs and practical application to multiple additional programs informed the final model consisting of five distinct mechanisms: motivation transformation, expert influence, confidence development, self-efficacy facilitation, and community of practice expansion. The mechanisms and related analysis emphasize that continuing professional development is heterogeneous, complex, and context dependent.

DISCUSSION

Shifting toward outcome-based and theory-based approaches facilitates further conceptual shifts at intraprogram and interprogram and interintervention levels toward more deliberate program development and evaluation, increased program complementarity and subsequent collaboration. It clarifies opportunities for intercalation of continuing professional development with other intervention sciences. The model presents a resource for practitioners, researchers, and policymakers to advance continuing professional development planning, coordination, and evaluation.

Development of the clinical reasoning competency scale for nurses

BACKGROUND

Clinical reasoning is emphasized as an important component of nursing education, since nurses' lack of clinical reasoning leads to incorrect clinical decision-making. Therefore, a tool for measuring clinical reasoning competency needs to be developed.

METHODS

This methodological study was conducted to develop the Clinical Reasoning Competency Scale (CRCS) and examine its psychometric properties. The attributes and preliminary items of the CRCS were developed based on a systematic literature review and in-depth interviews. The validity and reliability of the scale were evaluated among nurses.

RESULTS

The exploratory factor analysis was conducted for the construct validation. The total explained variance of the CRCS was 52.62%. The CRCS consists of 8 items for plan setting, 11 items for intervention strategy regulation, and 3 items for self-instruction. The Cronbach's α of the CRCS was 0.92. Criterion validity was verified with the Nurse Clinical Reasoning Competence (NCRC). The correlation between the total NCRC and CRCS scores was 0.78, all of which were significant correlations.

CONCLUSION

The CRCS is expected to provide raw scientific and empirical data for various intervention programs to develop and improve nurses' clinical reasoning competency.

Medical undergraduates' self-evaluation: before and after curriculum reform

BACKGROUND

In 2013, Taiwan launched a curriculum reform-the 7-year undergraduate medical education program was shortened to 6 years. This study explored the evaluation results from students regarding the curriculum reform and investigated graduates' perceptions regarding the curriculum organization of the two academic training programs affected by this curricular reform.

METHODS

A cross-sectional survey was conducted from May 14 to June 12, 2019. The 315 graduates from both the 7-year and 6-year curriculum programs in the same medical school in Taipei were invited to participate in this study. In total, 197 completed questionnaires were received, representing a response rate of 62.5%. The results of the principal component analysis confirmed the validity of the constructs employed in this self-administered questionnaire.

RESULTS

The t-test results yielded two main findings. First, the graduates from the 6-year program had significantly lower scores for preparedness for the upcoming postgraduate-year residency training than did their 7-year program counterparts. Additionally, the male graduates had significantly higher scores in terms of perceptions regarding curriculum organization and preparedness for postgraduate-year residency training than the female graduates. The results of stepwise regression also indicated that the sex difference was significantly correlated with graduates' readiness for their postgraduate-year residency training.

CONCLUSION

To avoid sex disparities in career development, a further investigation of female medical students' learning environment and conditions is necessary. In addition to the cross-sectional study of students' perceptions, further repeated measurements of the objective academic or clinical performance of graduates in clinical settings are desirable.

Supervisors' intention to observe clinical task performance: an exploratory study using the theory of planned behaviour during postgraduate medical training

BACKGROUND

Direct observation of clinical task performance plays a pivotal role in competency-based medical education. Although formal guidelines require supervisors to engage in direct observations, research demonstrates that trainees are infrequently observed. Supervisors may not only experience practical and socio-cultural barriers to direct observations in healthcare settings, they may also question usefulness or have low perceived self-efficacy in performing direct observations. A better understanding of how these multiple factors interact to influence supervisors' intention to perform direct observations may help us to more effectively implement the aforementioned guidelines and increase the frequency of direct observations.

METHODS

We conducted an exploratory quantitative study, using the Theory of Planned Behaviour (TPB) as our theoretical framework. In applying the TPB, we transfer a psychological theory to medical education to get insight in the influence of cognitive and emotional processes on intentions to use direct observations in workplace based learning and assessment. We developed an instrument to investigate supervisors intention to perform direct observations. The relationships between the TPB measures of our questionnaire were explored by computing bivariate correlations using Pearson's R tests. Hierarchical regression analysis was performed in order to assess the impact of the respective TPB measures as predictors on the intention to perform direct observations.

RESULTS

In our study 82 GP supervisors completed our TPB questionnaire. We found that supervisors had a positive attitude towards direct observations. Our TPB model explained 45% of the variance in supervisors' intentions to perform them. Normative beliefs and past behaviour were significant determinants of this intention.

CONCLUSION

Our study suggests that supervisors use their past experiences to form intentions to perform direct observations in a careful, thoughtful manner and, in doing so, also take the preferences of the learner and other stakeholders potentially engaged in direct observations into consideration. These findings have potential implications for research into work-based assessments and the development of training interventions to foster a shared mental model on the use of direct observations.

Validation of a Teaching Effectiveness Assessment in Psychiatry Continuing Medical Education

OBJECTIVE

Little is known about factors associated with effective continuing medical education (CME) in psychiatry. The authors aimed to validate a method to assess psychiatry CME teaching effectiveness and to determine associations between teaching effectiveness scores and characteristics of presentations, presenters, and participants.

METHODS

This cross-sectional study was conducted at the Mayo Clinic Psychiatry Clinical Reviews and Psychiatry in Medical Settings. Presentations were evaluated using an eight-item CME teaching effectiveness instrument, its content based on previously published instruments. Factor analysis, internal consistency and interrater reliabilities, and temporal stability reliability were calculated. Associations were determined between teaching effectiveness scores and characteristics of presentations, presenters, and participants.

RESULTS

In total, 364 participants returned 246 completed surveys (response rate, 67.6%). Factor analysis revealed a unidimensional model of psychiatry CME teaching effectiveness. Cronbach α for the instrument was excellent at 0.94. Item mean score (SD) ranged from 4.33 (0.92) to 4.71 (0.59) on a 5-point scale. Overall interrater reliability was 0.84 (95% CI, 0.75-0.91), and temporal stability was 0.89 (95% CI, 0.77-0.97). No associations were found between teaching effectiveness scores and characteristics of presentations, presenters, and participants.

CONCLUSIONS

This study provides a new, validated measure of CME teaching effectiveness that could be used to improve psychiatry CME. In contrast to prior research in other medical specialties, CME teaching effectiveness scores were not associated with use of case-based or interactive presentations. This outcome suggests the need for distinctive considerations regarding psychiatry CME; a singular approach to CME teaching may not apply to all medical specialties.

Readiness to change for interprofessional collaboration in healthcare: Development and validation of a theory-based instrument

This paper presents a study that aimed to develop and validate a theory-based instrument for the assessment of readiness to change for interprofessional collaboration in healthcare (IPC-TTM). The instrument was developed in the Persian language and tested in the Iranian context. Healthcare professionals from medical and nursing professions participated in the assessment of validity and reliability of the instrument. We conducted this psychometric study in two phases: First, the questionnaire was developed based on the transtheoretical model (TTM) through literature review and expert panel. Then, in the validation phase, we held three modified Delphi rounds to assess the content and face validity of the questionnaire. We used confirmatory factor analysis (CFA) to evaluate the fit of the questionnaire as applied to modified TTM. Reliability of the final instrument was tested by assessing the test-retest reliability of instrument items with Kappa coefficient. We also calculated the intraclass correlation coefficient (ICC) and Cronbach's alpha to assess the test-retest reliability and internal consistency of the instrument sub-scales. The initial item pool consisted of 30 items and three sub-scales (Attitude, Intention, and Action). The content validity of the questionnaire was confirmed with 17 items. Based on the CFA results two additional items were deleted to increase the fit of the model. The final instrument was confirmed with 15 items and three sub-scales. Reliability assessment on the 15-item instrument showed an acceptable test-retest reliability of the instrument items. ICC values for the Attitude, Intention, and Action sub-scales of the instrument were calculated as 0.82, 0.73, and 0.71, respectively. Moreover, Cronbach's alpha for the Attitude, Intention, and Action sub-scales were 0.85, 0.73, and 0.77, respectively. This study offers a new theory-based instrument to measure readiness to change for interprofessional collaboration in healthcare in the Iranian context. The questionnaire can be used for 'needs assessment' in developing tailored educational interventions and self-assessments in interprofessional education studies.

Impractical CME programs: Influential parameters in Iran

Background: Traditional approaches in Continuing Medical Education (CME) appear to be ineffective in any improvement of the patients’ care, reducing the medical errors, and/or altering physicians' behaviors. However, they are still executed by the CME providers, and are popular among the majority of the physicians. In this study, we aimed to explore the parameters involved in the degree of effectiveness of CME program in Iran.

Methods: In this study, 31 participants, consisting of general practitioners, CME experts and providers were recruited to participate in in-depth interviews and field observations concerning experiences with CME. Application was made of the qualitative paradigm along with the qualitative content analysis, using grounded theory data analysis methodology (constant comparative analysis).

Results: Based on the participants’ experiences, the insufficient consistency between the training program contents and the demands of GPs, in addition to the non-beneficiary programs for the physicians and the non-comprehensive educational designs, created a negative attitude to the continuing education among physicians. This could be defined by an unrealistic continuing education program, which is the main theme here.

Conclusion: Impracticable continuing education has created a negative attitude toward the CME programs among physicians so much that they consider these programs less important, resulting in attending the said programs without any specific aim: they dodge absenteeism just to get the credit points. Evidently, promoting CME programs to improve the performance of the physicians requires factual needs assessment over and above adaptation of the contents to the physicians’ performance.

Development of the Computerized Model of Performance-Based Measurement System to Measure Nurses' Clinical Competence

Critical thinking skills and clinical competence are for providing quality patient care. The purpose of this study is to develop the Computerized Model of Performance-Based Measurement system based on the Clinical Reasoning Model. The system can evaluate and identify learning needs for clinical competency and be used as a learning tool to increase clinical competency by using computers. The system includes 10 high-risk, high-volume clinical case scenarios coupled with questions testing clinical reasoning, interpersonal, and technical skills. Questions were sequenced to reflect patients' changing condition and arranged by following the process of collecting and managing information, diagnosing and differentiating urgency of problems, and solving problems. The content validity and known-groups validity was established. The Kuder-Richardson Formula 20 was 0.90 and test-retest reliability was supported (r = 0.78). Nursing educators can use the system to understand students' needs for achieving clinical competence, and therefore, educational plans can be made to better prepare students and facilitate their smooth transition to a future clinical environment. Clinical nurses can use the system to evaluate their performance-based abilities and weakness in clinical reasoning. Appropriate training programs can be designed and implemented to practically promote nurses' clinical competence and quality of patient care.

Applicability of the theory of planned behavior in explaining the general practitioners eLearning use in continuing medical education

BACKGROUND

General practitioners (GP) update their knowledge and skills by participating in continuing medical education (CME) programs either in a traditional or an e-Learning format. GPs' beliefs about electronic format of CME have been studied but without an explicit theoretical framework which makes the findings difficult to interpret. In other health disciplines, researchers used theory of planned behavior (TPB) to predict user's behavior.

METHODS

In this study, an instrument was developed to investigate GPs' intention to use e-Learning in CME based on TPB. The goodness of fit of TPB was measured using confirmatory factor analysis and the relationship between latent variables was assessed using structural equation modeling.

RESULTS

A total of 148 GPs participated in the study. Most of the items in the questionnaire related well to the TPB theoretical constructs, and the model had good fitness. The perceived behavioral control and attitudinal constructs were included, and the subjective norms construct was excluded from the structural model. The developed questionnaire could explain 66 % of the GPs' intention variance.

CONCLUSIONS

The TPB could be used as a model to construct instruments that investigate GPs' intention to participate in e-Learning programs in CME. The findings from the study will encourage CME managers and researchers to explore the developed instrument as a mean to explain and improve the GPs' intentions to use eLearning in CME.

Evaluation and Exploration of Favorable QTL Alleles for Salt Stress Related Traits in Cotton Cultivars (G. hirsutum L.)

Soil salinization is one of the major problems in global agricultural production. Cotton is a pioneer crop with regard to salt stress tolerance, and can be used for saline-alkali land improvement. The large-scale detection of salt tolerance traits in cotton accessions, and the identification of elite quantitative trait loci (QTLs)/genes for salt-tolerance have been very important in salt tolerance breeding. Here, 43 advanced salt-tolerant and 31 highly salt-sensitive cultivars were detected by analyzing ten salt tolerance related traits in 304 upland cotton cultivars. Among them, 11 advanced salt-tolerance and eight highly salt-sensitive cultivars were consistent with previously reported results. Association analysis of ten salt-tolerance related traits and 145 SSRs was performed, and a total of 95 significant associations were detected; 17, 41, and 37 of which were associated with germinative index, seedling stage physiological index, and four seedling stage biochemical indexes, respectively. Of these associations, 20 SSR loci were simultaneously associated with two or more traits. Furthermore, we detected 117 elite alleles associated with salt-tolerance traits, 4 of which were reported previously. Among these loci, 44 (37.60%) were rare alleles with a frequency of less than 5%, 6 only existed in advanced salt-tolerant cultivars, and 2 only in highly salt-sensitive cultivars. As a result, 13 advanced salt-tolerant cultivars were selected to assemble the optimal cross combinations by computer simulation for the development of salt-tolerant accessions. This study lays solid foundations for further improvements in cotton salt-tolerance by referencing elite germplasms, alleles associated with salt-tolerance traits, and optimal crosses.

A Theory-Based Study of Factors Explaining General Practitioners' Intention to Use and Participation in Electronic Continuing Medical Education

INTRODUCTION

Electronic modes of continuing medical education (eCME) can provide an appropriate and scalable way of updating the knowledge and skills of general practitioners (GPs). To optimize the adoption of eCME and develop efficient and cost-effective eCME programs, factors explaining GPs' intention to use eCME must first be elucidated.

METHODS

Using the Theory of Planned Behavior as a framework, we developed a questionnaire and administered it to GPs in seven CME seminars in Isfahan, Iran, in 2014. Three domains of GPs' intention to use eCME were measured: attitudes, perceived behavioral control, and subjective norms. We used linear and logistic regression to identify the main predictors of intention and behavior.

RESULTS

GPs who had high score in perceived behavioral control and a more positive attitude toward e-learning had a higher intention to adopt it for CME. In contrast, subjective norms (eg, social pressures to use eCME) were not a predictor. Attitude toward usefulness of eCME was the main predictor of being an actual eCME user.

DISCUSSION

Perceived behavioral control and attitude constitute the main predictors of the intention to use eCME. Establishing discussions forums and strengthening organizational support for eCME through an increased awareness among clinical superiors and CME managers would be expected to increase GPs' intention to use eCME.

The development and psychometric testing of a theory-based instrument to evaluate nurses' perception of clinical reasoning competence

AIM

The purpose of the study was to develop and psychometrically test the Nurses Clinical Reasoning Scale.

BACKGROUND

Clinical reasoning is an essential skill for providing safe and quality patient care. Identifying pre-graduates' and nurses' needs and designing training courses to improve their clinical reasoning competence becomes a critical task. However, there is no instrument focusing on clinical reasoning in the nursing profession.

DESIGN

Cross-sectional design was used. This study included the development of the scale, a pilot study that preliminary tested the readability and reliability of the developed scale and a main study that implemented and tested the psychometric properties of the developed scale.

METHOD

The Nurses Clinical Reasoning Scale was developed based on the Clinical Reasoning Model. The scale includes 15 items using a Likert five-point scale. Data were collected from 2013-2014. Two hundred and fifty-one participants comprising clinical nurses and nursing pre-graduates completed and returned the questionnaires in the main study. The instrument was tested for internal consistency and test-retest reliability. Its validity was tested with content, construct and known-groups validity.

RESULTS

One factor emerged from the factor analysis. The known-groups validity was confirmed. The Cronbach's alpha for the entire instrument was 0·9.

CONCLUSION

The reliability and validity of the Nurses Clinical Reasoning Scale were supported. The scale is a useful tool and can be easily administered for the self-assessment of clinical reasoning competence of clinical nurses and future baccalaureate nursing graduates. Study limitations and further recommendations are discussed.

Sensibility of a new instrument to assess clinical reasoning in post-graduate orthopaedic manual physical therapy education

Sound application of clinical reasoning (CR) by the physical therapist is critical to achieving optimal patient outcomes. As such, it is important for institutions granting certification in orthopaedic manual physical therapy (OMPT) to ensure that the assessment of CR is sufficiently robust. At present, the dearth of validated instruments to assess CR in OMPT presents a serious challenge to certifying institutions. Moreover, the lack of documentation of the development process for instruments that measure CR pose additional challenges. The purpose of this study is to evaluate the sensibility of a newly developed instrument for assessing written responses to a test of CR in OMPT; a 'pilot' phase that examines instrument feasibility and acceptability. Using a sequential mixed-methods approach, Canadian OMPT examiners were recruited to first review and use the instrument. Participants completed a sensibility questionnaire followed by semi-structured interviews, the latter of which were used to elaborate on questionnaire responses regarding feasibility and acceptability. Eleven examiners completed the questionnaire and interviews. Questionnaire results met previously established sensibility criteria, while interview data revealed participants' (dis)comfort with exerting their own judgment and with the rating scale. Quantitative and qualitative data provided valuable insight regarding content validity and issues related to efficiency in assessing CR competence; all of which will ultimately inform further psychometric testing. While results suggest that the new instrument for assessing clinical reasoning in the Canadian certification context is sensible, future research should explore how rater judgment can be utilized effectively and the mental workload associated with appraising clinical reasoning.

Measuring the effectiveness of mentoring as a knowledge translation intervention for implementing empirical evidence: a systematic review

BACKGROUND

Mentoring as a knowledge translation (KT) intervention uses social influence among healthcare professionals to increase use of evidence in clinical practice.

AIM

To determine the effectiveness of mentoring as a KT intervention designed to increase healthcare professionals' use of evidence in clinical practice.

METHODS

A systematic review was conducted using electronic databases (i.e., MEDLINE, CINAHL), grey literature, and hand searching. Eligible studies evaluated mentoring of healthcare professionals responsible for patient care to enhance the uptake of evidence into practice. Mentoring is defined as (a) a mentor more experienced than mentee; (b) individualized support based on mentee's needs; and (c) involved in an interpersonal relationship as indicated by mutual benefit, engagement, and commitment. Two reviewers independently screened citations for eligibility, extracted data, and appraised quality of studies. Data were analyzed descriptively.

RESULTS

Of 10,669 citations from 1988 to 2012, 10 studies were eligible. Mentoring as a KT intervention was evaluated in Canada, USA, and Australia. Exposure to mentoring compared to no mentoring improved some behavioral outcomes (one study). Compared to controls or other multifaceted interventions, multifaceted interventions with mentoring improved practitioners' knowledge (four of five studies), beliefs (four of six studies), and impact on organizational outcomes (three of four studies). There were mixed findings for changes in professionals' behaviors and impact on practitioners' and patients' outcomes: some outcomes improved, while others showed no difference.

LINKING EVIDENCE TO ACTION

Only one study evaluated the effectiveness of mentoring alone as a KT intervention and showed improvement in some behavioral outcomes. The other nine studies that evaluated the effectiveness of mentoring as part of a multifaceted intervention showed mixed findings, making it difficult to determine the added effect of mentoring. Further research is needed to identify effective mentoring as a KT intervention.

Development of a simple 12-item theory-based instrument to assess the impact of continuing professional development on clinical behavioral intentions

BACKGROUND

Decision-makers in organizations providing continuing professional development (CPD) have identified the need for routine assessment of its impact on practice. We sought to develop a theory-based instrument for evaluating the impact of CPD activities on health professionals' clinical behavioral intentions.

METHODS AND FINDINGS

Our multipronged study had four phases. 1) We systematically reviewed the literature for instruments that used socio-cognitive theories to assess healthcare professionals' clinically-oriented behavioral intentions and/or behaviors; we extracted items relating to the theoretical constructs of an integrated model of healthcare professionals' behaviors and removed duplicates. 2) A committee of researchers and CPD decision-makers selected a pool of items relevant to CPD. 3) An international group of experts (n = 70) reached consensus on the most relevant items using electronic Delphi surveys. 4) We created a preliminary instrument with the items found most relevant and assessed its factorial validity, internal consistency and reliability (weighted kappa) over a two-week period among 138 physicians attending a CPD activity. Out of 72 potentially relevant instruments, 47 were analyzed. Of the 1218 items extracted from these, 16% were discarded as improperly phrased and 70% discarded as duplicates. Mapping the remaining items onto the constructs of the integrated model of healthcare professionals' behaviors yielded a minimum of 18 and a maximum of 275 items per construct. The partnership committee retained 61 items covering all seven constructs. Two iterations of the Delphi process produced consensus on a provisional 40-item questionnaire. Exploratory factorial analysis following test-retest resulted in a 12-item questionnaire. Cronbach's coefficients for the constructs varied from 0.77 to 0.85.

CONCLUSION

A 12-item theory-based instrument for assessing the impact of CPD activities on health professionals' clinical behavioral intentions showed adequate validity and reliability. Further studies could assess its responsiveness to behavior change following CPD activities and its capacity to predict health professionals' clinical performance.

Twelve tips for getting started using mixed methods in medical education research

BACKGROUND

Mixed methods research, which is gaining popularity in medical education, provides a new and comprehensive approach for addressing teaching, learning, and evaluation issues in the field.

AIM

The aim of this article is to provide medical education researchers with 12 tips, based on consideration of current literature in the health professions and in educational research, for conducting and disseminating mixed methods research.

CONCLUSION

Engaging in mixed methods research requires consideration of several major components: the mixed methods paradigm, types of problems, mixed method designs, collaboration, and developing or extending theory. Mixed methods is an ideal tool for addressing a full range of problems in medical education to include development of theory and improving practice.

Factor analysis methods and validity evidence: a review of instrument development across the medical education continuum

PURPOSE

Instrument development consistent with best practices is necessary for effective assessment and evaluation of learners and programs across the medical education continuum. The author explored the extent to which current factor analytic methods and other techniques for establishing validity are consistent with best practices.

METHOD

The author conducted electronic and hand searches of the English-language medical education literature published January 2006 through December 2010. To describe and assess current practices, she systematically abstracted reliability and validity evidence as well as factor analysis methods, data analysis, and reported evidence from instrument development articles reporting the application of exploratory factor analysis and principal component analysis.

RESULTS

Sixty-two articles met eligibility criteria. They described 64 instruments and 95 factor analyses. Most studies provided at least one source of evidence based on test content. Almost all reported internal consistency, providing evidence based on internal structure. Evidence based on response process and relationships with other variables was reported less often, and evidence based on consequences of testing was not identified. Factor analysis findings suggest common method selection errors and critical omissions in reporting.

CONCLUSIONS

Given the limited reliability and validity evidence provided for the reviewed instruments, educators should carefully consider the available supporting evidence before adopting and applying published instruments. Researchers should design for, test, and report additional evidence to strengthen the argument for reliability and validity of these measures for research and practice.