Psychometric Properties of an Instrument Designed to Measure the Educational Quality of Graduate Training Programs

To assess the quality of residency education programs at an academic medical center for purposes of enhancing individual graduate medical education programs, we asked residents and fellows (N = 419) to evaluate their training programs using a Web-based questionnaire (response rate = 70%). Kruskal-Wallis tests, factor analysis, correlations, generalizability/decision studies, and mean plots were used to examine trainee responses and to assess the questionnaire’s measurement properties. Exploratory factor analysis indicated that the instrument had a threefactor structure that correlated highly with overall program rating. Cronbach’s alpha exceeded .80 for all factors, and decision studies revealed that 13 to 23 raters were needed to obtain G-coefficients greater than .70. Mean plots showed that the instrument could discriminate within and among training programs at the item level and the factor level, which should help target improvements across graduate training programs within large institutions.

Medical Student Performance

Studies of the evaluation of medical students' clinical performance frequently do not differentiate between ratings by house officer and attending staff evaluators. This practice is not appropriate, since research investigations have shown that house officers rate medical students' clinical performance higher and have higher interrater agreement than do attending staff This investigation studies one aspect of the validity of medical students' clinical performance ratings and demonstrates that there are higher correlations between house officer ratings of student knowledge and student cognitive ability scores than there are between attending staff evaluations and student ability scores.

Just imagine: new paradigms for medical education

For all its traditional successes, the current model of medical education in the United States and Canada is being challenged on issues of quality, throughput, and cost, a process that has exposed numerous shortcomings in its efforts to meet the needs of the nations' health care systems. A radical change in direction is required because the current path will not lead to a solution.The 2010 publication Educating Physicians: A Call for Reform of Medical School and Residency identifies several goals for improving the medical education system, and proposals have been made to reform medical education to meet these goals. Enacting these recommendations practically and efficiently, while training more health care providers at a lower cost, is challenging.To advance solutions, the authors review innovations that are disrupting higher education and describe a vision for using these to create a new model for competency-based, learner-centered medical education that can better meet the needs of the health care system while adhering to the spirit of the above proposals. These innovations include collaboration amongst medical schools to develop massive open online courses for didactic content; faculty working in small groups to leverage this online content in a "flipped-classroom" model; and digital badges for credentialing entrustable professional activities over the continuum of learning.

Examination of a clinical teaching effectiveness instrument used for summative faculty assessment

This study explores whether a clinical teaching effectiveness (CTE) instrument provides valid scores for summative faculty assessment. The sample included all CTE instruments (n = 10,087) that learners (N = 1,194) completed to assess clinical teachers (N = 872) during 1 academic year. The authors investigated response processes (e.g., missing data, straight-line responses, level of learner), internal structure (e.g., confirmatory and exploratory factor analysis), teaching ratings by learner group (medical student or resident), and relation to other variables (e.g., correlation with global rating). Response processes identified a high prevalence of straight-line responses (same rating across all items) and differential patterns of missing data by learner group. Medical students rated their teachers higher than residents, and CTE scores had different factor structures depending on learner group. High correlation coefficients of CTE items with a single rating of overall teaching performance suggest that learners consider global performance when assessing clinical teaching performance.

Methods to assess students' acquisition, application and integration of basic science knowledge in an innovative competency-based curriculum

BACKGROUND

The Cleveland Clinic Lerner College of Medicine was designed to encourage medical students to pursue careers as physician investigators. Our faculty decided that assessment should enhance learning and adopted only formative assessments to document student performance in relation to nine broad-based competencies. No grades are used to judge student performance throughout the 5-year program. Instead, assessments are competency-based, relate directly to performance standards, and are stored in e-Portfolios to track progress and document student achievement. The class size is limited to 32 students a year.

AIMS

Schools with competency-based curricula must provide students with formative feedback to identify performance gaps and monitor progress. We describe a systematic approach to assess medical knowledge using essay-type questions (CAPPs) and multiple choice questions (SAQs) to provide medical students with weekly, formative feedback about their abilities to acquire, apply and integrate basic and clinical science concepts.

METHOD

Processes for developing performance standards, creating assessment items, training faculty, reporting student performance and monitoring outcomes are described. A case study of a Year 1 course is presented with specific examples of CAPPs and SAQs to illustrate how formative assessment data are interpreted and reported in students' e-Portfolios.

RESULTS

Preliminary evidence suggests that CAPPs and SAQs have a positive impact on students' education, a justifiable cost in light of obtained benefits and growing acceptance among stakeholders. Two student cohorts performed significantly above the population mean on USMLE Step 1, which suggests that these assessment methods have not disadvantaged students. More evidence is needed to assess the reliability and validity of these tools for formative purposes.

CONCLUSIONS

Using assessment data for formative purposes may encourage application and integration of knowledge, help students identify performance gaps, foster student development of learning plans and promote student responsibility for learning. Discussion provides applications for institutions with larger classes to consider.

Cleveland Clinic Lerner College of Medicine: an innovative approach to medical education and the training of physician investigators

Cleveland Clinic Lerner College of Medicine (CCLCM) is an innovative, five-year medical education track within Case Western Reserve University School of Medicine (Case) with a focused mission to attract and educate a limited number of highly qualified persons who seek to become physician investigators. CCLCM curriculum governance, faculty appointments and promotions, and admissions committees are integrated with respective Case committees. The CCLCM curriculum is based on faculty-defined professional attributes that graduates are expected to develop. These attributes were used to create curricular and assessment principles that guided the development of an integrated basic science, clinical science, and research curriculum, conducted in an active learning environment. An organ-system approach is used to solidify an understanding of basic science discipline threads in the context of relevant clinical problems presented in PBL and case-based discussion formats. Clinical skills are introduced in the first year as part of the two-year longitudinal experience with a family practice or internal medicine physician. The research program provides all students with opportunities to learn and experience basic and translational research and clinical research before selecting a research topic for their 12- to 15-month master-level thesis project. All Case students participate in required and elective clinical curriculum after the second year, but CCLCM students return to the Cleveland Clinic on selected Friday afternoons for program-specific research and professionalism-learning activities. A unique portfolio-based assessment system is used to assess student achievements in nine competency areas, seven of which reflect the Accreditation Council for Graduate Medical Education competencies.

Developing a mentorship program for clinical researchers

INTRODUCTION

In academic health centers, the number of physician investigators and the number of research studies headed by clinicians has been declining. The U.S. Institute of Medicine and the National Institutes of Health suggest improved mentoring is important to reversing these trends.

METHODS

This is a case study review of the role of mentors in developing and sustaining clinical investigators at the Cleveland Clinic.

RESULTS

Issues influencing mentoring relationships at the Cleveland Clinic include whether mentoring contracts are formally or informally related and are agreed on; whether there are scheduled meetings for mentors and protégés; whether there are clearly articulated benchmarks and manageable steps; and whether there is a code of mentorship.

CONCLUSIONS

Interactive group formats can lead to development of a code of mentorship and increased awareness of faculty regarding clinical investigation.

A 47-year-old alcoholic man with progressive abnormal gait

Central pontine myelinolysis should be considered in the differential diagnosis of a patient with a history of alcoholism and malnutrition presenting with ataxia, regardless of serum sodium values. T2-weighted images are the most sensitive imaging technique, but changes may not be evident for weeks after the insult, and in addition, the insult may not be known. Supportive care is important to prevent complications, but no treatment has been found to be effective in treating the illness. Patient outcomes vary considerably and are difficult to predict.

Validity of three clinical performance assessments of internal medicine clerks

PURPOSE

To analyze the construct validity of three methods to assess the clinical performances of internal medicine clerks.

METHOD

A multitrait-multimethod (MTMM) study was conducted at the Case Western Reserve University School of Medicine to determine the convergent and divergent validity of a clinical evaluation form (CEF) completed by faculty and residents, an objective structured clinical examination (OSCE), and the medicine subject test of the National Board of Medical Examiners. Three traits were involved in the analysis: clinical skills, knowledge, and personal characteristics. A correlation matrix was computed for 410 third-year students who completed the clerkship between August 1988 and July 1991.

RESULTS

There was a significant (p < .01) convergence of the four correlations that assessed the same traits by using different methods. However, the four convergent correlations were of moderate magnitude (ranging from .29 to .47). Divergent validity was assessed by comparing the magnitudes of the convergence correlations with the magnitudes of correlations among unrelated assessments (i.e., different traits by different methods). Seven of nine possible coefficients were smaller than the convergent coefficients, suggesting evidence of divergent validity. A significant CEF method effect was identified.

CONCLUSION

There was convergent validity and some evidence of divergent validity with a significant method effect. The findings were similar for correlations corrected for attenuation. Four conclusions were reached: (1) the reliability of the OSCE must be improved, (2) the CEF ratings must be redesigned to further discriminate among the specific traits assessed, (3) additional methods to assess personal characteristics must be instituted, and (4) several assessment methods should be used to evaluate individual student performances.

Informed consent for blood transfusion as a transfusion medicine educational intervention

The aim here was to determine the effectiveness of a transfusion medicine educational intervention in a medicine core clerkship program. Third-year medical students enrolled in their medicine core clerkship rotations at tertiary care hospitals affiliated with our institution underwent a two-part educational intervention that incorporated a transfusion medicine curriculum within the context of the medicolegal, ethical and educational elements of informed consent. Part one was a 1-h didactic session on standards of practice for red blood cell transfusion. Part two was a 90-min multidisciplinary workshop on informed consent. The effectiveness of the educational intervention was analysed by an objective structured clinical evaluation. The student group receiving the educational intervention scored significantly higher than in the comparison group (65.8 +/- 9.2 vs. 54.1 +/- 10.56, P < 0.001). When student scores were used to determine changes in student response patterns over time, the largest change occurred in identifying possible other options to allogeneic blood transfusion. These results suggest that a transfusion medicine curriculum using an informed consent model can be used effectively as an educational intervention in a medicine core clerkship programme.

Assessment of hand function in Duchenne muscular dystrophy

Eighteen boys with Duchenne muscular dystrophy (DMD) were assessed for their ability to perform tasks involving wrist and hand function. Each subject was assessed using the Jebsen Test of Hand Function, range of motion measurements, and muscle strength tests. Writing and simulated page turning were performed successfully by boys in all age groups. Boys over age 15 had difficulty completing simulated feeding and picking up large and small objects. The muscle strength of the wrist extensors and the radial deviation range of motion at the wrist were found to be strongly correlated with six of the seven tasks assessed. These two clinical assessments appear to be good indicators of overall wrist and hand function. Life expectancy with DMD is increasing with advances in respiratory care making preservation of wrist and hand function, the major activity remaining with advanced disease, increasingly important.

Prevalence and correlates of hospital-based autologous blood programs: a statewide survey

To identify potential barriers to use of autologous blood procurement to minimize homologous blood transfusion needs during elective surgery, the authors conducted a telephone survey of 120 blood bank directors, representing 138 Ohio hospitals. The prevalence of autologous blood procurement facilities, estimated volume of autologous blood, and attitudes and perceptions of the directors toward autologous blood predeposit programs were assessed. Analysis of the data indicated that 30% of Ohio hospitals have autologous blood procurement facilities; larger hospitals were more likely to have this facility. Overall, 5.5% of transfusions involve predeposited autologous blood. No significant differences were found according to hospital bed size or whether the hospital had a procurement facility. Blood bank directors perceived surgeons to be knowledgeable about autologous predeposit; patient demand and surgical practice were felt to be more effective in promoting the use of autologous blood at the hospital than were blood bank efforts. Directors who had autologous predeposit procurement facilities perceived that the facility provided a marketing advantage. Respondents from larger hospitals were more likely to perceive that these programs could be financially self-sufficient. The authors conclude that an economic cost-benefit analysis of hospital-based autologous blood procurement programs is important. Positive findings may influence transfusion services to adopt autologous blood procurement programs, whereas negative findings may convince hospitals that community blood donor facilities can provide better autologous blood procurement.

A retrospective analysis of grand rounds in continuing medical education

Grand Rounds activities are an integral component of medical education. They present clinical problems in medicine by focusing on current or interesting cases. Grand Rounds originated as part of residency training wherein new information was taught and clinical reasoning skills were enhanced. Today, Grand Rounds also are used in continuing medical education as an instructional method for maintaining and improving clinical skills of practicing physicians. This study reviewed 43 Grand Rounds activities conducted by 12 departments in five affiliated hospitals as part of a program in Continuing Medical Education. All the Grand Rounds activities were developed to meet Category 1 requirements. They included the departmental Grand Rounds and the more specialized divisional Grand Rounds. An analysis of the 1138 physicians who attended one or more of the 1079 meetings of these Grand Rounds produced a preliminary description of a "typical" Grand Rounds activity. Analyses of these data suggest that Grand Rounds provides an important educational activity for medical students and graduate medical trainees as well as practicing physicians. The Category 1 accreditation requirements may serve to improve the educational outcomes of these activities and they provide an excellent mechanism for faculty role-modeling and improving clinical problem-solving skills.

Correlates of community resource information needs of arthritis patients

One hundred eighty-nine ambulatory arthritis patients were interviewed to assess needs for information about community resource services. Participants were asked how frequently they needed 66 specific information items. Three categories of resource needs in arthritis care were identified: general information, personal care and medical services, and assistive devices and equipment. In the first category, results indicated that the greater the impairment in functional ability, the greater the need for general information about arthritis. When functional ability was held constant, younger patients with lower family income displayed a greater need for arthritis resource information. Lower functional ability and family income were associated with a greater need for personal care and medical service information. Lower functional ability contributed the most to the need for information about assistive devices and equipment. We suggest that, as an essential component of the long-term care of patients with rheumatic diseases, individuals who provide educational counseling should clearly understand the community resource needs of patients, and should be able to match articulated needs with the existing services. Health professionals should be aware that certain patient characteristics (specifically, family income, functional ability, and age) are related to a need for this type of information. The large percentage of patients who could not name their rheumatic disease diagnosis (65%) presents a major concern and challenge to patient educators.

High blood pressure in older Americans. The First National Health and Nutrition Examination Survey

Data from a representative sample of the U.S. adult population obtained during 1971-1975 were analyzed to provide a profile of blood pressure (BP) levels and related nutritional and sociodemographic factors. Older adults (aged 55-74 years) had a twofold greater prevalence of high BP than younger adults (25-54 years), and older black persons had the highest rates. Isolated systolic elevation was uncommon under 54 years of age, but occurred in 5% to 10% of adults over 55 years and was less common than systolic-diastolic elevation. In older adults, body mass (weight/height2) had the strongest relationship to BP of all the nutritional variables. Alcohol consumption and dietary calcium and phosphorus were associated with high BP, but dietary sodium and salt use were not. The serum calcium/phosphorus ratio and serum urate were significantly higher in older adults with high BP. In general, the variables associated with elevated BP in older adults were similar to those in younger adults, although the strengths of the associations differed. Associations of factors useful for nonpharmacologic prevention and management of high BP in older persons were suggested from this survey.

Blood pressure and nutrition in adults. The National Health and Nutrition Examination Survey

To provide a contemporary profile of blood pressure and nutritional and sociodemographic relationships in the adult US population, data from the first National Health and Nutrition Examination Survey ( NHANES -I), 1971-1975, were analyzed. Systolic and diastolic blood pressures increased with increasing age, but trends were different by sex and race groups. Body mass index (weight/ height2 ) was the nutritional factor most strongly and consistently related to blood pressure. Among dietary constituents, alcohol consumption and calcium and phosphorus intake were the only variables having consistent and independent relationships to blood pressure. Sodium content of food and salt use had no relationship, and sodium/potassium food content had only an inconsistent association. Regarding serum nutritional measures, serum calcium was directly related and serum phosphorus was inversely related to blood pressure. Serum urate, serum aspartate aminotransferase, and hemoglobin were also independently related to systolic and diastolic blood pressures. There were few important differences by race or sex in these correlates. These observations from a representative sample of the US population have useful implications for prevention and treatment of high blood pressure.

The effect of nonlinear transformations on a Likert scale

The analysis of data collected from behavioral assessment instruments is typically conducted using parametric statistics, with little or no reference given to the underlying nature of the scale being used. If the nature of the distances between the scale points is not understood, the concept of normality of the distribution becomes clouded. An empirical approach to studying this problem was developed, using responses to a clinical performance evaluation instrument that uses a four-point behaviorally-anchored scale. Various combinations of nonlinear transformations were applied to the evaluation responses. The factorial structure of the fifteen items constituting the evaluation form was minimally affected by the transformations, suggesting that parametric statistics can be applied to behaviorally-anchored rating scales.

Resident selection: applicant selection criteria compared with performance

One hundred three surgical house officers, 41 in general surgery and 62 in specialty surgery, entered the residency program from 1975 through 1979. Their application data, including the National Board of Medical Examiners test part I scores, preclinical course honors, clinical course honors, surgical clerkship honors, election to Alpha Omega Alpha (AOA), published research, medical school grading system, medical school rating, and National Resident Matching Program rank, were tabulated. Each house officer's performance was assessed by monthly faculty evaluation and by annual standard American Board of Surgery In-Training Examination (ABSITE) score through the second year of the program. The application data were compared with the performance data to determine significant indicators of success. Factor analysis computed on the monthly evaluation forms identified a knowledge factor and an interpersonal skills factor. Statistical analyses were used to study the relationships among the independent (preentry) variables and the knowledge, interpersonal skills, and ABSITE postentry variables. The results were significant (P less than 0.05) for medical school honors, election to AOA, and medical school grading system. No significance was found for the remaining preentry variables. The knowledge and interpersonal skill evaluation scores for the house officer I and II years were significantly correlated. Although the postentry assessment of knowledge correlated with certain applicant data, the assessment of interpersonal factors was not statistically related.

An examination of diabetes educational assessment forms

A survey was mailed to 1,518 health professionals to determine what information should be collected in an assessment of the educational needs of diabetic patients. Of the 802 respondents who returned com pleted surveys, 529 (66 percent) indicated that they were currently using an assessment form to deter mine the educational needs of their patients. Copies of 203 of these forms were collected and analyzed to determine the types of information health providers obtain from diabetes patients through the use of an assess ment form. The contents of the col lected forms were compared with the factor analysis of the results of the mailed survey. This comparison revealed a large discrepancy be tween what the total group of survey respondents considered important to assess and what was actually being assessed in the forms collected from some of the respondents. Multiple reasons may account for these results: a lack of awareness of exist ing assessment tools, a lack of skill in assessment methodology, a lack of readily available techniques which are transferable to busy practice set tings, and/or a lack of specification of program objectives.

A computerized cataloging management system for health science audiovisuals

This paper describes the implementation of the Stanford Public Information and Retrieval System (SPIRES) by the University of Michigan Medical School Learning Resources Center. SPIRES is a bibliographic data base management system which offers on-line search capabilities and retrieval of data in programmable formats. The Learning Resource Center utilizes SPIRES for the interactive retrieval of cataloging data, bibliographical compilations, and book catalog production.

Factors affecting the educational diagnosis of diabetic patients

An accurate determination of the educational needs of diabetic patients is based on a wide range of patient characteristics. In the development of an assessment instrument to determine the educational needs of patients, the relative importance of numerous patient characteristics was determined. A questionnaire was mailed to 1518 health professionals who are either members of the American Association of Diabetes Educators or community health nurses in the state of Michigan. Analyses of the responses to this questionnaire have provided information regarding the knowledge, psychosocial, and demographic characteristics of patients, which are thought to be important in an educational diagnosis. Questions to measure these identified characteristics will be incorporated into an instrument that will assess the educational needs of diabetic patients.