The relevance to clinical practice of the certifying examination in internal medicine

The effectiveness of cognitive forcing strategies to decrease diagnostic error: an exploratory study

BACKGROUND

Cognitive forcing strategies, a form of metacognition, have been advocated as a strategy to prevent diagnostic error. Increasingly, curricula are being implemented in medical training to address this error. Yet there is no experimental evidence that these curricula are effective.

DESCRIPTION

This was an exploratory, prospective study using consecutive enrollment of 56 senior medical students during their emergency medicine rotation. Students received interactive, standardized cognitive forcing strategy training.

EVALUATION

Using a cross-over design to assess transfer between similar (to instructional cases) and novel diagnostic cases, students were evaluated on 6 test cases. Forty-seven students were immediately tested and 9 were tested 2 weeks later. Data were analyzed using descriptive statistics and a McNemar chi-square test.

CONCLUSIONS

This is the first study to explore the impact of cognitive forcing strategy training on diagnostic error. Our preliminary findings suggest that application and retention is poor. Further large studies are required to determine if transfer across diagnostic formats occurs.

Guidelines for interpretation of some common indicators of residency program performance

OBJECTIVE

To provide national norms for indicators of residency-training program quality and information on their reproducibility.

PARTICIPANTS

The 364 residency-training programs that had 4 or more candidates take the 1989 to 1991 certifying examination in internal medicine for the first time.

DESIGN

Within each residency, program directors' ratings of medical knowledge, certifying examination scores, and certification status (pass or fail) were available for each candidate from 1989 to 1991. Means of these data were calculated for each program for each year of the study. To provide a way of comparing an individual program with all other programs, percentiles are reported for each year. To assess the precision of the measures, generalizability theory was applied and confidence intervals for all data are reported for programs of various size (1 to 25 residents taking the examination) and over the years (1 to 3).

RESULTS

Over the 3 years of the study, knowledge ratings, certification rates, and composite scores declined slightly. The correlations between program ratings of medical knowledge and the composite scores ranged from .47 to .60 and certification rates ranged from .44 to .55. The confidence intervals around all of the program performance indicators are relatively large and are affected most by the number of residents in the program. There is little variability across the years.

CONCLUSIONS

In smaller programs the precision of the performance indicators is poor; in programs with only a few residents they are virtually meaningless. On the positive side, programs are relatively stable and aggregating indicators over years is a reasonable way to increase their precision in assessing program performance. If the goal of program evaluation is to identify programs at the extremes, especially those at the low end, then such data may help guide program directors and educators.

Certification in internal medicine: 1989-1992

OBJECTIVE

To determine whether changes in the demographic/educational mix of those entering internal medicine from 1986 to 1989 were associated with differences among them at the time of certification.

PARTICIPANTS

Included in the study were all candidates for the 1989 to 1992 American Board of Internal Medicine certifying examinations in internal medicine.

MEASUREMENTS

Demographic information and medical school, residency training, and examination experience were available for each candidate. Data defining quality, size, and number of subspecialties were available for internal medicine training programs.

RESULTS

From 1990 to 1992, the total number of men and women candidates increased as did the numbers of foreign-citizen non-U.S. medical school graduates and osteopathic medical school graduates; the number of U.S. medical school graduates remained nearly constant and the number of U.S.-citizen graduates of non-U.S. medical schools declined. The pass rates for all groups of first-time examination takers decreased, while the ratings of program directors remained relatively constant. Program quality, size, and number of subspecialty programs had modest positive relationships with examination performance.

CONCLUSIONS

Changes in the characteristics of those entering internal medicine from 1986 to 1989 were associated with declines in performance at the time of certification. These declines occurred in all content areas of the test and were apparent regardless of program quality. These data identify some of the challenges internal medicine faces in the years ahead.