Problem Area Formation as an Element of Computer Aided Diagnosis: A Comparison of Two Strategies Within Quick Medical Reference (QMR)

Influence of case and physician characteristics on perceptions of decision support systems

OBJECTIVE

This study examines how characteristics of clinical cases and physician users relate to the users' perceptions of the usefulness of the Quick Medical Reference (QMR) and their confidence in their diagnoses when supported by the decision support system.

METHODS

A national sample (N = 108) of 67 internists, 35 family physicians, and 6 other U.S. physicians used QMR to assist in the diagnosis of written clinical cases. Three sets of eight cases stratified by diagnostic difficulty and the potential of QMR to produce high-quality information were used. A 2 x 2 repeated-measures analysis of variance was used to test whether these factors were associated with perceived usefulness of QMR and physicians' diagnostic confidence after using QMR. Correlations were computed among physician characteristics, ratings of QMR usefulness, and physicians' confidence in their own diagnoses, and between usefulness or confidence and actual diagnostic performance.

RESULTS

The analyses showed that QMR was perceived to be significantly more useful (P < 0.05) on difficult cases, on cases where QMR could provide high-quality information, by non-board-certified physicians, and when diagnostic confidence was lower. Diagnostic confidence was higher when comfort with using certain QMR functions was higher. The ratings of usefulness or diagnostic confidence were not consistently correlated with diagnostic performance.

CONCLUSIONS

The results suggest that users' diagnostic confidence and perceptions of QMR usefulness may be associated more with their need for decision support than with their actual diagnostic performance when using the system. Evaluators may fail to find a diagnostic decision support system useful if only easy cases are tested, if correct diagnoses are not in the system's knowledge base, or when only highly trained physicians use the system.

Effects of a decision support system on physicians' diagnostic performance

PURPOSE

This study examines how the information provided by a diagnostic decision support system for clinical cases of varying diagnostic difficulty affects physicians' diagnostic performance.

METHODS

A national sample of 67 internists, 35 family physicians, and 6 other physicians used the Quick Medical Reference (QMR) diagnostic decision support system to assist them in the diagnosis of written clinical cases. Three sets of eight cases, stratified by diagnostic difficulty and the potential of QMR to produce high-quality information, were used. The effects of using QMR on three measures of physicians' diagnostic performance were analyzed using analyses of variance.

RESULTS

Physicians' diagnostic performance was significantly higher (p < 0.01) on the easier cases and the cases for which QMR could provide higher-quality information.

CONCLUSIONS

Physicians' diagnostic performance can be strongly influenced by the quality of information the system produces and the type of cases on which the system is used.

Desiderata for product labeling of medical expert systems

The proliferation and increasing complexity of medical expert systems raise ethical and legal concerns about the ability of practitioners to protect their patients from defective or misused software. Appropriate product labeling of expert systems can help clinical users to understand software indications and limitations. Mechanisms of action and knowledge representation schema should be explained in layperson's terminology. User qualifications and resources available for acquiring the skills necessary to understand and critique the system output should be listed. The processes used for building and maintaining the system's knowledge base are key determinants of the product's quality, and should be carefully documented. To meet these desiderata, a printed label is insufficient. The authors suggest a new, more active, model of product labeling for medical expert systems that involves embedding 'knowledge of the knowledge base', creating user-specific data, and sharing global information using the Internet.

An appraisal of INTERNIST-I

INTERNIST-I was an expert system designed in the early 1970's to diagnose multiple diseases in internal medicine by modelling the behaviour of clinicians. Its form and operation are described, and evaluations of the system are surveyed. The major result of the project was its knowledge base which has been used in successor systems for medical education and clinical use. We also survey the effects of the project through these systems, and conclude that the most successful of them in the near future is likely to be Quick Medical Reference (QMR) when used as an "electronic textbook" of medicine.

Performance of four computer-based diagnostic systems

BACKGROUND

Computer-based diagnostic systems are available commercially, but there has been limited evaluation of their performance. We assessed the diagnostic capabilities of four internal medicine diagnostic systems: Dxplain, Iliad, Meditel, and QMR.

METHODS

Ten expert clinicians created a set of 105 diagnostically challenging clinical case summaries involving actual patients. Clinical data were entered into each program with the vocabulary provided by the program's developer. Each of the systems produced a ranked list of possible diagnoses for each patient, as did the group of experts. We calculated scores on several performance measures for each computer program.

RESULTS

No single computer program scored better than the others on all performance measures. Among all cases and all programs, the proportion of correct diagnoses ranged from 0.52 to 0.71, and the mean proportion of relevant diagnoses ranged from 0.19 to 0.37. On average, less than half the diagnoses on the experts' original list of reasonable diagnoses were suggested by any of the programs. However, each program suggested an average of approximately two additional diagnoses per case that the experts found relevant but had not originally considered.

CONCLUSIONS

The results provide a profile of the strengths and limitations of these computer programs. The programs should be used by physicians who can identify and use the relevant information and ignore the irrelevant information that can be produced.