Evaluation of operational failures in clinical decision making

Advances in Quantitative Techniques for Making Medical Decisions

Articles relating decision theory to medical problems have appeared in the medical literature over the past thirtyyears. Mostof the work important to recent advances has been performed in the past 10 to 15 years. We describe the progress which occurred between 1978 and 1988: an increased acceptance of quantitative approaches, advances in analytical techniques, simplification of older methods to enhance accessibility, and a better understanding of the interface between methods that prescribe how decisions should be made and those that describe how decisions actually are made. We also discuss problems that have not been overcome and those which reflect the "growing pains" of success, including increased scrutiny by non-methodologist medical content experts.

Assessing explicit error reporting in the narrative electronic medical record using keyword searching

BACKGROUND

Many types of medical errors occur in and outside of hospitals, some of which have very serious consequences and increase cost. Identifying errors is a critical step for managing and preventing them. In this study, we assessed the explicit reporting of medical errors in the electronic record.

METHOD

We used five search terms "mistake," "error," "incorrect," "inadvertent," and "iatrogenic" to survey several sets of narrative reports including discharge summaries, sign-out notes, and outpatient notes from 1991 to 2000. We manually reviewed all the positive cases and identified them based on the reporting of physicians.

RESULT

We identified 222 explicitly reported medical errors. The positive predictive value varied with different keywords. In general, the positive predictive value for each keyword was low, ranging from 3.4 to 24.4%. Therapeutic-related errors were the most common reported errors and these reported therapeutic-related errors were mainly medication errors.

CONCLUSION

Keyword searches combined with manual review indicated some medical errors that were reported in medical records. It had a low sensitivity and a moderate positive predictive value, which varied by search term. Physicians were most likely to record errors in the Hospital Course and History of Present Illness sections of discharge summaries. The reported errors in medical records covered a broad range and were related to several types of care providers as well as non-health care professionals.

Methodologic challenges in developing and implementing measures of quality for child health care

OBJECTIVE

To review the major building blocks in measurement of quality for child health care, with recommendations for future research.

METHODS

We describe a framework of building blocks for quality measurement and discuss how an investigator's choices for each component are constrained by the special features of child health care.

RESULTS

Methodologic challenges for children's health care include developmental change and dependency on others, fragmentary care and inadequate health care data, unusual care settings, potential for long-term consequences, proxy reporting of outcomes and patient experience, small sample sizes, and lack of evidence that links processes and outcomes of care and of methods for risk adjustment. We cite examples of child-specific measures of quality that illustrate solutions to these challenges.

CONCLUSIONS

Children are different from adults, and measures of health care quality for children must differ from those for adults. We suggest future research on measures of quality directed toward overcoming the methodologic problems specific to child health care.

Pediatric asthma care in the emergency department: measuring the quality of history-taking and discharge planning

The National Asthma Education and Prevention Program NAEPP Guidelines include recommendations for history-taking and discharge planning during an asthma visit, but there are no tools to measure performance. The objectives of this study were to define and operationalize key elements of history-taking and discharge planning, to develop a tool for measuring these elements, and to evaluate the quality of history-taking and discharge planning in the emergency department (ED) during visits for asthma using the new tool. Expert opinion and extensive literature review were used to develop a 13-item checklist containing items that should be documented during history-taking and provided during discharge planning for an ED visit for an acute asthma exacerbation by children. A convenience sample of 90 pediatric emergency medicine physicians and allergists rated each item in the checklist. The checklist was used to score audiotapes of asthma visits in the ED. Subjects were 154 parents of asthmatic children aged 4-9 years seeking care in nine inner-city EDs affiliated with asthma centers participating in the National Cooperative Inner-City Asthma Study and the physician/providers who delivered care. Seven of the 13 items on the checklist were rated as required to be performed by more than 90% of the allergist/pediatric emergency medicine physicians. Only 10% of the 154 visits included all seven of the highly rated items, whereas 19% of the visits included three or fewer. Only 7 of the 13 items (54%) were performed in more than 50% of the visits, and 4 items were performed in fewer than 25% of visits. Based on expert ratings, the checklist for measuring elements of history-taking and discharge planning during asthma visits appears to have considerable face validity. In the visits studied, the overall performance of these elements was low. Interventions to improve performance on the checklist might lead to improved care for children with asthma who frequent the ED.

A randomized trial of "corollary orders" to prevent errors of omission

OBJECTIVE

Errors of omission are a common cause of systems failures. Physicians often fail to order tests or treatments needed to monitor/ameliorate the effects of other tests or treatments. The authors hypothesized that automated, guideline-based reminders to physicians, provided as they wrote orders, could reduce these omissions.

DESIGN

The study was performed on the inpatient general medicine ward of a public teaching hospital. Faculty and housestaff from the Indiana University School of Medicine, who used computer workstations to write orders, were randomized to intervention and control groups. As intervention physicians wrote orders for 1 of 87 selected tests or treatments, the computer suggested corollary orders needed to detect or ameliorate adverse reactions to the trigger orders. The physicians could accept or reject these suggestions.

RESULTS

During the 6-month trial, reminders about corollary orders were presented to 48 intervention physicians and withheld from 41 control physicians. Intervention physicians ordered the suggested corollary orders in 46.3% of instances when they received a reminder, compared with 21.9% compliance by control physicians (p < 0.0001). Physicians discriminated in their acceptance of suggested orders, readily accepting some while rejecting others. There were one third fewer interventions initiated by pharmacists with physicians in the intervention than control groups.

CONCLUSION

This study demonstrates that physician workstations, linked to a comprehensive electronic medical record, can be an efficient means for decreasing errors of omissions and improving adherence to practice guidelines.

Consistency in performance among primary care practitioners

OBJECTIVES

The authors studied the consistency of performance of individual physicians to evaluate the identification of outlier practitioners as a strategy for improving patient care.

METHODS

The authors used a data base containing information on 430 practitioners caring for 6,090 patients in 16 group practices. The authors analyzed inter- and intraphysician differences in performance on the basis of review criteria for 8 patient care guidelines. These criteria allowed for a variety of acceptable clinical strategies, incorporated decision tree logic, and included input from participating practitioners. The authors took steps to maximize validity and controlled for potentially confounding characteristics of patients and practitioners. The authors identified outliers, evaluated the significance of differences between outliers and nonoutliers, and studied variations in performance across cases and guidelines in conformance with guidelines.

RESULTS

The authors identified a few statistically significant outliers. Correlations for performance across cases seen by a given physician were low. The highest positive correlation for performance between any pair of guidelines was 0.32.

CONCLUSIONS

The performance of a given practitioner is highly variable from patient to patient and from guideline to guideline. Thus, strategies focusing solely on substandard outliers will miss opportunities to improve performance.

The Ambulatory Care Medical Audit Demonstration Project. Research design

The authors describe the design of and statistical analyses involved in the Ambulatory Care Medical Audit Demonstration Project, which tested feasibility, cost, and effectiveness of cycles that met quality assurance requirements in eight pediatric and eight general medicine group practices at four teaching hospitals and six health centers. The authors used a concurrent crossover design using randomized cycles of quality assurance so that a practice was a control site for one guideline and an experimental site for another. For 12 months before and 18 months during and after quality assurance experimental interventions, the authors measured practitioner conformance to review criteria for patient-care guidelines believed to improve outcomes, including four internal medicine patient-care guidelines (ie, follow-up of low hematocrit, cancer screening for women, follow-up of high serum glucose, and monitoring of patients treated with digoxin) and four pediatric patient-care guidelines (ie, follow-up of positive urine cultures, screening for disease and immunizing infants, management of acute gastroenteritis, and management of acute ear infection). The authors distinguished review criteria whose performance depended on personal efforts of practitioners from those that concerned performance dependent on the practice's system for reporting test results and calling patients to return for care.

Variability in clinical systems: applying modern quality control methods to health care

BACKGROUND

Quality health care depends on timely completion of sequences of clinical care. This study evaluated the concepts and tools of quality management for measuring system performance in ambulatory care.

STUDY DESIGN

Clinical staff in nine centers of a group model health maintenance organization described the procedure for ordering and completing a complete blood count, mammogram, and surgical consultation. Variability was noted among the processes as intended and as actually performed, as well as inconsistencies reported within and among centers. In two centers investigators tracked performance of key sequences of care and the achievement of other key quality characteristics believed desired by physicians and patients. Computerized patient records and departmental files were the only available sources of data for assessing completion and followup of tests and consultations. Even these data were difficult to obtain and, in many instances, incomplete.

RESULTS

Although data were often difficult to obtain, the quality management techniques used were helpful in revealing process failures that appeared to be the result of design flaws built into the clinical systems.

CONCLUSION

Robust process designs and improved management information systems for monitoring these processes are recommended to reduce variability and improve the quality of clinical care.

Using decision analysis to communicate the importance of quality assurance studies

Decision analysis, which structures the information used by physicians in making medical decisions, has the potential for being a valuable quality assurance (QA) tool. Probability trees derived from decision analysis studies can be used to make quantitative, graphic connections between QA data and adverse medical outcomes. This article provides four hypothetical examples of how decision analysis studies can help providers understand the relationship between QA data and patient welfare.

Intermediate, indeterminate, and uninterpretable diagnostic test results

Diagnostic tests do not always yield positive or negative results; sometimes the results are intermediate, indeterminate, or uninterpretable. No consensus exists for the incorporation of such results into data assessment. Conventional Bayesian analysis leads investigators to either exclude patients with non-positive, non-negative results from their studies or categorize such results into inappropriate cells of the standard four-cell decision matrix. The authors propose a standardized method for reporting results in studies dealing with diagnostic test use and discuss how researchers should expand the four-cell matrix to six cells when non-positive, non-negative results occur. They suggest that the six-cell matrix with new operational definitions of sensitivity, specificity, likelihood ratios, and test yield should be adopted routinely. In addition, they define the different types of non-positive, non-negative results and demonstrate how clinicians can use tree-structured decision analysis from the six-cell matrix. While their method does not solve all problems posed by non-positive, non-negative results, it does suggest a standard method for reporting these results and utilizing all the data in decision making.