An expert system for performance-based direct delivery of published clinical evidence

A scoping review of medical practice variation research within the informatics literature

RATIONALE

We performed a scoping review of informatics core literature about medical practice variation (MPV) as an agile summary of the subject in our field.

MATERIALS AND METHODS

The Ovid integrated database was searched between 1946 and 2022 to identify MPV studies published in major informatics journals and conference proceedings. Two reviewers performed relevance screening, with assistance from another independent reviewer for adjudication. The included articles were then thematically analyzed and summarized through discussion among all three reviewers.

RESULTS

A total of 43 articles were included and went through the thematic analysis. About half (n = 21) of the included articles were published in conference proceedings. Five articles reported the effect of MPV on patient outcomes. The variation of interest was most frequently in treatment decisions. In terms of the role informatics played (multiple roles allowed), 39 (90.7%) articles pertained to detection of MPV, 5 were about prevention of MPV and 4 about learning from MPV.

DISCUSSION

MPV remains a critical issue in health care, yet most informatics research has been focused on simple tasks such as automating the detection of MPV and assessing compliance to decision-support systems, and less focused on addressing the causes of variation or supporting learning from variation.

CONCLUSION

Our scoping review found that informatics studies have focused on detecting of MPV, especially variability in treatments and deviation from practice guidelines. Technological advances should promote more informatics research focused on explaining and learning from MPV.

Application of Artificial Intelligence in Community-Based Primary Health Care: Systematic Scoping Review and Critical Appraisal

BACKGROUND

Research on the integration of artificial intelligence (AI) into community-based primary health care (CBPHC) has highlighted several advantages and disadvantages in practice regarding, for example, facilitating diagnosis and disease management, as well as doubts concerning the unintended harmful effects of this integration. However, there is a lack of evidence about a comprehensive knowledge synthesis that could shed light on AI systems tested or implemented in CBPHC.

OBJECTIVE

We intended to identify and evaluate published studies that have tested or implemented AI in CBPHC settings.

METHODS

We conducted a systematic scoping review informed by an earlier study and the Joanna Briggs Institute (JBI) scoping review framework and reported the findings according to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis-Scoping Reviews) reporting guidelines. An information specialist performed a comprehensive search from the date of inception until February 2020, in seven bibliographic databases: Cochrane Library, MEDLINE, EMBASE, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), ScienceDirect, and IEEE Xplore. The selected studies considered all populations who provide and receive care in CBPHC settings, AI interventions that had been implemented, tested, or both, and assessed outcomes related to patients, health care providers, or CBPHC systems. Risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Two authors independently screened the titles and abstracts of the identified records, read the selected full texts, and extracted data from the included studies using a validated extraction form. Disagreements were resolved by consensus, and if this was not possible, the opinion of a third reviewer was sought. A third reviewer also validated all the extracted data.

RESULTS

We retrieved 22,113 documents. After the removal of duplicates, 16,870 documents were screened, and 90 peer-reviewed publications met our inclusion criteria. Machine learning (ML) (41/90, 45%), natural language processing (NLP) (24/90, 27%), and expert systems (17/90, 19%) were the most commonly studied AI interventions. These were primarily implemented for diagnosis, detection, or surveillance purposes. Neural networks (ie, convolutional neural networks and abductive networks) demonstrated the highest accuracy, considering the given database for the given clinical task. The risk of bias in diagnosis or prognosis studies was the lowest in the participant category (4/49, 4%) and the highest in the outcome category (22/49, 45%).

CONCLUSIONS

We observed variabilities in reporting the participants, types of AI methods, analyses, and outcomes, and highlighted the large gap in the effective development and implementation of AI in CBPHC. Further studies are needed to efficiently guide the development and implementation of AI interventions in CBPHC settings.

Impact of clinical reminder redesign on physicians' priority decisions

OBJECTIVE

Computerized clinical reminder (CCR) systems can improve preventive service delivery by providing patient-specific reminders at the point of care. However, adherence varies between individual CCRs and is correlated to resolution time amongst other factors. This study aimed to evaluate how a proposed CCR redesign providing information explaining why the CCRs occurred would impact providers' prioritization of individual CCRs.

DESIGN

Two CCR designs were prototyped to represent the original and the new design, respectively. The new CCR design incorporated a knowledge-based risk factor repository, a prioritization mechanism, and a role-based filter. Sixteen physicians participated in a controlled experiment to compare the use of the original and the new CCR systems. The subjects individually simulated a scenario-based patient encounter, followed by a semi-structured interview and survey.

MEASUREMENTS

We collected and analyzed the order in which the CCRs were prioritized, the perceived usefulness of each design feature, and semi-structured interview data.

RESULTS

We elicited the prioritization heuristics used by the physicians, and found a CCR system needed to be relevant, easy to resolve, and integrated with workflow. The redesign impacted 80% of physicians and 44% of prioritization decisions. Decisions were no longer correlated to resolution time given the new design. The proposed design features were rated useful or very useful.

CONCLUSION

This study demonstrated that the redesign of a CCR system using a knowledge-based risk factor repository, a prioritization mechanism, and a role-based filter can impact clinicians' decision making. These features are expected to ultimately improve the quality of care and patient safety.

Heterogeneous effect of an Emergency Department Expert Charting System

STUDY OBJECTIVE

We compare results from different modules (occupational exposure to blood and body fluids, low back pain, and fever in children) of the Emergency Department Expert Charting System. Each module of this electronic medical record provides real-time advice based on clinical guidelines embedded in the software.

METHODS

We used a staggered off-on-off interrupted time-series design with an intent-to-treat analysis to implement the Emergency Department Expert Charting System in a university hospital emergency department for the treatment of fever in children, low back pain, and occupational exposure to blood and body fluids. We measured the quality of documentation as the percentage of essential items contained in the medical record and discharge instructions, the percentage of appropriate testing and treatment decisions, median charges per patient visit, physician satisfaction by pre-experiment and postexperiment questionnaires, and patient satisfaction by telephone questionnaire.

RESULTS

The Emergency Department Expert Charting System improved documentation rates for all modules. The Emergency Department Expert Charting System consistently improved the appropriateness of diagnostic testing and treatment decisions for patients with occupational exposure to blood and body fluids while decreasing median charges. For the low back pain and fever in children modules, improvements in appropriateness of testing and treatment were less consistent and did not result in a decrease in charges. Although physicians were generally supportive of the intervention, the physicians' use of the Emergency Department Expert Charting System and satisfaction with the modules were greatest with occupational exposure to blood and body fluids and least with fever in children. For each presenting complaint, mean patient satisfaction was highest during the Emergency Department Expert Charting System phase.

CONCLUSION

The delivery of guidelines through an electronic medical record with background decision support improved documentation, patient care, and patient satisfaction, although effects were heterogeneous across presenting complaints. The optimal guideline implementation strategy likely varies with the nature of the clinical problem and the type of health care delivery setting. For selected problems, delivering guidelines in this format through the Internet holds great promise for modifying physician behavior and improving care (see http://www.needlestick.mednet.ucla.edu ).

Data mining methods find demographic predictors of preterm birth

BACKGROUND

Preterm births in the United States increased from 11.0% to 11.4% between 1996 and 1997; they continue to be a complex healthcare problem in the United States.

OBJECTIVE

The objective of this research was to compare traditional statistical methods with emerging new methods called data mining or knowledge discovery in databases in identifying accurate predictors of preterm births.

METHOD

An ethnically diverse sample (N = 19,970) of pregnant women provided data (1,622 variables) for new methods of analysis. Preterm birth predictors were evaluated using traditional statistical and newer data mining analyses.

RESULTS

Seven demographic variables (maternal age and binary coding for county of residence, education, marital status, payer source, race, and religion) yielded a .72 area under the curve using Receiving Operating Characteristic curves to test predictive accuracy. The addition of hundreds of other variables added only a .03 to the area under the curve.

CONCLUSION

Similar results across data mining methods suggest that results are data-driven and not method-dependent, and that demographic variables offer a small set of parsimonious variables with reasonable accuracy in predicting preterm birth outcomes in a racially diverse population.

Setting performance standards for medical practice: a theoretical framework

BACKGROUND

The assessment of performance in the real world of medical practice is now widely accepted as the goal of assessment at the postgraduate level. This is largely a validity issue, as it is recognised that tests of knowledge and in clinical simulations cannot on their own really measure how medical practitioners function in the broader health care system. However, the development of standards for performance-based assessment is not as well understood as in competency assessment, where simulations can more readily reflect narrower issues of knowledge and skills. This paper proposes a theoretical framework for the development of standards that reflect the more complex world in which experienced medical practitioners work.

METHODS

The paper reflects the combined experiences of a group of education researchers and the results of literature searches that included identifying current health system data sources that might contribute information to the measurement of standards.

CONCLUSION

Standards that reflect the complexity of medical practice may best be developed through an "expert systems" analysis of clinical conditions for which desired health care outcomes reflect the contribution of several health professionals within a complex, three-dimensional, contextual model. Examples of the model are provided, but further work is needed to test validity and measurability.