Iliad: Moving Medical Decision-Making into New Frontiers

Should electronic differential diagnosis support be used early or late in the diagnostic process? A multicentre experimental study of Isabel

Background

Diagnostic errors unfortunately remain common. Electronic differential diagnostic support (EDS) systems may help, but it is unclear when and how they ought to be integrated into the diagnostic process.

Objective

To explore how much EDS improves diagnostic accuracy, and whether EDS should be used early or late in the diagnostic process.

Setting

6 Canadian medical schools. A volunteer sample of 67 medical students, 62 residents in internal medicine or emergency medicine, and 61 practising internists or emergency medicine physicians were recruited in May through June 2020.

Intervention

Participants were randomised to make use of EDS either early (after the chief complaint) or late (after the complete history and physical is available) in the diagnostic process while solving each of 16 written cases. For each case, we measured the number of diagnoses proposed in the differential diagnosis and how often the correct diagnosis was present within the differential.

Results

EDS increased the number of diagnostic hypotheses by 2.32 (95% CI 2.10 to 2.49) when used early in the process and 0.89 (95% CI 0.69 to 1.10) when used late in the process (both p<0.001). Both early and late use of EDS increased the likelihood of the correct diagnosis being present in the differential (7% and 8%, respectively, both p<0.001). Whereas early use increased the number of diagnostic hypotheses (most notably for students and residents), late use increased the likelihood of the correct diagnosis being present in the differential regardless of one’s experience level.

Conclusions and relevance

EDS increased the number of diagnostic hypotheses and the likelihood of the correct diagnosis appearing in the differential, and these effects persisted irrespective of whether EDS was used early or late in the diagnostic process.

RDAD: A Machine Learning System to Support Phenotype-Based Rare Disease Diagnosis

DNA sequencing has allowed for the discovery of the genetic cause for a considerable number of diseases, paving the way for new disease diagnostics. However, due to the lack of clinical samples and records, the molecular cause for rare diseases is always hard to identify, significantly limiting the number of rare Mendelian diseases diagnosed through sequencing technologies. Clinical phenotype information therefore becomes a major resource to diagnose rare diseases. In this article, we adopted both a phenotypic similarity method and a machine learning method to build four diagnostic models to support rare disease diagnosis. All the diagnostic models were validated using the real medical records from RAMEDIS. Each model provides a list of the top 10 candidate diseases as the prediction outcome and the results showed that all models had a high diagnostic precision (≥98%) with the highest recall reaching up to 95% while the models with machine learning methods showed the best performance. To promote effective diagnosis for rare disease in clinical application, we developed the phenotype-based Rare Disease Auxiliary Diagnosis system (RDAD) to assist clinicians in diagnosing rare diseases with the above four diagnostic models. The system is freely accessible through http://www.unimd.org/RDAD/.

Discussion of "Computational Electrocardiography: Revisiting Holter ECG Monitoring"

This article is part of a For-Discussion-Section of Methods of Information in Medicine about the paper "Computational Electrocardiography: Revisiting Holter ECG Monitoring" written by Thomas M. Deserno and Nikolaus Marx. It is introduced by an editorial. This article contains the combined commentaries invited to independently comment on the paper of Deserno and Marx. In subsequent issues the discussion can continue through letters to the editor.

An update and further testing of a knowledge-based diagnostic clinical decision support system for musculoskeletal disorders of the shoulder for use in a primary care setting

RATIONALE, AIMS AND OBJECTIVES

A prototype diagnostic clinical decision support system (CDSS) was developed to assist primary care clinicians (general practitioners) in clinical decision making, aimed at reducing diagnostic errors. The prototype CDSS showed some promise with high levels of validity and reliability; however, issues regarding the underlying Bayesian belief network (BBN), small sample size and use of radiological imaging as a gold standard measure were highlighted that required further investigation before considering clinical testing.

METHODS

The prototype CDSS was reviewed and updated based on computer science literature and expert (orthopaedic consultant) opinion. The updated CDSS was tested by comparing its diagnostic outcome against the diagnosis of 93 case studies as determined by expert opinion combined with arthroscopy findings or radiological imaging.

RESULTS

The updated CDSS showed significant high levels of sensitivity (91%), specificity (98%), positive likelihood ratio (53.12) and negative likelihood ratio (0.08) with a kappa value of 0.88 to a confidence level of 99% compared with expert diagnosis combined with arthroscopy findings or radiological imaging.

CONCLUSIONS

The results suggest that the updated CDSS has addressed the issues highlighted from the initial research while maintaining high levels of validity and reliability. The updated CDSS is now ready for clinical testing.

Natural language processing: state of the art and prospects for significant progress, a workshop sponsored by the National Library of Medicine

Natural language processing (NLP) is crucial for advancing healthcare because it is needed to transform relevant information locked in text into structured data that can be used by computer processes aimed at improving patient care and advancing medicine. In light of the importance of NLP to health, the National Library of Medicine (NLM) recently sponsored a workshop to review the state of the art in NLP focusing on text in English, both in biomedicine and in the general language domain. Specific goals of the NLM-sponsored workshop were to identify the current state of the art, grand challenges and specific roadblocks, and to identify effective use and best practices. This paper reports on the main outcomes of the workshop, including an overview of the state of the art, strategies for advancing the field, and obstacles that need to be addressed, resulting in recommendations for a research agenda intended to advance the field.

Decision support in dermatology and medicine: history and recent developments

This article is focused on diagnostic decision support tools and will provide a brief history of clinical decision support (CDS), examine the components of CDS and its associated terminology, and discuss recent developments in the use and application of CDS systems, particularly in the field of dermatology. For this article, we use CDS to mean an interactive system allowing input of patient-specific information and providing customized medical knowledge-based results via automated reasoning, for example, a set of rules and/or an underlying logic, and associations.

Clinical diagnostics in human genetics with semantic similarity searches in ontologies

The differential diagnostic process attempts to identify candidate diseases that best explain a set of clinical features. This process can be complicated by the fact that the features can have varying degrees of specificity, as well as by the presence of features unrelated to the disease itself. Depending on the experience of the physician and the availability of laboratory tests, clinical abnormalities may be described in greater or lesser detail. We have adapted semantic similarity metrics to measure phenotypic similarity between queries and hereditary diseases annotated with the use of the Human Phenotype Ontology (HPO) and have developed a statistical model to assign p values to the resulting similarity scores, which can be used to rank the candidate diseases. We show that our approach outperforms simpler term-matching approaches that do not take the semantic interrelationships between terms into account. The advantage of our approach was greater for queries containing phenotypic noise or imprecise clinical descriptions. The semantic network defined by the HPO can be used to refine the differential diagnosis by suggesting clinical features that, if present, best differentiate among the candidate diagnoses. Thus, semantic similarity searches in ontologies represent a useful way of harnessing the semantic structure of human phenotypic abnormalities to help with the differential diagnosis. We have implemented our methods in a freely available web application for the field of human Mendelian disorders.

Computer-assisted diagnostic decision support: history, challenges, and possible paths forward

This paper presents a brief history of computer-assisted diagnosis, including challenges and future directions. Some ideas presented in this article on computer-assisted diagnostic decision support systems (CDDSS) derive from prior work by the author and his colleagues (see list in Acknowledgments) on the INTERNIST-1 and QMR projects. References indicate the original sources of many of these ideas.

A universal model of diagnostic reasoning

Clinical judgment is a critical aspect of physician performance in medicine. It is essential in the formulation of a diagnosis and key to the effective and safe management of patients. Yet, the overall diagnostic error rate remains unacceptably high. In more than four decades of research, a variety of approaches have been taken, but a consensus approach toward diagnostic decision making has not emerged. In the last 20 years, important gains have been made in psychological research on human judgment. Dual-process theory has emerged as the predominant approach, positing two systems of decision making, System 1 (heuristic, intuitive) and System 2 (systematic, analytical). The author proposes a schematic model that uses the theory to develop a universal approach toward clinical decision making. Properties of the model explain many of the observed characteristics of physicians' performance. Yet the author cautions that not all medical reasoning and decision making falls neatly into one or the other of the model's systems, even though they provide a basic framework incorporating the recognized diverse approaches. He also emphasizes the complexity of decision making in actual clinical situations and the urgent need for more research to help clinicians gain additional insight and understanding regarding their decision making.

Grand challenges in clinical decision support

There is a pressing need for high-quality, effective means of designing, developing, presenting, implementing, evaluating, and maintaining all types of clinical decision support capabilities for clinicians, patients and consumers. Using an iterative, consensus-building process we identified a rank-ordered list of the top 10 grand challenges in clinical decision support. This list was created to educate and inspire researchers, developers, funders, and policy-makers. The list of challenges in order of importance that they be solved if patients and organizations are to begin realizing the fullest benefits possible of these systems consists of: improve the human-computer interface; disseminate best practices in CDS design, development, and implementation; summarize patient-level information; prioritize and filter recommendations to the user; create an architecture for sharing executable CDS modules and services; combine recommendations for patients with co-morbidities; prioritize CDS content development and implementation; create internet-accessible clinical decision support repositories; use freetext information to drive clinical decision support; mine large clinical databases to create new CDS. Identification of solutions to these challenges is critical if clinical decision support is to achieve its potential and improve the quality, safety and efficiency of healthcare.

Diagnostic omission errors in acute paediatric practice: impact of a reminder system on decision-making

Background

Diagnostic error is a significant problem in specialities characterised by diagnostic uncertainty such as primary care, emergency medicine and paediatrics. Despite wide-spread availability, computerised aids have not been shown to significantly improve diagnostic decision-making in a real world environment, mainly due to the need for prolonged system consultation. In this study performed in the clinical environment, we used a Web-based diagnostic reminder system that provided rapid advice with free text data entry to examine its impact on clinicians' decisions in an acute paediatric setting during assessments characterised by diagnostic uncertainty.

Methods

Junior doctors working over a 5-month period at four paediatric ambulatory units consulted the Web-based diagnostic aid when they felt the need for diagnostic assistance. Subjects recorded their clinical decisions for patients (differential diagnosis, test-ordering and treatment) before and after system consultation. An expert panel of four paediatric consultants independently suggested clinically significant decisions indicating an appropriate and 'safe' assessment. The primary outcome measure was change in the proportion of 'unsafe' workups by subjects during patient assessment. A more sensitive evaluation of impact was performed using specific validated quality scores. Adverse effects of consultation on decision-making, as well as the additional time spent on system use were examined.

Results

Subjects attempted to access the diagnostic aid on 595 occasions during the study period (8.6% of all medical assessments); subjects examined diagnostic advice only in 177 episodes (30%). Senior House Officers at hospitals with greater number of available computer workstations in the clinical area were most likely to consult the system, especially out of working hours. Diagnostic workups construed as 'unsafe' occurred in 47/104 cases (45.2%); this reduced to 32.7% following system consultation (McNemar test, p < 0.001). Subjects' mean 'unsafe' workups per case decreased from 0.49 to 0.32 (p < 0.001). System advice prompted the clinician to consider the 'correct' diagnosis (established at discharge) during initial assessment in 3/104 patients. Median usage time was 1 min 38 sec (IQR 50 sec – 3 min 21 sec). Despite a modest increase in the number of diagnostic possibilities entertained by the clinician, no adverse effects were demonstrable on patient management following system use. Numerous technical barriers prevented subjects from accessing the diagnostic aid in the majority of eligible patients in whom they sought diagnostic assistance.

Conclusion

We have shown that junior doctors used a Web-based diagnostic reminder system during acute paediatric assessments to significantly improve the quality of their diagnostic workup and reduce diagnostic omission errors. These benefits were achieved without any adverse effects on patient management following a quick consultation.

Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasi-experimental study

Background

Computerized decision support systems (DSS) have mainly focused on improving clinicians' diagnostic accuracy in unusual and challenging cases. However, since diagnostic omission errors may predominantly result from incomplete workup in routine clinical practice, the provision of appropriate patient- and context-specific reminders may result in greater impact on patient safety. In this experimental study, a mix of easy and difficult simulated cases were used to assess the impact of a novel diagnostic reminder system (ISABEL) on the quality of clinical decisions made by various grades of clinicians during acute assessment.

Methods

Subjects of different grades (consultants, registrars, senior house officers and medical students), assessed a balanced set of 24 simulated cases on a trial website. Subjects recorded their clinical decisions for the cases (differential diagnosis, test-ordering and treatment), before and after system consultation. A panel of two pediatric consultants independently provided gold standard responses for each case, against which subjects' quality of decisions was measured. The primary outcome measure was change in the count of diagnostic errors of omission (DEO). A more sensitive assessment of the system's impact was achieved using specific quality scores; additional consultation time resulting from DSS use was also calculated.

Results

76 subjects (18 consultants, 24 registrars, 19 senior house officers and 15 students) completed a total of 751 case episodes. The mean count of DEO fell from 5.5 to 5.0 across all subjects (repeated measures ANOVA, p < 0.001); no significant interaction was seen with subject grade. Mean diagnostic quality score increased after system consultation (0.044; 95% confidence interval 0.032, 0.054). ISABEL reminded subjects to consider at least one clinically important diagnosis in 1 in 8 case episodes, and prompted them to order an important test in 1 in 10 case episodes. Median extra time taken for DSS consultation was 1 min (IQR: 30 sec to 2 min).

Conclusion

The provision of patient- and context-specific reminders has the potential to reduce diagnostic omissions across all subject grades for a range of cases. This study suggests a promising role for the use of future reminder-based DSS in the reduction of diagnostic error.

Computer-assisted diagnosis system in digestive endoscopy

The purpose of this paper is to present an intelligent atlas of indexed endoscopic lesions that could be used in computer-assisted diagnosis as reference data. The development of such a system requires a mix of medical and engineering skills for analyzing and reproducing the cognitive processes that underlie the medical decision-making process. The analysis of both endoscopists experience and endoscopic terminologies developed by professional associations shows that diagnostic reasoning in digestive endoscopy uses a scene-object approach. The objects correspond to the endoscopic findings and the medical context of examination and the scene to the endoscopic diagnosis. According to expert assessment, the classes of endoscopic findings and diagnoses, their primitive characteristics (or indices), and their relationships have been listed. Each class describes an endoscopic finding or diagnosis in an intensive way. The retrieval method is based on a similarity metric that estimates the membership value of the case under investigation and the prototype of the class. A simulation test with randomized objects demonstrates a good classification of endoscopic findings. The correct class is the unique response in 68% of the tested objects, the first of multiple responses in 28%. Four descriptors are shown to be of major importance in the classification algorithm: anatomic location, shape, color, and relief. At the present time, the application database contains approximately 150 endoscopic images and is accessible via Internet. Experiments are in progress with endoscopists for the validation of the system and for the understanding of the similarity between images. The next step will integrate the system in a learning tool for junior endoscopists.

Evaluation of two different models of semi-automatic knowledge acquisition for the medical consultant system CADIAG-II/RHEUMA

As part of a plan to promote semi-automatic knowledge acquisition for the medical consultant system CADIAG-II/RHEUMA, this study sought to explore and cope with the variability of results that may be anticipated when performing knowledge acquisition with patient data from different patient settings. Patient data were drawn both from a published study for the classification of rheumatoid arthritis (RA) and from a large database of rheumatological patient charts developed for the CADIAG-II/RHEUMA system. An analysis of the relationships between RA and selected CADIAG-II/RHEUMA symptoms was done using two models. In one of them, we controlled for the differences in baseline frequencies of symptoms and diseases in the two study populations as an important factor influencing the results of the calculations. Other factors that were identified included inconsistent definitions of symptoms and diseases, and the different composition of study groups in the two study populations. By eliminating differences in baseline frequencies as the most important bias, the results obtained from the two different knowledge sources became more consistent. All remaining inconsistencies and uncertainties about the contribution and relative importance of the factors were formalized using fuzzy intervals.

The basis for using the Internet to support the information needs of primary care

Synthesizing the state of the art from the published literature, this review assesses the basis for employing the Internet to support the information needs of primary care. The authors survey what has been published about the information needs of clinical practice, including primary care, and discuss currently available information resources potentially relevant to primary care. Potential methods of linking information needs with appropriate information resources are described in the context of previous classifications of clinical information needs. Also described is the role that existing terminology mapping systems, such as the National Library of Medicine's Unified Medical Language System, may play in representing and linking information needs to answers.

Performance of four computer-based diagnostic systems

BACKGROUND

Computer-based diagnostic systems are available commercially, but there has been limited evaluation of their performance. We assessed the diagnostic capabilities of four internal medicine diagnostic systems: Dxplain, Iliad, Meditel, and QMR.

METHODS

Ten expert clinicians created a set of 105 diagnostically challenging clinical case summaries involving actual patients. Clinical data were entered into each program with the vocabulary provided by the program's developer. Each of the systems produced a ranked list of possible diagnoses for each patient, as did the group of experts. We calculated scores on several performance measures for each computer program.

RESULTS

No single computer program scored better than the others on all performance measures. Among all cases and all programs, the proportion of correct diagnoses ranged from 0.52 to 0.71, and the mean proportion of relevant diagnoses ranged from 0.19 to 0.37. On average, less than half the diagnoses on the experts' original list of reasonable diagnoses were suggested by any of the programs. However, each program suggested an average of approximately two additional diagnoses per case that the experts found relevant but had not originally considered.

CONCLUSIONS

The results provide a profile of the strengths and limitations of these computer programs. The programs should be used by physicians who can identify and use the relevant information and ignore the irrelevant information that can be produced.

Computer-assisted diagnosis of rheumatic disorders

A review of the literature regarding computer-assisted diagnosis of rheumatic diseases is presented. After a general outline of the history and goals of computer programs intended to support physicians in the diagnostic process, 14 systems or projects are described. The scope of seven of these is general internal medicine, and the other seven are intended exclusively for rheumatic problems. The majority of these systems are prototypes. To date, none of them is widely used by physicians. Preliminary evaluation studies and/or independent reviews have been reported for all of the systems. The need for further evaluation studies is recognized, and strategies to carry these out are outlined. Furthermore, the potential usefulness for patient care and education is discussed. It is concluded that a new and interesting field is being developed that deserves more attention among rheumatologists.

The effect of incomplete knowledge on the diagnoses of a computer consultant system

The knowledge bases (KBs) of diagnostic decision support systems are often incomplete, and gaps in the KB could potentially lead systems to reach diagnoses that are implausible to physicians. To investigate this possibility we studied Iliad (Version 2.01), a computer consultant system that generates differential diagnosis across the domain of internal medicine. Data from the history, physical examination, and laboratory findings of 50 grand-rounds cases were entered into Iliad by a computer consultant aware of the diagnosis but blinded to its presence or absence in Iliad's KB. Two experienced internists were asked to diagnose these cases before and after seeing the results of the computer consultation, and to assess the plausibility of the computer's diagnoses. Twenty-eight of the 50 cases (56.0%) were diseases contained in Iliad's KB. After seeing Iliad's diagnoses for cases in the KB, physicians assigned to their correct diagnoses a higher mean ranked position (1.5 versus 2.0, p less than 0.008) and a higher mean probability (84.0% versus 77.6%, p less than 0.008) compared with their pre-Iliad values, whereas for cases not in the KB, mean position and probability for correct diagnoses did not change. Physician diagnostic accuracy did not change after consultation on cases included or not included in the KB. After adjusting for case difficulty, mean plausibility of Iliad's diagnoses was judged significantly higher (on a seven-point scale) for cases in the KB than for cases not in the KB (4.2 versus 3.2, p less than 0.02).(ABSTRACT TRUNCATED AT 250 WORDS)

Computer-assisted diagnosis of acute azotaemia: diagnostic strategy and diagnostic criteria

This report describes the diagnostic strategy, cost-effectiveness and diagnostic accuracy of a rule-based, backward chaining diagnostic expert system designed to assist in diagnosing the cause(s) of acute azotaemia. The diagnostic strategy of the expert system seems to be cost-effective compared with that of a renal unit and the diagnostic criteria of the expert system accurately imitate the diagnoses of the renal unit. It is suggested that the expert system is diagnostically reliable and may reduce cost in the diagnostic work-up of patients with acute azotaemia.