Evaluation of a clinical decision support system for rare diseases: a qualitative study

Gynaecological Artificial Intelligence Diagnostics (GAID) GAID and Its Performance as a Tool for the Specialist Doctor

BACKGROUND

Human-centric artificial intelligence (HCAI) aims to provide support systems that can act as peer companions to an expert in a specific domain, by simulating their way of thinking and decision-making in solving real-life problems. The gynaecological artificial intelligence diagnostics (GAID) assistant is such a system. Based on artificial intelligence (AI) argumentation technology, it was developed to incorporate, as much as possible, a complete representation of the medical knowledge in gynaecology and to become a real-life tool that will practically enhance the quality of healthcare services and reduce stress for the clinician. Our study aimed to evaluate GAIDS' efficacy and accuracy in assisting the working expert gynaecologist during day-to-day clinical practice.

METHODS

Knowledge-based systems utilize a knowledge base (theory) which holds evidence-based rules ("IF-THEN" statements) that are used to prove whether a conclusion (such as a disease, medication or treatment) is possible or not, given a set of input data. This approach uses argumentation frameworks, where rules act as claims that support a specific decision (arguments) and argue for its dominance over others. The result is a set of admissible arguments which support the final decision and explain its cause.

RESULTS

Based on seven different subcategories of gynaecological presentations-bleeding, endocrinology, cancer, pelvic pain, urogynaecology, sexually transmitted infections and vulva pathology in fifty patients-GAID demonstrates an average overall closeness accuracy of zero point eighty-seven. Since the system provides explanations for supporting a diagnosis against other possible diseases, this evaluation process further allowed for a learning process of modular improvement in the system of the diagnostic discrepancies between the system and the specialist.

CONCLUSIONS

GAID successfully demonstrates an average accuracy of zero point eighty-seven when measuring the closeness of the system's diagnosis to that of the senior consultant. The system further provides meaningful and helpful explanations for its diagnoses that can help clinicians to develop an increasing level of trust towards the system. It also provides a practical database, which can be used as a structured history-taking assistant and a friendly, patient record-keeper, while improving precision by providing a full list of differential diagnoses. Importantly, the design and implementation of the system facilitates its continuous development with a set methodology that allows minimal revision of the system in the face of new information. Further large-scale studies are required to evaluate GAID more thoroughly and to identify its limiting boundaries.

Web-based eHealth Clinical Decision Support System as a tool for the treat-to-target management of patients with systemic lupus erythematosus: development and initial usability evaluation

BACKGROUND

Treat-to-target (T2T) is a therapeutic strategy currently being studied for its application in systemic lupus erythematosus (SLE). Patients and rheumatologists have little support in making the best treatment decision in the context of a T2T strategy, thus, the use of information technology for systematically processing data and supporting information and knowledge may improve routine decision-making practices, helping to deliver value-based care.

OBJECTIVE

To design and develop an online Clinical Decision Support Systems (CDSS) tool "SLE-T2T", and test its usability for the implementation of a T2T strategy in the management of patients with SLE.

METHODS

A prototype of a CDSS was conceived as a web-based application with the task of generating appropriate treatment advice based on entered patients' data. Once developed, a System Usability Score (SUS) questionnaire was implemented to test whether the eHealth tool was user-friendly, comprehensible, easy-to-deliver and workflow-oriented. Data from the participants' comments were synthesised, and the elements in need for improvement were identified.

RESULTS

The beta version web-based system was developed based on the interim usability and acceptance evaluation. 7 participants completed the SUS survey. The median SUS score of SLE-T2T was 79 (scale 0 to 100), categorising the application as 'good' and indicating the need for minor improvements to the design.

CONCLUSIONS

SLE-T2T is the first eHealth tool to be designed for the management of SLE patients in a T2T context. The SUS score and unstructured feedback showed high acceptance of this digital instrument for its future use in a clinical trial.

Requirements analysis for an AI-based clinical decision support system for general practitioners: a user-centered design process

BACKGROUND

As the first point of contact for patients with health issues, general practitioners (GPs) are frequently confronted with patients presenting with non-specific symptoms of unclear origin. This can result in delayed, prolonged or false diagnoses. To accelerate and improve the diagnosis of diseases, clinical decision support systems would appear to be an appropriate tool. The objective of the project 'Smart physician portal for patients with unclear disease' (SATURN) is to employ a user-centered design process based on the requirements analysis presented in this paper to develop an artificial Intelligence (AI)-based diagnosis support system that specifically addresses the needs of German GPs.

METHODS

Requirements analysis for a GP-specific diagnosis support system was conducted in an iterative process with five GPs. First, interviews were conducted to analyze current workflows and the use of digital applications in cases of diagnostic uncertainty (as-is situation). Second, we focused on collecting and prioritizing tasks to be performed by an ideal smart physician portal (to-be situation) in a workshop. We then developed a task model with corresponding user requirements.

RESULTS

Numerous GP-specific user requirements were identified concerning the tasks and subtasks: performing data entry (open system, enter patient data), reviewing results (receiving and evaluating results), discussing results (with patients and colleagues), scheduling further diagnostic procedures, referring to specialists (select, contact, make appointments), and case closure. Suggested features particularly concerned the process of screening and assessing results: e.g., the system should focus more on atypical patterns of common diseases than on rare diseases only, display probabilities of differential diagnoses, ensure sources and results are transparent, and mark diagnoses that have already been ruled out. Moreover, establishing a means of using the platform to communicate with colleagues and transferring patient data directly from electronic patient records to the system was strongly recommended.

CONCLUSIONS

Essential user requirements to be considered in the development and design of a diagnosis system for primary care could be derived from the analysis. They form the basis for mockup-development and system engineering.

Ten Topics to Get Started in Medical Informatics Research

The vast and heterogeneous data being constantly generated in clinics can provide great wealth for patients and research alike. The quickly evolving field of medical informatics research has contributed numerous concepts, algorithms, and standards to facilitate this development. However, these difficult relationships, complex terminologies, and multiple implementations can present obstacles for people who want to get active in the field. With a particular focus on medical informatics research conducted in Germany, we present in our Viewpoint a set of 10 important topics to improve the overall interdisciplinary communication between different stakeholders (eg, physicians, computational experts, experimentalists, students, patient representatives). This may lower the barriers to entry and offer a starting point for collaborations at different levels. The suggested topics are briefly introduced, then general best practice guidance is given, and further resources for in-depth reading or hands-on tutorials are recommended. In addition, the topics are set to cover current aspects and open research gaps of the medical informatics domain, including data regulations and concepts; data harmonization and processing; and data evaluation, visualization, and dissemination. In addition, we give an example on how these topics can be integrated in a medical informatics curriculum for higher education. By recognizing these topics, readers will be able to (1) set clinical and research data into the context of medical informatics, understanding what is possible to achieve with data or how data should be handled in terms of data privacy and storage; (2) distinguish current interoperability standards and obtain first insights into the processes leading to effective data transfer and analysis; and (3) value the use of newly developed technical approaches to utilize the full potential of clinical data.

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system

Background

For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.

Objective

The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS.

Methods

The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered.

Results

The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment.

Conclusion

The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12553-023-00763-1.

Design, implementation, and evaluation of the computer-aided clinical decision support system based on learning-to-rank: collaboration between physicians and machine learning in the differential diagnosis process

BACKGROUND

We are researching, developing, and publishing the clinical decision support system based on learning-to-rank. The main objectives are (1) To support for differential diagnoses performed by internists and general practitioners and (2) To prevent diagnostic errors made by physicians. The main features are that "A physician inputs a patient's symptoms, findings, and test results to the system, and the system outputs a ranking list of possible diseases".

METHOD

The software libraries for machine learning and artificial intelligence are TensorFlow and TensorFlow Ranking. The prediction algorithm is Learning-to-Rank with the listwise approach. The ranking metric is normalized discounted cumulative gain (NDCG). The loss functions are Approximate NDCG (A-NDCG). We evaluated the machine learning performance on k-fold cross-validation. We evaluated the differential diagnosis performance with validated cases.

RESULTS

The machine learning performance of our system was much higher than that of the conventional system. The differential diagnosis performance of our system was much higher than that of the conventional system. We have shown that the clinical decision support system prevents physicians' diagnostic errors due to confirmation bias.

CONCLUSIONS

We have demonstrated that the clinical decision support system is useful for supporting differential diagnoses and preventing diagnostic errors. We propose that differential diagnosis by physicians and learning-to-rank by machine has a high affinity. We found that information retrieval and clinical decision support systems have much in common (Target data, learning-to-rank, etc.). We propose that Clinical Decision Support Systems have the potential to support: (1) recall of rare diseases, (2) differential diagnoses for difficult-to-diagnoses cases, and (3) prevention of diagnostic errors. Our system can potentially evolve into an explainable clinical decision support system.

Usability of the IDDEAS prototype in child and adolescent mental health services: A qualitative study for clinical decision support system development

INTRODUCTION

Child and adolescent mental health services (CAMHS) clinical decision support system (CDSS) provides clinicians with real-time support as they assess and treat patients. CDSS can integrate diverse clinical data for identifying child and adolescent mental health needs earlier and more comprehensively. Individualized Digital Decision Assist System (IDDEAS) has the potential to improve quality of care with enhanced efficiency and effectiveness.

METHODS

We examined IDDEAS usability and functionality in a prototype for attention deficit hyperactivity disorder (ADHD), using a user-centered design process and qualitative methods with child and adolescent psychiatrists and clinical psychologists. Participants were recruited from Norwegian CAMHS and were randomly assigned patient case vignettes for clinical evaluation, with and without IDDEAS. Semi-structured interviews were conducted as one part of testing the usability of the prototype following a five-question interview guide. All interviews were recorded, transcribed, and analyzed following qualitative content analysis.

RESULTS

Participants were the first 20 individuals from the larger IDDEAS prototype usability study. Seven participants explicitly stated a need for integration with the patient electronic health record system. Three participants commended the step-by-step guidance as potentially helpful for novice clinicians. One participant did not like the aesthetics of the IDDEAS at this stage. All participants were pleased about the display of the patient information along with guidelines and suggested that wider guideline coverage will make IDDEAS much more useful. Overall, participants emphasized the importance of maintaining the clinician as the decision-maker in the clinical process, and the overall potential utility of IDDEAS within Norwegian CAMHS.

CONCLUSION

Child and adolescent mental health services psychiatrists and psychologists expressed strong support for the IDDEAS clinical decision support system if better integrated in daily workflow. Further usability assessments and identification of additional IDDEAS requirements are necessary. A fully functioning, integrated version of IDDEAS has the potential to be an important support for clinicians in the early identification of risks for youth mental disorders and contribute to improved assessment and treatment of children and adolescents.

A visual analytic tool, VIADS, to assist the hypothesis generation process in clinical research—A usability study using mixed methods (Preprint)

BACKGROUND

Visualization can be a powerful tool for comprehending datasets, especially when they can be represented via hierarchical structures. Enhanced comprehension can facilitate the development of scientific hypotheses. However, the inclusion of excessive data can make a visualization overwhelming.

OBJECTIVE

We developed a Visual Interactive Analytic tool for filtering and summarizing large health Data Sets (VIADS) coded with hierarchical terminologies. In this study, we evaluated the usability of VIADS for visualizing data sets of patient diagnoses and procedures coded in the International Classification of Diseases, ninth revisions, clinical modification (ICD-9-CM).

METHODS

We used mixed methods in the study. A group of 12 clinical researchers participated in the generation of data-driven hypotheses using the same datasets and time frame (a 1-hour training session and a 2-hour study session), utilizing VIADS via the think-aloud protocol. The audio and screen activities were recorded remotely. A modified version of the System Usability Scale (SUS) survey and a brief survey with open-ended questions were administered after the study to assess the usability of VIADS and verify their intense usage experience of VIADS.

RESULTS

The range of SUS scores was 37.5 - 87.5. The mean SUS score for VIADS was 71.88 (out of a possible 100, standard deviation: 14.62 ), and the median SUS was 75. The participants unanimously agreed that VIADS offers new perspectives on data sets (100%), while 75% agreed that VIADS facilitates understanding, presentation, and interpretation of underlying datasets. The comments on the utility of VIADS were positive and aligned well with the design objectives of VIADS. The answers to the open-ended questions in the modified SUS provided specific suggestions regarding potential improvements in VIADS, and identified problems in usability were used to update the tool.

CONCLUSIONS

This usability study demonstrates that VIADS is a usable tool for analyzing secondary datasets with good average usability, SUS score, and favorable utility. Currently, VIADS accepts datasets with hierarchical codes and their corresponding frequencies. Consequently, only specific types of use cases are supported by the analytical results. Participants agreed, however, that VIADS provides new perspectives on datasets and is relatively easy to use. The functionalities mostly appreciated by participants were VIADS' ability to filter, summarize, compare, and visualize data.

CLINICALTRIAL

INTERNATIONAL REGISTERED REPORT

RR2-10.2196/39414.

Evaluation of an Electronic Medical Record Module for Nursing Documentation in Paediatric Palliative Care: Involvement of Nurses with a Think-Aloud Approach

Background: Paediatric palliative care (PPC) is a noncurative approach to the care of children and adolescents with life-limiting and life-threatening illnesses. Electronic medical records (EMRs) play an important role in documenting such complex processes. Despite their benefits, they can introduce unintended consequences if future users are not involved in their development. Aim: The aim of this study was to evaluate the acceptance of a novel module for nursing documentation by nurses working in the context of PPC. Methods: An observational study employing concurrent think-aloud and semi-structured qualitative interviews were conducted with 11 nurses working in PPC. Based on the main determinants of the unified theory of acceptance and use of technology (UTAUT), data were analysed using qualitative content analysis. Results: The main determinants of UTAUT were found to potentially influence acceptance of the novel module. Participants perceived the module to be self-explanatory and intuitive. Some adaptations, such as the reduction of fragmentation in the display, the optimization of confusing mouseover fields, and the use of familiar nursing terminology, are reasonable ways of increasing software adoption. Conclusions: After adaptation of the modules based on the results, further evaluation with the participation of future users is required.

Participatory Design of a Medication Module in an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Approach with Nurses and Physicians

BACKGROUND

Electronic medical records (EMRs) play a key role in improving documentation and quality of care in paediatric palliative care (PPC). Inadequate EMR design can cause incorrect prescription and administration of medications. Due to the fact of complex diseases and the resulting high level of medical complexity, patients in PPC are vulnerable to medication errors. Consequently, involving users in the development process is important. Therefore, the aim of this study was to evaluate the acceptance of a medication module from the perspective of potential users in PPC and to involve them in the development process.

METHODS

A qualitative observational study was conducted with 10 nurses and four physicians using a concurrent think-aloud protocol and semi-structured qualitative interviews. A qualitative content analysis was applied based on a unified theory of acceptance and use of technology.

RESULTS

Requirements from the user's perspective could be identified as possible influences on acceptance and actual use. Requirements were grouped into the categories "performance expectancies" and "effort expectancies".

CONCLUSIONS

The results serve as a basis for further development. Attention should be given to the reduction of display fragmentation, as it decreases cognitive load. Further approaches to evaluation should be taken.

[What is confirmed in the diagnostics of autoinflammatory fever diseases?]

Periodic fever syndromes (PFS) are a group of rare autoinflammatory diseases, which are characterized by disorders of the innate immune reaction and life-long recurrent episodes of inflammatory symptoms. This article describes the diagnostic approach. In addition to the patient medical history, physical examination and laboratory determinations, gene tests are becoming increasingly more important. The panel diagnostics using high throughput sequencing or next generation sequencing (NGS) is the method of choice for the detection of a genetic cause of PFS. This article discusses the diagnostic decision support systems (DDSS) that can play a future role in the diagnosis of rare diseases, especially those with complex patterns of symptoms.

Participatory Design of an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Study with Nurses and Physicians

Background: Electronic medical records (EMRs) offer a promising approach to mapping and documenting the complex information gathered in paediatric palliative care (PPC). However, if they are not well developed, poorly implemented EMRs have unintended consequences that may cause harm to patients. One approach to preventing such harm is the involvement of users in the participatory design to ensure user acceptance and patient safety. Therefore, the aim of this study is to evaluate the acceptance of a novel patient chart module (PCM) as part of an EMR from the perspective of potential users in PPC and to involve these professionals in the design process. Methods: A qualitative observational study with N = 16 PPC professionals (n = 10 nurses, n = 6 physicians) was conducted, including concurrent think aloud (CTA) and semi-structured interviews. A structured content analysis based on the Unified Theory of Acceptance and Use of Technology was applied. Results: The results can be summarized in terms of general observations, performance expectancy, effort expectancy and facilitating conditions, all of which are likely to have a positive influence on acceptance of the PCM from the user perspective in the context of PPC. Conclusions: The involvement of users in the development of EMRs is important for meeting the requirements in PPC. Further software adaptations are necessary to implement these requirements.

[se-atlas.de-Medical care atlas for people with rare diseases]

Eine Erkrankung zählt in der Europäischen Union zu den Seltenen Erkrankungen (SE), wenn diese nicht mehr als 5 von 10.000 Menschen betrifft. Derzeit existiert mit mehr als 6000 SE eine sowohl große als auch heterogene Menge an unterschiedlichen Krankheitsbilder, die in ihrer Symptomatik komplex, vielschichtig und damit im medizinischen Alltag schwierig einzuordnen sind. Dies erschwert Diagnosefindung und Behandlung sowie das Auffinden eines passenden Ansprechpartners, da es nur wenige Experten für jede einzelne SE gibt. Der medizinische Versorgungsatlas für Seltene Erkrankungen www.se-atlas.de ermöglicht anhand von Erkrankungsnamen die Suche nach Versorgungseinrichtungen und Selbsthilfeorganisationen zu bestimmten SE und stellt die Suchergebnisse geografisch dar. Ebenso gibt er einen Überblick über alle deutschen Zentren für SE, die eine Anlaufstelle für betroffene Personen mit unklarer Diagnose darstellen. Der se-atlas dient als Kompass durch die heterogene Menge an Informationen über Versorgungseinrichtungen für SE und stellt niederschwellig Informationen für eine breite Nutzergruppe von Betroffenen bis hin zu Mitgliedern des medizinischen Versorgungsteams bereit.