RadTranslate: An Artificial Intelligence-Powered Intervention for Urgent Imaging to Enhance Care Equity for Patients With Limited English Proficiency During the COVID-19 Pandemic

Social Determinants of Health Framework to Identify and Reduce Barriers to Imaging in Marginalized Communities

Social determinants of health (SDOH) are conditions influencing individuals' health based on their environment of birth, living, working, and aging. Addressing SDOH is crucial for promoting health equity and reducing health outcome disparities. For conditions such as stroke and cancer screening where imaging is central to diagnosis and management, access to high-quality medical imaging is necessary. This article applies a previously described structural framework characterizing the impact of SDOH on patients who require imaging for their clinical indications. SDOH factors can be broadly categorized into five sectors: economic stability, education access and quality, neighborhood and built environment, social and community context, and health care access and quality. As patients navigate the health care system, they experience barriers at each step, which are significantly influenced by SDOH factors. Marginalized communities are prone to disparities due to the inability to complete the required diagnostic or screening imaging work-up. This article highlights SDOH that disproportionately affect marginalized communities, using stroke and cancer as examples of disease processes where imaging is needed for care. Potential strategies to mitigate these disparities include dedicating resources for clinical care coordinators, transportation, language assistance, and financial hardship subsidies. Last, various national and international health initiatives are tackling SDOH and fostering health equity.

A Novel Design-Thinking, Hospital Innovation Core Certificate Curriculum for Radiologists and Trainees: Creation, Implementation, and Multiyear Results

RATIONALE AND OBJECTIVES

Innovation is a crucial skill for physicians and researchers, yet traditional medical education does not provide instruction or experience to cultivate an innovative mindset. This study evaluates the effectiveness of a novel course implemented in an academic radiology department training program over a 5-year period designed to educate future radiologists on the fundamentals of medical innovation.

MATERIALS AND METHODS

A pre- and post-course survey and examination were administered to residents who participated in the innovation course (MESH Core) from 2018 to 2022. Respondents were first evaluated on their subjective comfort level, understanding, and beliefs on innovation-related topics using a 5-point Likert-scale survey. Respondents were also administered a 21-question multiple-choice exam to test their objective knowledge of innovation-related topics.

RESULTS

Thirty-eight residents participated in the survey (response rate 95%). Resident understanding, comfort and belief regarding innovation-related topics improved significantly (P < .0001) on all nine Likert-scale questions after the course. After the course, a significant majority of residents either agreed or strongly agreed that technological innovation should be a core competency for the residency curriculum, and that a workshop to prototype their ideas would be beneficial. Performance on the course exam showed significant improvement (48% vs 86%, P < .0001). The overall course experience was rated 5 out of 5 by all participants.

CONCLUSION

MESH Core demonstrates long-term success in educating future radiologists on the basic concepts of medical technological innovation. Years later, residents used the knowledge and experience gained from MESH Core to successfully pursue their own inventions and innovative projects. This innovation model may serve as an approach for other institutions to implement training in this domain.

Empathy and Equity: Key Considerations for Large Language Model Adoption in Health Care

The growing presence of large language models (LLMs) in health care applications holds significant promise for innovative advancements in patient care. However, concerns about ethical implications and potential biases have been raised by various stakeholders. Here, we evaluate the ethics of LLMs in medicine along 2 key axes: empathy and equity. We outline the importance of these factors in novel models of care and develop frameworks for addressing these alongside LLM deployment.

A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research

Since its release, ChatGPT has taken the world by storm with its utilization in various fields of life. This review's main goal was to offer a thorough and fact-based evaluation of ChatGPT's potential as a tool for medical and dental research, which could direct subsequent research and influence clinical practices.

METHODS

Different online databases were scoured for relevant articles that were in accordance with the study objectives. A team of reviewers was assembled to devise a proper methodological framework for inclusion of articles and meta-analysis.

RESULTS

11 descriptive studies were considered for this review that evaluated the accuracy of ChatGPT in answering medical queries related to different domains such as systematic reviews, cancer, liver diseases, diagnostic imaging, education, and COVID-19 vaccination. The studies reported different accuracy ranges, from 18.3 % to 100 %, across various datasets and specialties. The meta-analysis showed an odds ratio (OR) of 2.25 and a relative risk (RR) of 1.47 with a 95 % confidence interval (CI), indicating that the accuracy of ChatGPT in providing correct responses was significantly higher compared to the total responses for queries. However, significant heterogeneity was present among the studies, suggesting considerable variability in the effect sizes across the included studies.

CONCLUSION

The observations indicate that ChatGPT has the ability to provide appropriate solutions to questions in the medical and dentistry areas, but researchers and doctors should cautiously assess its responses because they might not always be dependable. Overall, the importance of this study rests in shedding light on ChatGPT's accuracy in the medical and dentistry fields and emphasizing the need for additional investigation to enhance its performance. © 2017 Elsevier Inc. All rights reserved.

Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot

OBJECTIVE

Despite rising popularity and performance, studies evaluating the use of large language models for clinical decision support are lacking. Here, we evaluate ChatGPT (Generative Pre-trained Transformer)-3.5 and GPT-4's (OpenAI, San Francisco, California) capacity for clinical decision support in radiology via the identification of appropriate imaging services for two important clinical presentations: breast cancer screening and breast pain.

METHODS

We compared ChatGPT's responses to the ACR Appropriateness Criteria for breast pain and breast cancer screening. Our prompt formats included an open-ended (OE) and a select all that apply (SATA) format. Scoring criteria evaluated whether proposed imaging modalities were in accordance with ACR guidelines. Three replicate entries were conducted for each prompt, and the average of these was used to determine final scores.

RESULTS

Both ChatGPT-3.5 and ChatGPT-4 achieved an average OE score of 1.830 (out of 2) for breast cancer screening prompts. ChatGPT-3.5 achieved a SATA average percentage correct of 88.9%, compared with ChatGPT-4's average percentage correct of 98.4% for breast cancer screening prompts. For breast pain, ChatGPT-3.5 achieved an average OE score of 1.125 (out of 2) and a SATA average percentage correct of 58.3%, as compared with an average OE score of 1.666 (out of 2) and a SATA average percentage correct of 77.7%.

DISCUSSION

Our results demonstrate the eventual feasibility of using large language models like ChatGPT for radiologic decision making, with the potential to improve clinical workflow and responsible use of radiology services. More use cases and greater accuracy are necessary to evaluate and implement such tools.

Imaging Artificial Intelligence: A Framework for Radiologists to Address Health Equity, From the AJR Special Series on DEI

Artificial intelligence (AI) holds promise for helping patients access new and individualized health care pathways while increasing efficiencies for health care practitioners. Radiology has been at the forefront of this technology in medicine; many radiology practices are implementing and trialing AI-focused products. AI also holds great promise for reducing health disparities and promoting health equity. Radiology is ideally positioned to help reduce disparities given its central and critical role in patient care. The purposes of this article are to discuss the potential benefits and pitfalls of deploying AI algorithms in radiology, specifically highlighting the impact of AI on health equity; to explore ways to mitigate drivers of inequity; and to enhance pathways for creating better health care for all individuals, centering on a practical framework that helps radiologists address health equity during deployment of new tools.

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study

BACKGROUND

Large language model (LLM)-based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated.

OBJECTIVE

This study aimed to evaluate ChatGPT's capacity for ongoing clinical decision support via its performance on standardized clinical vignettes.

METHODS

We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared its accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. Accuracy was measured by the proportion of correct responses to the questions posed within the clinical vignettes tested, as calculated by human scorers. We further conducted linear regression to assess the contributing factors toward ChatGPT's performance on clinical tasks.

RESULTS

ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=-15.8%; P<.001) and clinical management (β=-7.4%; P=.02) question types.

CONCLUSIONS

ChatGPT achieves impressive accuracy in clinical decision-making, with increasing strength as it gains more clinical information at its disposal. In particular, ChatGPT demonstrates the greatest accuracy in tasks of final diagnosis as compared to initial diagnosis. Limitations include possible model hallucinations and the unclear composition of ChatGPT's training data set.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

Justice and Innovation in Radiology

Imaging is a central determinant of health outcomes, and radiologic disparities can cascade throughout a patient's illness course. Innovative efforts in radiology are constant, but innovation that is driven by short-term profit-making incentives without explicit regard for principles of justice can lead to exclusion of the vulnerable from potential benefits and widening of inequities. Accordingly, we must consider the ways in which the field of radiology can shape innovative efforts to ensure that innovation ameliorates injustice instead of exacerbating it. The authors propose a distinction between approaches to innovation that prioritize justice and those that do not. The authors argue that the field's institutional incentives should be adjusted to prioritize forms of innovation that are likely to ameliorate imaging inequities, and they provide examples of initial steps that can be taken to make these adjustments. The authors propose the term justice-oriented innovation as a way of describing forms of innovation that are motivated by reducing injustice and can reasonably be expected to do so.

Equity within AI systems: What can health leaders expect?

Artificial Intelligence (AI) for health has a great potential; it has already proven to be successful in enhancing patient outcomes, facilitating professional work and benefiting administration. However, AI presents challenges related to health equity defined as an opportunity for people to reach their fullest health potential. This article discusses the opportunities and challenges that AI presents in health and examines ways in which inequities related to AI can be mitigated.

A Framework for Developing Health Equity Initiatives in Radiology

PURPOSE

In recent years, radiology departments have increasingly recognized the extent of health care disparities related to imaging and image-guided interventions. The goal of this article is to provide a framework for developing a health equity initiative in radiology and to articulate key defining factors.

METHODS

This article leverages the experience of three academic radiology departments and explores key principles that emerged when observing the experiences of these departments that have begun to engage in health equity-focused work.

RESULTS

A four-component framework is described for a health equity initiative in radiology consisting of (1) environmental scan and blueprint, (2) design and implementation, (3) initiative evaluation, and (4) community engagement. Key facilitators include a comprehensive environmental scan, early stakeholder engagement and consensus building, implementation science design thinking, and multitiered community engagement.

CONCLUSIONS

All radiology organizations should strive to develop, pilot, and evaluate novel initiatives that promote equitable access to high-quality imaging services. Establishing systems for high-quality data collection is critical to success. An implementation science approach provides a robust framework for developing and testing novel health equity initiatives in radiology. Community engagement is critical at all stages of the health equity initiative time line.

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow

IMPORTANCE

Large language model (LLM) artificial intelligence (AI) chatbots direct the power of large training datasets towards successive, related tasks, as opposed to single-ask tasks, for which AI already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as virtual physicians, has not yet been evaluated.

OBJECTIVE

To evaluate ChatGPT's capacity for ongoing clinical decision support via its performance on standardized clinical vignettes.

DESIGN

We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity.

SETTING

ChatGPT, a publicly available LLM.

PARTICIPANTS

Clinical vignettes featured hypothetical patients with a variety of age and gender identities, and a range of Emergency Severity Indices (ESIs) based on initial clinical presentation.

EXPOSURES

MSD Clinical Manual vignettes.

MAIN OUTCOMES AND MEASURES

We measured the proportion of correct responses to the questions posed within the clinical vignettes tested.

RESULTS

ChatGPT achieved 71.7% (95% CI, 69.3% to 74.1%) accuracy overall across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI, 67.8% to 86.1%), and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI, 54.2% to 66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=-15.8%, p<0.001) and clinical management (β=-7.4%, p=0.02) type questions.

CONCLUSIONS AND RELEVANCE

ChatGPT achieves impressive accuracy in clinical decision making, with particular strengths emerging as it has more clinical information at its disposal.

Practical Approaches to Advancing Health Equity in Radiology, From the AJR Special Series on DEI

Despite significant advances in healthcare, many patients from medically underserved populations are impacted by existing healthcare disparities. Radiologists are uniquely positioned to decrease health disparities and advance health equity efforts in their practices. However, literature on practical tools for advancing radiology health equity efforts applicable to a wide variety of patient populations and care settings is lacking. Therefore, this article seeks to equip radiologists with an evidence-based and practical knowledge toolkit of health equity strategies, presented in terms of four pillars of research, clinical care, education, and innovation. For each pillar, equity efforts across diverse patient populations and radiology practice settings are examined through the lens of existing barriers, current best practices, and future directions, incorporating practical examples relevant to a spectrum of patient populations. Health equity efforts provide an opportune window to transform radiology through personalized care delivery that is responsive to diverse patient needs. Guided by compassion and empathy as core principles of health equity, leveraging the four pillars provides a helpful framework to advance health equity efforts as a step towards social justice in health.

Artificial Intelligence Applications in Health Care Practice: A Scoping Review (Preprint)

Background

Artificial intelligence (AI) is often heralded as a potential disruptor that will transform the practice of medicine. The amount of data collected and available in health care, coupled with advances in computational power, has contributed to advances in AI and an exponential growth of publications. However, the development of AI applications does not guarantee their adoption into routine practice. There is a risk that despite the resources invested, benefits for patients, staff, and society will not be realized if AI implementation is not better understood.

Objective

The aim of this study was to explore how the implementation of AI in health care practice has been described and researched in the literature by answering 3 questions: What are the characteristics of research on implementation of AI in practice? What types and applications of AI systems are described? What characteristics of the implementation process for AI systems are discernible?

Methods

A scoping review was conducted of MEDLINE (PubMed), Scopus, Web of Science, CINAHL, and PsycINFO databases to identify empirical studies of AI implementation in health care since 2011, in addition to snowball sampling of selected reference lists. Using Rayyan software, we screened titles and abstracts and selected full-text articles. Data from the included articles were charted and summarized.

Results

Of the 9218 records retrieved, 45 (0.49%) articles were included. The articles cover diverse clinical settings and disciplines; most (32/45, 71%) were published recently, were from high-income countries (33/45, 73%), and were intended for care providers (25/45, 56%). AI systems are predominantly intended for clinical care, particularly clinical care pertaining to patient-provider encounters. More than half (24/45, 53%) possess no action autonomy but rather support human decision-making. The focus of most research was on establishing the effectiveness of interventions (16/45, 35%) or related to technical and computational aspects of AI systems (11/45, 24%). Focus on the specifics of implementation processes does not yet seem to be a priority in research, and the use of frameworks to guide implementation is rare.

Conclusions

Our current empirical knowledge derives from implementations of AI systems with low action autonomy and approaches common to implementations of other types of information systems. To develop a specific and empirically based implementation framework, further research is needed on the more disruptive types of AI systems being implemented in routine care and on aspects unique to AI implementation in health care, such as building trust, addressing transparency issues, developing explainable and interpretable solutions, and addressing ethical concerns around privacy and data protection.

Equitable Implementation of Artificial Intelligence in Medical Imaging: What Can be Learned from Implementation Science?

Artificial intelligence (AI) has been rapidly adopted in various health care domains. Molecular imaging, accordingly, has demonstrated growing academic and commercial interest in AI. Unprepared and inequitable implementation and scale-up of AI in health care may pose challenges. Implementation of AI, as a complex intervention, may face various barriers, at individual, interindividual, organizational, health system, and community levels. To address these barriers, recommendations have been developed to consider health equity as a critical lens to sensitize implementation, engage stakeholders in implementation and evaluation, recognize and incorporate the iterative nature of implementation, and integrate equity and implementation in early-stage AI research.