Characterization and analysis of the proposals submitted to the PET-Health Interprofessionality in Brazil: advancements and future directions

The Program of Education through Work for Health (PET-Health), with a focus on interprofessionality, is one of the actions of the Plan for the Strengthening of Interprofessional Education in Brazil. This research aimed to systematically analyze the characteristics of the proposals submitted to the public notice of the PET-Health Interprofessionality specifically in relation to the theoretical-conceptual and methodological alignment of interprofessional education (IPE). The study is a qualitative document content analysis. We analyzed one hundred and twenty projects submitted to the selection process from institutions participating in the PET-Health Interprofessionality. Content analysis followed three steps: pre-analysis, exploration of the material, and treatment and interpretation of results. Seven categories were identified: a) alignment with the theoretical-conceptual frameworks of IPE, b) curriculum changes, c) faculty development with a focus on IPE, d) articulation among objectives, actions, and results expected related to IPE, e) strategies for monitoring and evaluation, f) involvement of users/families and community, and g) development of collaborative competencies. We conclude that while some advancements have been made, there remains a need for more in-depth discussion in Brazil to ensure the development of competencies capable of assuring more integral, resolute, and safer healthcare services, with capacity to (re)signify user-centered care in the planning and delivery of healthcare.

Using implementation science to operate as a learning health system to improve outcomes in early psychosis

AIM

Early interventions are well understood to improve psychosis outcomes, but their successful implementation remains limited. This article introduces a three-step roadmap for advancing the implementation of evidence-based practices to operate as a learning health system, which can be applied to early interventions for psychosis and is intended for an audience that is relatively new to systematic approaches to implementation.

METHODS

The roadmap is grounded in implementation science, which specializes in methods to promote routine use of evidence-based innovations. The roadmap draws on learning health system principles that call for commitment of leadership, application of evidence, examination of care experiences, and study of health outcomes. Examples are discussed for each roadmap step, emphasizing both data- and stakeholder-related considerations applicable throughout the roadmap.

CONCLUSIONS

Early psychosis care is a promising topic through which to discuss the critical need to move evidence into practice. Despite remarkable advances in early psychosis interventions, population-level impact of those interventions is yet to be realized. By providing an introduction to how implementation science principles can be operationalized in a learning health system and sharing examples from early psychosis care, this article prompts inclusion of a wider audience in essential discourse on the role that implementation science can play for moving evidence into practice for other realms of psychiatric care as well. To this end, the proposed roadmap can serve as a conceptual guiding template and framework through which various psychiatric services can methodically pursue timely implementation of evidence-based interventions for higher quality care and improved outcomes.

Implications of clinical simulation in motivation for learning: scoping review

OBJECTIVE

To identify, synthesize, and analyze the scientific knowledge produced regarding the implications of using clinical simulation for undergraduate nursing or medical students' motivation for learning.

METHODS

The search for articles was conducted between July 28 and August 3, 2022, on the PubMed/MEDLINE, Scopus, Web of Science, and SciELO databases. The following was used for the search: P - undergraduate students attending Nursing or Medicine courses; C - motivation for learning, and C - skills and clinical simulation laboratory. The following research question guided the study: "What are the implications of clinical simulation on the motivation for learning of undergraduate students of nursing and medicine?" Of the 1,783 articles found, 13 were included in the sample for analysis. All stages of the selection process were carried out by two independent evaluators. The results were presented as charts and a discursive report.

RESULTS

The studies analyzed indicated the beneficial effects of clinical simulation on students' motivation, in addition to other gains such as competencies, technical and non-technical skills, knowledge, belonging, autonomy, clinical judgment, critical and reflective thinking, self-efficacy and decreased anxiety, self-management, and improvements in learning and learning climate.

CONCLUSION

Clinical simulation provides a positive learning environment favorable to the development of technical and interpersonal skills and competencies, and raising the level of motivational qualities.

Learning from data in dentistry: Summary of the third annual OpenWide conference

The overarching goal of the third scientific oral health symposium was to introduce the concept of a learning health system to the dental community and to identify and discuss cutting-edge research and strategies using data for improving the quality of dental care and patient safety. Conference participants included clinically active dentists, dental researchers, quality improvement experts, informaticians, insurers, EHR vendors/developers, and members of dental professional organizations and dental service organizations. This report summarizes the main outputs of the third annual OpenWide conference held in Houston, Texas, on October 12, 2022, as an affiliated meeting of the American Dental Association (ADA) 2022 annual conference.

The prototype of a frailty learning health system: The HARMONY Model

Introduction

Rapid translation of research findings into clinical practice through innovation is critical to improve health systems and patient outcomes. Access to efficient systems of learning underpinned with real-time data are the future of healthcare. This type of health system will decrease unwarranted clinical variation, accelerate rapid evidence translation, and improve overall healthcare quality.

Methods

This paper aims to describe The HARMONY model (acHieving dAta-dRiven quality iMprovement to enhance frailty Outcomes using a learNing health sYstem), a new frailty learning health system model of implementation science and practice improvement. The HARMONY model provides a prototype for clinical quality registry infrastructure and partnership within health care.

Results

The HARMONY model was applied to the Western Sydney Clinical Frailty Registry as the prototype exemplar. The model networks longitudinal frailty data into an accessible and useable format for learning. Creating local capability that networks current data infrastructures to translate and improve quality of care in real-time.

Conclusion

This prototype provides a model of registry data feedback and quality improvement processes in an inpatient aged care and rehabilitation hospital setting to help reduce clinical variation, enhance research translation capacity, and improve care quality.

Developing a relational playbook for cardiology teams to cultivate supportive learning environments, enhance clinician well-being, and veteran care

Introduction

Despite the Veterans Health Administration (VA) efforts to become a learning health system (LHS) and high-reliability organization (HRO), interventions to build supportive learning environments within teams are not reliably implemented, contributing to high levels of burnout, turnover, and variation in care. Supportive learning environments build capabilities for teaching and learning, empower teams to safely trial and adapt new things, and adopt highly reliable work practices (eg, debriefs). Innovative approaches to create supportive learning environments are needed to advance LHS and HRO theory and research into practice.

Methods

To guide the identification of evidence-based interventions that cultivate supportive learning environments, the authors used a longitudinal, mixed-methods design and LHS and HRO frameworks. We partnered with the 81 VA cardiac catheterization laboratories and conducted surveys, interviews, and literature reviews that informed a Relational Playbook for Cardiology Teams.

Results

The Relational Playbook resources and 50 evidence-based interventions are organized into five LHS and HRO-guided chapters: Create a positive culture, teamwork, leading teams, joy in work, communication, and high reliability. The interventions are designed for managers to integrate into existing meetings or trainings to cultivate supportive learning environments.

Conclusions

LHS and HRO frameworks describe how organizations can continually learn and deliver nearly error-free services. The Playbook resources and interventions translate LHS and HRO frameworks for real-world implementation by healthcare managers. This work will cultivate supportive learning environments, employee well-being, and Veteran safety while providing insights into LHS and HRO theory, research, and practice.

Using a Rapid Learning Health System for Stratified Care in Emerging Adult Mental Health Services: Protocol for the Implementation of Patient-Reported Outcome Measures

BACKGROUND

Mental illness among emerging adults is often difficult to ameliorate due to fluctuating symptoms and heterogeneity. Recently, innovative approaches have been developed to improve mental health care for emerging adults, including (1) implementing patient-reported outcome measures (PROMs) to assess illness severity and inform stratified care to assign emerging adults to a treatment modality commensurate with their level of impairment and (2) implementing a rapid learning health system in which data are continuously collected and analyzed to generate new insights, which are then translated to clinical practice, including collaboration among clients, health care providers, and researchers to co-design and coevaluate assessment and treatment strategies.

OBJECTIVE

The aim of the study is to determine the feasibility and acceptability of implementing a rapid learning health system to enable a measurement-based, stratified care treatment strategy for emerging adults.

METHODS

This study takes place at a specialty clinic serving emerging adults (age 16-24 years) in Calgary, Canada, and involves extensive collaboration among researchers, providers, and youth. The study design includes six phases: (1) developing a transdiagnostic platform for PROMs, (2) designing an initial stratified care model, (3) combining the implementation of PROMs with stratified care, (4) evaluating outcomes and disseminating results, (5) modification of stratified care based on data derived from PROMs, and (6) spread and scale to new sites. Qualitative and quantitative feedback will be collected from health care providers and youth throughout the implementation process. These data will be analyzed at regular intervals and used to modify the way future services are delivered. The RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework is used to organize and evaluate implementation according to 3 key objectives: improving treatment selection, reducing average wait time and treatment duration, and increasing the value of services.

RESULTS

This project was funded through a program grant running from 2021 to 2026. Ethics approval for this study was received in February 2023. Presently, we have developed a system of PROMs and organized clinical services into strata of care. We will soon begin using PROMs to assign clients to a stratum of care and using feedback from youth and clinicians to understand how to improve experiences and outcomes.

CONCLUSIONS

This study has key implications for researchers and clinicians looking to understand how to customize emerging adult mental health services to improve the quality of care and satisfaction with care. This study has significant implications for mental health care systems as part of a movement toward value-based health care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/51667.

Development of data processing algorithm to calculate adherence for adults with cystic fibrosis using inhaled therapy - a multi-center observational study within the CFHealthHub learning health system

OBJECTIVES

To develop a robust algorithm to accurately calculate 'daily complete dose counts' for inhaled medicines, used in percent adherence calculations, from electronically-captured nebulizer data within the CFHealthHub Learning Health System.

METHODS

A multi-center, cross-sectional study involved participants and clinicians reviewing real-world inhaled medicine usage records and triangulating them with objective nebulizer data to establish a consensus on 'daily complete dose counts.' An algorithm, which used only objective nebulizer data, was then developed using a derivation dataset and evaluated using internal validation dataset. The agreement and accuracy between the algorithm-derived and consensus-derived 'daily complete dose counts' was examined, with the consensus-derived count as the reference standard.

RESULTS

Twelve people with CF participated. The algorithm derived a 'daily complete dose count' by screening out 'invalid' doses (those <60s in duration or run in cleaning mode), combining all doses starting within 120s of each other, and then screening out all doses with duration < 480s which were interrupted by power supply failure. The kappa co-efficient was 0.85 (0.71-0.91) in the derivation and 0.86 (0.77-0.94) in the validation dataset.

CONCLUSIONS

The algorithm demonstrated strong agreement with the participant-clinician consensus, enhancing confidence in CFHealthHub data. Publishingdata processing methods can encourage trust in digital endpoints and serve as an exemplar for other projects.

Public health surveillance and the data, information, knowledge, intelligence and wisdom paradigm

This article points out deficiencies in present-day definitions of public health surveillance, which include data collection, analysis, interpretation and dissemination, but not public health action. Controlling a public health problem of concern requires a public health response that goes beyond information dissemination. It is undesirable to have public health divided into data generation processes (public health surveillance) and data use processes (public health response), managed by two separate groups (surveillance experts and policy-makers). It is time to rethink the need to modernize the definition of public health surveillance, inspired by the authors' enhanced Data, Information, Knowledge, Intelligence and Wisdom model. Our recommendations include expanding the scope of public health surveillance beyond information dissemination to comprise actionable knowledge (intelligence); mandating surveillance experts to assist policy-makers in making evidence-informed decisions; encouraging surveillance experts to become policy-makers; and incorporating public health literacy training - from data to knowledge to wisdom - into the curricula for all public health professionals. Work on modernizing the scope and definition of public health surveillance will be a good starting point.

Implementation of a Clinical, Patient-Level Dashboard at a Mental Health Hospital: Lessons Learned from Two Pilot Clinics

The Centre for Addiction and Mental Health has implemented mechanisms to standardize routine data collection with the vision of a Learning Health System. To improve clinical decision-making and patient outcomes, a clinical dashboard was implemented to provide a real-time visualization of data from patient self-assessments and other physical and mental health indicators. This case report shares early findings of dashboard implementation to understand user uptake and improve fidelity of the technology and processes that need to support adoption. Moreover, these findings will inform the strategy and development of a hospital-wide scalable dashboard that will span across clinical areas and leverage artificial intelligence to continuously improve patient outcomes and equitable care delivery.

Efficacy of Digital Outreach Strategies for Collecting Smoking Data: Pragmatic Randomized Trial

BACKGROUND

Tobacco smoking is an important risk factor for disease, but inaccurate smoking history data in the electronic medical record (EMR) limits the reach of lung cancer screening (LCS) and tobacco cessation interventions. Patient-generated health data is a novel approach to documenting smoking history; however, the comparative effectiveness of different approaches is unclear.

OBJECTIVE

We designed a quality improvement intervention to evaluate the effectiveness of portal questionnaires compared to SMS text message-based surveys, to compare message frames, and to evaluate the completeness of patient-generated smoking histories.

METHODS

We randomly assigned patients aged between 50 and 80 years with a history of tobacco use who identified English as a preferred language and have never undergone LCS to receive an EMR portal questionnaire or a text survey. The portal questionnaire used a "helpfulness" message, while the text survey tested frame types informed by behavior economics ("gain," "loss," and "helpfulness") and nudge messaging. The primary outcome was the response rate for each modality and framing type. Completeness and consistency with documented structured smoking data were also evaluated.

RESULTS

Participants were more likely to respond to the text survey (191/1000, 19.1%) compared to the portal questionnaire (35/504, 6.9%). Across all text survey rounds, patients were less responsive to the "helpfulness" frame compared with the "gain" frame (odds ratio [OR] 0.29, 95% CI 0.09-0.91; P<.05) and "loss" frame (OR 0.32, 95% CI 11.8-99.4; P<.05). Compared to the structured data in the EMR, the patient-generated data were significantly more likely to be complete enough to determine LCS eligibility both compared to the portal questionnaire (OR 34.2, 95% CI 3.8-11.1; P<.05) and to the text survey (OR 6.8, 95% CI 3.8-11.1; P<.05).

CONCLUSIONS

We found that an approach using patient-generated data is a feasible way to engage patients and collect complete smoking histories. Patients are likely to respond to a text survey using "gain" or "loss" framing to report detailed smoking histories. Optimizing an SMS text message approach to collect medical information has implications for preventative and follow-up clinical care beyond smoking histories, LCS, and smoking cessation therapy.

A call-to-action: integrate a learning health system framework into longitudinal population studies to improve health response in Africa

Longitudinal population studies (LPSs) in Africa have the potential to become powerful engines of change by adopting a learning health system (LHS) framework. This is a call-to-action opinion and highlights the importance of integrating an LHS approach into LPSs, emphasizing their transformative potential to improve population health response, drive evidence-based decision making, and enhance community well-being. Operators of LPS platforms, community members, government officials, and funding agencies have a role to contribute to this transformative journey of driving evidence-based interventions, promoting health equity, and fostering long-term public health solutions for African communities.

Building a vertically integrated genomic learning health system: The biobank at the Colorado Center for Personalized Medicine

Precision medicine initiatives across the globe have led to a revolution of repositories linking large-scale genomic data with electronic health records, enabling genomic analyses across the entire phenome. Many of these initiatives focus solely on research insights, leading to limited direct benefit to patients. We describe the biobank at the Colorado Center for Personalized Medicine (CCPM Biobank) that was jointly developed by the University of Colorado Anschutz Medical Campus and UCHealth to serve as a unique, dual-purpose research and clinical resource accelerating personalized medicine. This living resource currently has more than 200,000 participants with ongoing recruitment. We highlight the clinical, laboratory, regulatory, and HIPAA-compliant informatics infrastructure along with our stakeholder engagement, consent, recontact, and participant engagement strategies. We characterize aspects of genetic and geographic diversity unique to the Rocky Mountain region, the primary catchment area for CCPM Biobank participants. We leverage linked health and demographic information of the CCPM Biobank participant population to demonstrate the utility of the CCPM Biobank to replicate complex trait associations in the first 33,674 genotyped individuals across multiple disease domains. Finally, we describe our current efforts toward return of clinical genetic test results, including high-impact pathogenic variants and pharmacogenetic information, and our broader goals as the CCPM Biobank continues to grow. Bringing clinical and research interests together fosters unique clinical and translational questions that can be addressed from the large EHR-linked CCPM Biobank resource within a HIPAA- and CLIA-certified environment.

Training the next generation of delivery science researchers: 10-year experience of a post-doctoral research fellowship program within an integrated care system

Introduction

Learning health systems require a workforce of researchers trained in the methods of identifying and overcoming barriers to effective, evidence-based care. Most existing postdoctoral training programs, such as NIH-funded postdoctoral T32 awards, support basic and epidemiological science with very limited focus on rigorous delivery science methods for improving care. In this report, we present the 10-year experience of developing and implementing a Delivery Science postdoctoral fellowship embedded within an integrated health care delivery system.

Methods

In 2012, the Kaiser Permanente Northern California Division of Research designed and implemented a 2-year postdoctoral Delivery Science Fellowship research training program to foster research expertise in identifying and addressing barriers to evidence-based care within health care delivery systems.

Results

Since 2014, 20 fellows have completed the program. Ten fellows had PhD-level scientific training, and 10 fellows had clinical doctorates (eg, MD, RN/PhD, PharmD). Fellowship alumni have graduated to faculty research positions at academic institutions (9), and research or clinical organizations (4). Seven alumni now hold positions in Kaiser Permanente's clinical operations or medical group (7).

Conclusions

This delivery science fellowship program has succeeded in training graduates to address delivery science problems from both research and operational perspectives. In the next 10 years, additional goals of the program will be to expand its reach (eg, by developing joint research training models in collaboration with clinical fellowships) and strengthen mechanisms to support transition from fellowship to the workforce, especially for researchers from underrepresented groups.

Implementing the learning health system paradigm within academic health centers

Introduction

The learning health system (LHS) concept represents a bold innovation that combines organizational learning, strategic analysis of patient data, stakeholder engagement and the systematic translation of research into practice - all in service of improving the quality of health care delivered across the organization. This innovation has been diffused and widely adopted by healthcare organizations over the past 15 years, but academic health centers (AHCs) have been slower on the uptake. The irony is that AHCs have the resources (e.g., trained researchers, sophisticated clinical data systems, informatics infrastructure) that are necessary to do the highest-quality and most impactful LHS work.

Methods

Based on a review of publications describing how AHCs have implemented LHS work, as well as the authors' direct experience promoting the adoption of the LHS paradigm at Atrium Health Wake Forest Baptist (AHWFB), we:identify a set of factors that have inhibited broader adoption of the LHS paradigm among AHCs; distinguish between the forms of LHS work that are consistent and inconsistent with the mission of AHCs; and offer recommendations for broader adoption and fuller implementation of the LHS paradigm.

Results

The LHS paradigm represents an expansion of the scientific paradigm which serves as the foundation of research enterprise within AHCs. Both paradigms value rigorous studies of new treatments and practices, including pragmatic clinical trials. The LHS paradigm also places a high value on quality improvement studies, organizational learning, and the translation of research findings into improved patient care and operations within the local health system. The two paradigms differ on the origin of the research question, i.e., a pressing patient-care issue facing the health system versus the investigator's own research interests. Academic researchers have been disincentivized from pursuing at least some forms of LHS research. However, a growing number of AHCs are finding ways to integrate the LHS paradigm into their research enterprise, either by providing research faculty with institutional funding to cover their effort on studies that address the health system's priority issues, or by establishing an institute dedicated to LHS research.

Conclusions

The LHS paradigm is a disruptive intervention for AHCs, one that was initially resisted but is increasingly being embraced. AHCs are developing strategies for conducting LHS research, typically in parallel to the more traditional biomedical science that is core to academic medicine. Full implementation of the LHS paradigm will require further alignment between LHS and science, including a shift in the criteria for promotion and tenure to support those researchers who choose to focus on the pressing issues facing the health system.

Ambulatory Care Coordination Data Gathering and Use

Care coordination is a crucial component of healthcare systems. However, little is known about data needs and uses in ambulatory care coordination practice. Therefore, the purpose of this study was to identify information gathered and used to support care coordination in ambulatory settings. Survey respondents (33) provided their demographics and practice patterns, including use of electronic health records, as well as data gathered and used. Most of the respondents were nurses, and they described varying practice settings and patterns. Although most described at least partial use of electronic health records, two respondents described paper documentation systems. More than 25% of respondents gathered and used most of the 72 data elements, with collection and use often occurring in multiple locations and contexts. This early study demonstrates significant heterogeneity in ambulatory care coordination data usage. Additional research is necessary to identify common data elements to support knowledge development in the context of a learning health system.

Frameworks, guidelines, and tools to develop a learning health system for Indigenous health: An environmental scan for Canada

Introduction

First Nations, Inuit, and Métis (FNIM) peoples experience systemic health disparities within Ontario's healthcare system. Learning health systems (LHS) is a rapidly growing interdisciplinary area with the potential to address these inequitable health outcomes through a comprehensive health system that draws on science, informatics, incentives, and culture for ongoing innovation and improvement. However, global literature is in its infancy with grounding theories and principles still emerging. In addition, there is inadequate information on LHS within Ontario's health care context.

Methods

We conducted an environmental scan between January and April 2021 and again in June 2022 to identify existing frameworks, guidelines, and tools for designing, developing, implementing, and evaluating an LHS.

Results

We found 37 relevant sources. This paper maps the literature and identifies gaps in knowledge based on five key pillars: (a) data and evidence-driven, (b) patient-centeredness, (c) system-supported, (d) cultural competencies enabled, and (e) the learning health system.

Conclusion

We provide recommendations for implementation accordingly. The literature on LHS provides a starting point to address the health disparities of FNIM peoples within the healthcare system but Indigenous community partnerships in LHS development and operation will be key to success.

Benchmarking the symptom-checking capabilities of ChatGPT for a broad range of diseases

OBJECTIVE

This study evaluates ChatGPT's symptom-checking accuracy across a broad range of diseases using the Mayo Clinic Symptom Checker patient service as a benchmark.

METHODS

We prompted ChatGPT with symptoms of 194 distinct diseases. By comparing its predictions with expectations, we calculated a relative comparative score (RCS) to gauge accuracy.

RESULTS

ChatGPT's GPT-4 model achieved an average RCS of 78.8%, outperforming the GPT-3.5-turbo by 10.5%. Some specialties scored above 90%.

DISCUSSION

The test set, although extensive, was not exhaustive. Future studies should include a more comprehensive disease spectrum.

CONCLUSION

ChatGPT exhibits high accuracy in symptom checking for a broad range of diseases, showcasing its potential as a medical training tool in learning health systems to enhance care quality and address health disparities.

Informatics approaches to improve the quality of dental care

Despite technological advances, challenges exist in US dental care, including variations in quality of care, access and untreated dental needs. The implementation of learning health systems (LHSs) in dentistry can help to address these challenges. LHSs use robust informatics infrastructure including data and technology to continuously measure and improve the quality and safety of care and can help to reduce costs and improve patient outcomes. The use of EHRs and standardized diagnostic terminologies are highlighted, as they allow for the storage and sharing of patient data, providing a comprehensive view of a patient's medical and dental history, and can be used to identify patterns and trends to improve the delivery of care. The BigMouth Dental Data Repository is an example of an informatic platform that aggregates patient data from multiple institutions and is being used to for scientific inquiry to improve oral health.

A holistic approach to integrating patient, family, and lived experience voices in the development of the BrainHealth Databank: a digital learning health system to enable artificial intelligence in the clinic

Artificial intelligence, machine learning, and digital health innovations have tremendous potential to advance patient-centred, data-driven mental healthcare. To enable the clinical application of such innovations, the Krembil Centre for Neuroinformatics at the Centre for Addiction and Mental Health, Canada's largest mental health hospital, embarked on a journey to co-create a digital learning health system called the BrainHealth Databank (BHDB). Working with clinicians, scientists, and administrators alongside patients, families, and persons with lived experience (PFLE), this hospital-wide team has adopted a systems approach that integrates clinical and research data and practices to improve care and accelerate research. PFLE engagement was intentional and initiated at the conception stage of the BHDB to help ensure the initiative would achieve its goal of understanding the community's needs while improving patient care and experience. The BHDB team implemented an evolving, dynamic strategy to support continuous and active PFLE engagement in all aspects of the BHDB that has and will continue to impact patients and families directly. We describe PFLE consultation, co-design, and partnership in various BHDB activities and projects. In all three examples, we discuss the factors contributing to successful PFLE engagement, share lessons learned, and highlight areas for growth and improvement. By sharing how the BHDB navigated and fostered PFLE engagement, we hope to motivate and inspire the health informatics community to collectively chart their paths in PFLE engagement to support advancements in digital health and artificial intelligence.

Learning in real world practice: Identifying implementation strategies to integrate health-related social needs screening within a large health system

Introduction

Health systems have many incentives to screen patients for health-related social needs (HRSNs) due to growing evidence that social determinants of health impact outcomes and a new regulatory context that requires health equity measures. This study describes the experience of one large urban health system in scaling HRSN screening by implementing improvement strategies over five years, from 2018 to 2023.

Methods

In 2018, the health system adapted a 10-item HRSN screening tool from a widely used, validated instrument. Implementation strategies aimed to foster screening were retrospectively reviewed and categorized according to the Expert Recommendations for Implementing Change (ERIC) study. Statistical process control methods were utilized to determine whether implementation strategies contributed to improvements in HRSN screening activities.

Results

There were 280,757 HRSN screens administered across 311 clinical teams in the health system between April 2018 and March 2023. Implementation strategies linked to increased screening included integrating screening within an online patient portal (ERIC strategy: involve patients/consumers and family members), expansion to discrete clinical teams (ERIC strategy: change service sites), providing data feedback loops (ERIC strategy: facilitate relay of clinical data to providers), and deploying Community Health Workers to address HRSNs (ERIC strategy: create new clinical teams).

Conclusion

Implementation strategies designed to promote efficiency, foster universal screening, link patients to resources, and provide clinical teams with an easy-to-integrate tool appear to have the greatest impact on HRSN screening uptake. Sustained increases in screening demonstrate the cumulative effects of implementation strategies and the health system's commitment toward universal screening.

Applying anti-racist approaches to informatics: a new lens on traditional frames

Health organizations and systems rely on increasingly sophisticated informatics infrastructure. Without anti-racist expertise, the field risks reifying and entrenching racism in information systems. We consider ways the informatics field can recognize institutional, systemic, and structural racism and propose the use of the Public Health Critical Race Praxis (PHCRP) to mitigate and dismantle racism in digital forms. We enumerate guiding questions for stakeholders along with a PHCRP-Informatics framework. By focusing on (1) critical self-reflection, (2) following the expertise of well-established scholars of racism, (3) centering the voices of affected individuals and communities, and (4) critically evaluating practice resulting from informatics systems, stakeholders can work to minimize the impacts of racism. Informatics, informed and guided by this proposed framework, will help realize the vision of health systems that are more fair, just, and equitable.

Nurse Leader Perceptions of Data in the Veterans Health Administration: A Qualitative Evaluation

Healthcare systems and nursing leaders aim to make evidence-based nurse staffing decisions. Understanding how nurses use and perceive available data to support safe staffing can strengthen learning healthcare systems and support evidence-based practice, particularly given emerging data availability and specific nursing challenges in data usability. However, current literature offers sparse insight into the nature of data use and challenges in the inpatient nurse staffing management context. We aimed to investigate how nurse leaders experience using data to guide their inpatient staffing management decisions in the Veterans Health Administration, the largest integrated healthcare system in the United States. We conducted semistructured interviews with 27 Veterans Health Administration nurse leaders across five management levels, using a constant comparative approach for analysis. Participants primarily reported using data for quality improvement, organizational learning, and organizational monitoring and support. Challenges included data fragmentation, unavailability and unsuitability to user need, lack of knowledge about available data, and untimely reporting. Our findings suggest that prioritizing end-user experience and needs is necessary to better govern evidence-based data tools for improving nursing care. Continuous nurse leader involvement in data governance is integral to ensuring high-quality data for end-user nurses to guide their decisions impacting patient care.

One Health in a Digital World: Technology, Data, Information and Knowledge

OBJECTIVES

To describe the origins and growth of the One Health concept and its recent application in One Digital Health.

METHODS

Bibliometric review and critical discussion of emergent themes derived from co-occurrence of MeSH keywords.

RESULTS

The fundamental interrelationship between human health, animal health and the wider environment has been recognized since ancient times. One Health as a distinct term originated in 2004 and has been a rapidly growing concept of interest in the biomedical literature since 2017. One Digital Health has quickly established itself as a unifying construct that highlights the critical role of technology, data, information and knowledge to facilitate the interdisciplinary collaboration that One Health requires. The principal application domains of One Digital Health to date are in FAIR data integration and analysis, disease surveillance, antimicrobial stewardship and environmental monitoring.

CONCLUSIONS

One Health and One Digital Health offer powerful lenses to examine and address crises in our living world. We propose thinking in terms of Learning One Health Systems that can dynamically capture, integrate, analyse and monitor application of data across the biosphere.

A Neuro-Informatics Pipeline for Cerebrovascular Disease: Research Registry Development

BACKGROUND

Although stroke is well recognized as a critical disease, treatment options are often limited. Inpatient stroke encounters carry critical information regarding the mechanisms of stroke and patient outcomes; however, these data are typically formatted to support administrative functions instead of research. To support improvements in the care of patients with stroke, a substantive research data platform is needed.

OBJECTIVE

To advance a stroke-oriented learning health care system, we sought to establish a comprehensive research repository of stroke data using the Houston Methodist electronic health record (EHR) system.

METHODS

Dedicated processes were developed to import EHR data of patients with primary acute ischemic stroke, intracerebral hemorrhage (ICH), transient ischemic attack, and subarachnoid hemorrhage under a review board-approved protocol. Relevant patients were identified from discharge diagnosis codes and assigned registry patient identification numbers. For identified patients, extract, transform, and load processes imported EHR data of primary cerebrovascular disease admissions and available data from any previous or subsequent admissions. Data were loaded into patient-focused SQL objects to enable cross-sectional and longitudinal analyses. Primary data domains (admission details, comorbidities, laboratory data, medications, imaging data, and discharge characteristics) were loaded into separate relational tables unified by patient and encounter identification numbers. Computed tomography, magnetic resonance, and angiography images were retrieved. Imaging data from patients with ICH were assessed for hemorrhage characteristics and cerebral small vessel disease markers. Patient information needed to interface with other local and national databases was retained. Prospective patient outreach was established, with patients contacted via telephone to assess functional outcomes 30, 90, 180, and 365 days after discharge. Dashboards were constructed to provide investigators with data summaries to support access.

RESULTS

The Registry of Neurological Endpoint Assessments among Patients with Ischemic and Hemorrhagic Stroke (REINAH) database was constructed as a series of relational category-specific SQL objects. Encounter summaries and dashboards were constructed to draw from these objects, providing visual data summaries for investigators seeking to build studies based on REINAH data. As of June 2022, the database contains 18,061 total patients, including 1809 (10.02%) with ICH, 13,444 (74.43%) with acute ischemic stroke, 1221 (6.76%) with subarachnoid hemorrhage, and 3165 (17.52%) with transient ischemic attack. Depending on the cohort, imaging data from computed tomography are available for 85.83% (1048/1221) to 98.4% (1780/1809) of patients, with magnetic resonance imaging available for 27.85% (340/1221) to 85.54% (11,500/13,444) of patients. Outcome assessment has successfully contacted 56.1% (240/428) of patients after ICH, with 71.3% (171/240) of responders providing consent for assessment. Responders reported a median modified Rankin Scale score of 3 at 90 days after discharge.

CONCLUSIONS

A highly curated and clinically focused research platform for stroke data will establish a foundation for future research that may fundamentally improve poststroke patient care and outcomes.

Alpha NSW: What would it take to create a state-wide paediatric population-level learning health system?

BACKGROUND

The health and well-being of children in the first 2000 days has a lasting effect on educational achievement and long-term chronic disease in later life. However, the lack of integration between high-quality data, analytic capacity and timely health improvement initiatives means practitioners, service leaders and policymakers cannot use data effectively to plan and evaluate early intervention services and monitor high-level health outcomes.

OBJECTIVE

Our exploratory study aimed to develop an in-depth understanding of the system and clinical requirements of a state-wide paediatric learning health system (LHS) that uses routinely collected data to not only identify where the inequities and variation in care are, but also to also inform service development and delivery where it is needed most.

METHOD

Our approach included reviewing exemplars of how administrative data are used in Australia; consulting with clinical, policy and data stakeholders to determine their needs for a child health LHS; mapping the existing data points collected across the first 2000 days of a child's life and geospatially locating patterns of key indicators for child health needs.

RESULTS

Our study identified the indicators that are available and accessible to inform service delivery and demonstrated the potential of using routinely collected administrative data to identify the gap between health needs and service availability.

CONCLUSION

We recommend improving data collection, accessibility and integration to establish a state-wide LHS, whereby there is a streamlined process for data cleaning, analysis and visualisation to help identify populations in need in a timely manner.

A Review of Requirements for Information Models in Learning Health Systems

Learning Health System (LHS) and integrated care are challenged due to a fragmented health data landscape. An information model is agnostic to the underlying data structures and can potentially contribute to mitigating some of the gaps. In a research project, Valkyrie, we are exploring how metadata can be organized and used to promote service coordination and interoperability across levels of care. An information model is viewed as central in this context and as a future integrated LHS support. We examined the literature regarding property requirements for data, information and knowledge models in the context of semantic interoperability and an LHS. The requirements were elicited and synthesized into five guiding principles as a vocabulary to inform the information model design of Valkyrie. Further research on requirements and guiding principles for information model design and evaluation are welcomed.

Information Models Properties in Learning Health Systems: A Literature Review

Learning Health Systems (LHS) are challenged by fragmented health data. In Valkyrie, information models (IM) are explored as translators for the underlying, fragmented data structures and can potentially extend to support a future LHS. In this paper, a literature review was performed to search for property requirements for semantic interoperable IMs in the context of an LHS. The literature was examined and property requirements elicited in the context of an LHS.

Catalyzing dementia care through the learning health system and consumer health information technology

To advance care for persons with Alzheimer's disease and related dementias (ADRD), real-world health system effectiveness research must actively engage those affected to understand what works, for whom, in what setting, and for how long-an agenda central to learning health system (LHS) principles. This perspective discusses how emerging payment models, quality improvement initiatives, and population health strategies present opportunities to embed best practice principles of ADRD care within the LHS. We discuss how stakeholder engagement in an ADRD LHS when embedding, adapting, and refining prototypes can ensure that products are viable when implemented. Finally, we highlight the promise of consumer-oriented health information technologies in supporting persons living with ADRD and their care partners and delivering embedded ADRD interventions at scale. We aim to stimulate progress toward sustainable infrastructure paired with person- and family-facing innovations that catalyze broader transformation of ADRD care.

Harnessing the Collective Expertise of Patients, Care Partners, Clinical Teams, and Researchers Through a Coproduction Learning Health System: A Case Study of the Dartmouth Health Promise Partnership

The coproduction learning health system (CLHS) model extends the definition of a learning health system to explicitly bring together patients and care partners, health care teams, administrators, and scientists to share the work of optimizing health outcomes, improving care value, and generating new knowledge. The CLHS model highlights a partnership for coproduction that is supported by data that can be used to support individual patient care, quality improvement, and research. We provide a case study that describes the application of this model to transform care within an oncology program at an academic medical center.

Bridging the gap: A library-based collaboration to enhance data skills for clinical researchers

Introduction

Enterprise data warehouses (EDWs) serve as foundational infrastructure in a modern learning health system, housing clinical and other system-wide data and making it available for research, strategic, and quality improvement purposes. Building on a longstanding partnership between Northwestern University's Galter Health Sciences Library and the Northwestern Medicine Enterprise Data Warehouse (NMEDW), an end-to-end clinical research data management (cRDM) program was created to enhance clinical data workforce capacity and further expand related library-based services for the campus.

Methods

The training program covers topics such as clinical database architecture, clinical coding standards, and translation of research questions into queries for proper data extraction. Here we describe this program, including partners and motivations, technical and social components, integration of FAIR principles into clinical data research workflows, and the long-term implications for this work to serve as a blueprint of best practice workflows for clinical research to support library and EDW partnerships at other institutions.

Results

This training program has enhanced the partnership between our institution's health sciences library and clinical data warehouse to provide support services for researchers, resulting in more efficient training workflows. Through instruction on best practices for preserving and sharing outputs, researchers are given the tools to improve the reproducibility and reusability of their work, which has positive effects for the researchers as well as for the university. All training resources have been made publicly available so that those who support this critical need at other institutions can build on our efforts.

Conclusions

Library-based partnerships to support training and consultation offer an important vehicle for clinical data science capacity building in learning health systems. The cRDM program launched by Galter Library and the NMEDW is an example of this type of partnership and builds on a strong foundation of past collaboration, expanding the scope of clinical data support services and training on campus.

Stroke Learning Health Systems: A Topical Narrative Review With Case Examples

To our knowledge, the adoption of Learning Health System (LHS) concepts or approaches for improving stroke care, patient outcomes, and value have not previously been summarized. This topical review provides a summary of the published evidence about LHSs applied to stroke, and case examples applied to different aspects of stroke care from high and low-to-middle income countries. Our attempt to systematically identify the relevant literature and obtain real-world examples demonstrated the dissemination gaps, the lack of learning and action for many of the related LHS concepts across the continuum of care but also elucidated the opportunity for continued dialogue on how to study and scale LHS advances. In the field of stroke, we found only a few published examples of LHSs and health systems globally implementing some selected LHS concepts, but the term is not common. A major barrier to identifying relevant LHS examples in stroke may be the lack of an agreed taxonomy or terminology for classification. We acknowledge that health service delivery settings that leverage many of the LHS concepts do so operationally and the lessons learned are not shared in peer-reviewed literature. It is likely that this topical review will further stimulate the stroke community to disseminate related activities and use keywords such as learning health system so that the evidence base can be more readily identified.

Improving Stroke Measure Compliance and Outcomes Through Hospital Collaboration

Globally, national stroke registries have been shown to improve the quality of patient care and outcomes. However, registry utilization and implementation vary by country. In the United States, stroke-specific performance measures must be met to achieve and maintain stroke center certification awarded by the state or nationally accredited certifying bodies. The 2 stroke registries available in the United States are the American Heart Association Get With The Guidelines-Stroke registry, which is voluntary, and the Paul Coverdell National Acute Stroke Registry, funded competitively to states by the Centers for Disease Control and Prevention. Compliance with stroke processes of care is variable, and quality improvement initiatives among organizations have been shown to have an impact on improving stroke care delivery. However, the effectiveness of interorganizational continuous quality improvement approaches, especially among competing institutions, to improving stroke care is ambiguous, and no uniform governance for successful interhospital collaboration has been identified. The purpose of this article is to review national initiatives focused on interorganizational collaboration to improve stroke care delivery with a focus on interhospital collaboration in the United States to improve stroke performance measures specific to stroke center certification. The state of Kentucky's experience and utilization of the Institute for Healthcare Improvement Breakthrough Series model with key strategies for success will be discussed to serve as a foundation and empower novice stroke leaders in learning health systems. The models may be adapted internationally for application to stroke-specific care process improvement locally, regionally, and nationally; among organizations within the same health system or competing systems; and among organizations with funding or without funding to improve stroke performance measures.

Exploring Patient Journey Mapping and the Learning Health System: Scoping Review

BACKGROUND

Journey maps are visualization tools that can facilitate the diagrammatical representation of stakeholder groups by interest or function for comparative visual analysis. Therefore, journey maps can illustrate intersections and relationships between organizations and consumers using products or services. We propose that some synergies may exist between journey maps and the concept of a learning health system (LHS). The overarching goal of an LHS is to use health care data to inform clinical practice and improve service delivery processes and patient outcomes.

OBJECTIVE

The purpose of this review was to assess the literature and establish a relationship between journey mapping techniques and LHSs. Specifically, in this study, we explored the current state of the literature to answer the following research questions: (1) Is there a relationship between journey mapping techniques and an LHS in the literature? (2) Is there a way to integrate the data from journey mapping activities into an LHS? (3) How can the data gleaned from journey map activities be used to inform an LHS?

METHODS

A scoping review was conducted by querying the following electronic databases: Cochrane Database of Systematic Reviews (Ovid), IEEE Xplore, PubMed, Web of Science, Academic Search Complete (EBSCOhost), APA PsycInfo (EBSCOhost), CINAHL (EBSCOhost), and MEDLINE (EBSCOhost). Two researchers applied the inclusion criteria and assessed all articles by title and abstract in the first screen, using Covidence. Following this, a full-text review of included articles was done, with relevant data extracted, tabulated, and assessed thematically.

RESULTS

The initial search yielded 694 studies. Of those, 179 duplicates were removed. Following this, 515 articles were assessed during the first screening phase, and 412 were excluded, as they did not meet the inclusion criteria. Next, 103 articles were read in full, and 95 were excluded, resulting in a final sample of 8 articles that satisfied the inclusion criteria. The article sample can be subsumed into 2 overarching themes: (1) the need to evolve service delivery models in health care, and (2) the potential value of using patient journey data in an LHS.

CONCLUSIONS

This scoping review demonstrated the gap in knowledge regarding integrating the data from journey mapping activities into an LHS. Our findings highlighted the importance of using the data from patient experiences to enrich an LHS and provide holistic care. To satisfy this gap, the authors intend to continue this investigation to establish the relationship between journey mapping and the concept of LHSs. This scoping review will serve as phase 1 of an investigative series. Phase 2 will entail the creation of a holistic framework to guide and streamline data integration from journey mapping activities into an LHS. Lastly, phase 3 will provide a proof of concept to demonstrate how patient journey mapping activities could be integrated into an LHS.

Competitive intelligence: A precursor to a learning health system

Unlike other developed countries, the US healthcare system is largely privatized and highly competitive. This dynamic stifles effective information sharing, while the need for prompt and accurate evidence-based decision making has become crucial. Crises, like the COVID-19 pandemic, elevate the importance of quality decision making and exacerbate issues associated with the lack of a cohesive system to share information. Competitive intelligence (CI) is a discipline that encourages gathering, analyzing, and sharing information throughout a firm in order to develop and sustain competitive advantage. CI could be considered a precursor in establishing a learning organization (LO). Although CI research has focused on its process and value, little is found in the literature on how to integrate CI into an organization; this is particularly true in healthcare. A conceptual model is proposed to build and integrate a CI function and culture within a healthcare organization to encourage effective information sharing and knowledge development. In turn, this can provide a mechanism to create a learning health system (LHS). Although the model was developed specifically for US healthcare, it offers application to healthcare in other countries as well as most any industry.

Setting up a Governance Framework for Secondary Use of Routine Health Data in Nursing Homes: Development Study Using Qualitative Interviews

BACKGROUND

In the nursing home sector, reusing routinely recorded data from electronic health records (EHRs) for knowledge development and quality improvement is still in its infancy. Trust in appropriate and responsible reuse is crucial for patients and nursing homes deciding whether to share EHR data for these purposes. A data governance framework determines who may access the data, under what conditions, and for what purposes. This can help obtain that trust. Although increasing attention is being paid to data governance in the health care sector, little guidance is available on development and implementation of a data governance framework in practice.

OBJECTIVE

This study aims to describe the development process of a governance framework for the "Registry Learning from Data in Nursing Homes," a national registry for EHR data on care delivered by nursing home physicians (in Dutch: specialist ouderengeneeskunde) in Dutch nursing homes-to allow data reusage for research and quality improvement of care.

METHODS

Relevant stakeholders representing practices, policies, and research in the nursing home sector were identified. Semistructured interviews were conducted with 20 people from 14 stakeholder organizations. The main aim of the interviews was to explore stakeholders' perspectives regarding the Registry's aim, data access criteria, and governing bodies' tasks and composition. Interview topics and analyses were guided by 8 principles regarding governance for reusing health data, as described in the literature. Interview results, together with legal advice and consensus discussions by the Registry's consortium partners, were used to shape the rules, regulations, and governing bodies of the governance framework.

RESULTS

Stakeholders valued the involvement of nursing home residents and their representatives, nursing home physicians, nursing homes' boards of directors, and scientists and saw this as a prerequisite for a trustworthy data governance framework. For the Registry, involvement of these groups can be achieved through a procedure in which residents can provide their consent or objection to the reuse of the data, transparency about the decisions made, and providing them a position in a governing body. In addition, a data request approval procedure based on predefined assessment criteria indicates that data reuse by third parties aligns with the aims of the Registry, benefits the nursing home sector, and protects the privacy of data subjects.

CONCLUSIONS

The stakeholders' views, expertise, and knowledge of other frameworks and relevant legislation serve to inform the application of governance principles to the contexts of both the nursing home sector and the Netherlands. Many different stakeholders were involved in the development of the Registry Learning from Data in Nursing Homes' governance framework and will continue to be involved. Engagement of the full range of stakeholders in an early stage of governance framework development is important to generate trust in appropriate and responsible data reuse.

A validated artificial intelligence-based pipeline for population-wide primary immunodeficiency screening

BACKGROUND

Identification of patients with underlying inborn errors of immunity and inherent susceptibility to infection remains challenging. The ensuing protracted diagnostic odyssey for such patients often results in greater morbidity and suboptimal outcomes, underscoring a need to develop systematic methods for improving diagnostic rates.

OBJECTIVE

The principal aim of this study is to build and validate a generalizable analytical pipeline for population-wide detection of infection susceptibility and risk of primary immunodeficiency.

METHODS

This prospective, longitudinal cohort study coupled weighted rules with a machine learning classifier for risk stratification. Claims data were analyzed from a diverse population (n = 427,110) iteratively over 30 months. Cohort outcomes were enumerated for new diagnoses, hospitalizations, and acute care visits. This study followed TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) standards.

RESULTS

Cohort members initially identified as high risk were proportionally more likely to receive a diagnosis of primary immunodeficiency compared to those at low-medium risk or those without claims of interest respectively (9% vs 1.5% vs 0.2%; P < .001, chi-square test). Subsequent machine learning stratification enabled an annualized individual snapshot of complexity for triaging referrals. This study's top-performing machine learning model for visit-level prediction used a single dense layer neural network architecture (area under the receiver-operator characteristic curve = 0.98; F1 score = 0.98).

CONCLUSIONS

A 2-step analytical pipeline can facilitate identification of individuals with primary immunodeficiency and accurately quantify clinical risk.

Creating Value Through Learning Health Systems: The Alberta Strategic Clinical Network Experience

Design, implementation, and evaluation of effective multicomponent interventions typically take decades before value is realized even when value can be measured. Value-based health care, an approach to improving patient and health system outcomes, is a way of organizing health systems to transform outcomes and achieve the highest quality of care and the best possible outcomes with the lowest cost. We describe 2 case studies of value-based health care optimized through a learning health system framework that includes Strategic Clinical Networks. Both cases demonstrate the acceleration of evidence to practice through scientific, financial, structural administrative supports and partnerships. Clinical practice interventions in both cases, one in perioperative services and the other in neonatal intensive care, were implemented across multiple hospital sites. The practical application of using an innovation pipeline as a structural process is described and applied to these cases. A value for money improvement calculator using a benefits realization approach is presented as a mechanism/tool for attributing value to improvement initiatives that takes advantage of available system data, customizing and making the data usable for frontline managers and decision makers. Health care leaders will find value in the descriptions and practical information provided.

The Alliance for Healthier Communities as a Learning Health System for primary care: A qualitative analysis in Ontario, Canada

RATIONALE, AIMS AND OBJECTIVES

A learning health system model can be used to efficiently evaluate and incorporate evidence-based care into practice. However, there is a paucity of evidence describing key organizational attributes needed to ensure a successful learning health system within primary care. We interviewed stakeholders for a primary care learning health system in Ontario, Canada (the Alliance for Healthier Communities) to identify strengths and areas for improvement.

METHOD

We conducted a qualitative descriptive study using individual semistructured interviews with Alliance stakeholders between December 2019 and March 2020. The Alliance delivers community-governed primary healthcare through 109 organizations including Community Health Centres (CHCs). All CHC staff within the Alliance were invited to participate. Interviews were audio-recorded and transcribed verbatim. We performed a thematic analysis using a team approach.

RESULTS

We interviewed 29 participants across six CHCs, including Executive Directors, managers, healthcare providers and data support staff. We observed three foundational elements necessary for a successful learning health system within primary care: shared organizational goals and culture, data quality and resources. Building on this foundation, people are needed to drive the learning health system, and this is conditional on their level of engagement. The main factors motivating staff member's engagement with the learning health system included their drive to help improve patient care, focusing on initiatives of personal interest and understanding the purpose of different initiatives. Areas for improvement were identified such as the ability to extract and use data to inform changes in real-time, better engagement and protected time for providers to do improvement work, and more staff dedicated to data extraction and analysis.

CONCLUSIONS

We identified key components needed to establish a learning health system in primary care. Similar primary care organizations in Canada and elsewhere can use these insights to guide their development as learning health systems.

A conceptual framework to describe and evaluate a socially accountable learning health system: Development and application in a northern, rural, and remote setting

Health care and academic institutions are increasingly committing to social accountability, a strategic shift that requires priorities, activities, and evaluations to be co-determined with all relevant partners. Consequently, governments, accreditors, funders, and communities are calling for these institutions to communicate their progress towards social accountability. The purpose of this study was to develop a conceptual framework around a socially accountable learning health system. This article presents an integrated analysis of two studies: (i) a narrative review of 11 prominent social accountability and health services conceptual frameworks and (ii) a reflexive thematic analysis of 18 key informant interviews. Using a systematic conceptual framework development and integrated theory of change/realist evaluation methodologies, we describe a synthesis of these findings to develop a conceptual framework for describing and evaluating socially accountable health professional education. The resulting framework describes assessment phases of social accountability, transitions between phases, learning cycles, and the actors and systems that collectively mobilise social accountability at multiple levels in health and education systems. The framework can be used to evaluate interventions or characterise progress towards social accountability in different settings, as illustrated in the example at the end of the paper. The framework emphasises the significance of designing, mobilising, and evaluating social accountability as part of a contextualised learning health system.

The distillation method: A novel approach for analyzing randomized trials when exposure to the intervention is diluted

OBJECTIVE

To introduce a novel analytical approach for randomized controlled trials that are underpowered because of low participant enrollment or engagement.

DATA SOURCES

Reanalysis of data for 805 patients randomized as part of a pilot complex care intervention in 2015-2016 in a large delivery system. In the pilot randomized trial, only 64.6% of patients assigned to the intervention group participated.

STUDY DESIGN

A case study and simulation. The "Distillation Method" capitalizes on the frequently observed correlation between the probability of subjects' participation or engagement in the intervention and the magnitude of benefit they experience. The novel method involves three stages: first, it uses baseline covariates to generate predicted probabilities of participation. Next, these are used to produce nested subsamples of the randomized intervention and control groups that are more concentrated with subjects who were likely to participate/engage. Finally, for the outcomes of interest, standard statistical methods are used to re-evaluate intervention effectiveness in these concentrated subsets.

DATA EXTRACTION METHODS

We assembled secondary data on patients who were randomized to the pilot intervention for one year prior to randomization and two follow-up years. Data included program enrollment status, membership data, demographics, utilization, costs, and clinical data.

PRINCIPAL FINDINGS

Using baseline covariates only, Generalized Boosted Regression Models predicting program enrollment performed well (AUC 0.884). We then distilled the full randomized sample to increasing levels of concentration and reanalyzed program outcomes. We found statistically significant differences in outpatient utilization and emergency department utilization (both follow-up years), and in total costs (follow-up year two only) at select levels of population concentration.

CONCLUSIONS

By offering an internally valid analytic framework, the Distillation Method can increase the power to detect effects by redefining the estimand to subpopulations with higher enrollment probabilities and stronger average treatment effects while maintaining the original randomization.

The Penn Medicine BioBank: Towards a Genomics-Enabled Learning Healthcare System to Accelerate Precision Medicine in a Diverse Population

The Penn Medicine BioBank (PMBB) is an electronic health record (EHR)-linked biobank at the University of Pennsylvania (Penn Medicine). A large variety of health-related information, ranging from diagnosis codes to laboratory measurements, imaging data and lifestyle information, is integrated with genomic and biomarker data in the PMBB to facilitate discoveries and translational science. To date, 174,712 participants have been enrolled into the PMBB, including approximately 30% of participants of non-European ancestry, making it one of the most diverse medical biobanks. There is a median of seven years of longitudinal data in the EHR available on participants, who also consent to permission to recontact. Herein, we describe the operations and infrastructure of the PMBB, summarize the phenotypic architecture of the enrolled participants, and use body mass index (BMI) as a proof-of-concept quantitative phenotype for PheWAS, LabWAS, and GWAS. The major representation of African-American participants in the PMBB addresses the essential need to expand the diversity in genetic and translational research. There is a critical need for a "medical biobank consortium" to facilitate replication, increase power for rare phenotypes and variants, and promote harmonized collaboration to optimize the potential for biological discovery and precision medicine.

The US Food and Drug Administration Sentinel System: a national resource for a learning health system

The US Food and Drug Administration (FDA) created the Sentinel System in response to a requirement in the FDA Amendments Act of 2007 that the agency establish a system for monitoring risks associated with drug and biologic products using data from disparate sources. The Sentinel System has completed hundreds of analyses, including many that have directly informed regulatory decisions. The Sentinel System also was designed to support a national infrastructure for a learning health system. Sentinel governance and guiding principles were designed to facilitate Sentinel's role as a national resource. The Sentinel System infrastructure now supports multiple non-FDA projects for stakeholders ranging from regulated industry to other federal agencies, international regulators, and academics. The Sentinel System is a working example of a learning health system that is expanding with the potential to create a global learning health system that can support medical product safety assessments and other research.

Describing a complex primary health care population to support future decision support initiatives

Introduction

Developing decision support tools using data from a health care organization, to support care within that organization, is a promising paradigm to improve care delivery and population health. Descriptive epidemiology may be a valuable supplement to stakeholder input towards selection of potential initiatives and to inform methodological decisions throughout tool development. We additionally propose that to properly characterize complex populations in large-scale descriptive studies, both simple statistical and machine learning techniques can be useful.

Objective

To describe sociodemographic, clinical, and health care use characteristics of primary care clients served by the Alliance for Healthier Communities, which provides team-based primary health care through Community Health Centres (CHCs) across Ontario, Canada.

Methods

We used electronic health record data from adult ongoing primary care clients served by CHCs in 2009-2019. We performed traditional table-based summaries for each characteristic; and applied three unsupervised learning techniques to explore patterns of common condition co-occurrence, care provider teams, and care frequency.

Results

There were 221,047 eligible clients. Sociodemographics: We described 13 characteristics, stratified by CHC type and client multimorbidity status. Clinical characteristics: Eleven-year prevalence of 24 investigated conditions ranged from 1% (Hepatitis C) to 63% (chronic musculoskeletal problem) with non-uniform risk across the care history; multimorbidity was common (81%) with variable co-occurrence patterns. Health care use characteristics: Most care was provided by physician and nursing providers, with heterogeneous combinations of other provider types. A subset of clients had many issues addressed within single-visits and there was within- and between-client variability in care frequency. In addition to substantive findings, we discuss methodological considerations for future decision support initiatives.

Conclusions

We demonstrated the use of methods from statistics and machine learning, applied with an epidemiological lens, to provide an overview of a complex primary care population and lay a foundation for stakeholder engagement and decision support tool development.

Patient-reported outcomes via electronic health record portal versus telephone: a pragmatic randomized pilot trial of anxiety or depression symptoms in epilepsy

Objective

To close gaps between research and clinical practice, tools are needed for efficient pragmatic trial recruitment and patient-reported outcome collection. The objective was to assess feasibility and process measures for patient-reported outcome collection in a randomized trial comparing electronic health record (EHR) patient portal questionnaires to telephone interview among adults with epilepsy and anxiety or depression symptoms.

Materials and Methods

Recruitment for the randomized trial began at an epilepsy clinic visit, with EHR-embedded validated anxiety and depression instruments, followed by automated EHR-based research screening consent and eligibility assessment. Fully eligible individuals later completed telephone consent, enrollment, and randomization. Participants were randomized 1:1 to EHR portal versus telephone outcome assessment, and patient-reported and process outcomes were collected at 3 and 6 months, with primary outcome 6-month retention in EHR arm (feasibility target: ≥11 participants retained).

Results

Participants (N = 30) were 60% women, 77% White/non-Hispanic, with mean age 42.5 years. Among 15 individuals randomized to EHR portal, 10 (67%, CI 41.7%-84.8%) met the 6-month retention endpoint, versus 100% (CI 79.6%-100%) in the telephone group (P = 0.04). EHR outcome collection at 6 months required 11.8 min less research staff time per participant than telephone (5.9, CI 3.3-7.7 vs 17.7, CI 14.1-20.2). Subsequent telephone contact after unsuccessful EHR attempts enabled near complete data collection and still saved staff time.

Discussion

In this randomized study, EHR portal outcome assessment did not meet the retention feasibility target, but EHR method saved research staff time compared to telephone.

Conclusion

While EHR portal outcome assessment was not feasible, hybrid EHR/telephone method was feasible and saved staff time.

Evaluating a learning health system initiative: Lessons learned during COVID-19 in Saskatchewan, Canada

Introduction

Evaluating a learning health system (LHS) encourages continuous system improvement and collaboration within the healthcare system. Although LHS is a widely accepted concept, there is little knowledge about evaluating an LHS. To explore the outputs and outcomes of an LHS model, we evaluated the COVID-19 Evidence Support Team (CEST) in Saskatchewan, Canada, an initiative to rapidly review scientific evidence about COVID-19 for decision-making. By evaluating this program during its formation, we explored how and to what extent the CEST initiative was used by stakeholders. An additional study aim was to understand how CEST could be applied as a functional LHS and the value of similar knowledge-to-action cycles.

Methods

Using a formative evaluation design, we conducted qualitative interviews with key informants (KIs) who were involved with COVID-19 response strategies in Saskatchewan. Transcripts were analyzed using reflexive thematic analysis to identify key themes. A program logic model was created to represent the inputs, activities, outputs, and outcomes of the CEST initiative.

Results

Interview data from 11 KIs were collated under three overarching categories: (1) outputs, (2) short-term outcomes, and (3) long-term outcomes from the CEST initiative. Overall, participants found the CEST initiative improved speed and access to reliable information, supported and influenced decision-making and public health strategies, leveraged partnerships, increased confidence and reassurance, and challenged misinformation. Themes relating to the long-term outcomes of the initiative included improving coordination, awareness, and using good judgment and planning to integrate CEST sustainably into the health system.

Conclusion

This formative evaluation demonstrated that CEST was a valued program and a promising LHS model for Saskatchewan. The future direction involves addressing program recommendations to implement this model as a functional LHS in Saskatchewan.

A call to integrate health equity into learning health system research training

In 2016, the Agency for Healthcare Research and Quality (AHRQ) recommended seven domains for training and mentoring researchers in learning health systems (LHS) science. Health equity was not included as a competency domain. This commentary from scholars in the Consortium for Applied Training to Advance the Learning health system with Scholars/Trainees (CATALyST) K12 program recommends that competency domains be extended to reflect growing demands for evidence on health inequities and interventions to alleviate them. We present real-life case studies from scholars in an LHS research training program that illustrate facilitators, challenges, and potential solutions at the program, funder, and research community-level to receiving training and mentorship in health equity-focused LHS science. We recommend actions in four areas for LHS research training programs: (a) integrate health equity throughout the current LHS domains; (b) develop training and mentoring in health equity; (c) establish program evaluation standards for consideration of health equity; and (d) bring forth relevant, extant expertise from the areas of health disparities research, community-based participatory research, and community-engaged health services research. We emphasize that LHS research must acknowledge and build on the substantial existing contributions, mainly by scholars of color, in the health equity field.

Patient-Reported Outcome Dashboards Within the Electronic Health Record to Support Shared Decision-making: Protocol for Co-design and Clinical Evaluation With Patients With Advanced Cancer and Chronic Kidney Disease

BACKGROUND

Patient-reported outcomes-symptoms, treatment side effects, and health-related quality of life-are important to consider in chronic illness care. The increasing availability of health IT to collect patient-reported outcomes and integrate results within the electronic health record provides an unprecedented opportunity to support patients' symptom monitoring, shared decision-making, and effective use of the health care system.

OBJECTIVE

The objectives of this study are to co-design a dashboard that displays patient-reported outcomes along with other clinical data (eg, laboratory tests, medications, and appointments) within an electronic health record and conduct a longitudinal demonstration trial to evaluate whether the dashboard is associated with improved shared decision-making and disease management outcomes.

METHODS

Co-design teams comprising study investigators, patients with advanced cancer or chronic kidney disease, their care partners, and their clinicians will collaborate to develop the dashboard. Investigators will work with clinic staff to implement the co-designed dashboard for clinical testing during a demonstration trial. The primary outcome of the demonstration trial is whether the quality of shared decision-making increases from baseline to the 3-month follow-up. Secondary outcomes include longitudinal changes in satisfaction with care, self-efficacy in managing treatments and symptoms, health-related quality of life, and use of costly and potentially avoidable health care services. Implementation outcomes (ie, fidelity, appropriateness, acceptability, feasibility, reach, adoption, and sustainability) during the co-design process and demonstration trial will also be collected and summarized.

RESULTS

The dashboard co-design process was completed in May 2020, and data collection for the demonstration trial is anticipated to be completed by the end of July 2022. The results will be disseminated in at least one manuscript per study objective.

CONCLUSIONS

This protocol combines stakeholder engagement, health care coproduction frameworks, and health IT to develop a clinically feasible model of person-centered care delivery. The results will inform our current understanding of how best to integrate patient-reported outcome measures into clinical workflows to improve outcomes and reduce the burden of chronic disease on patients and health care systems.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/38461.