Paper 4: Search methods and approaches for living guidelines

Assessing the coverage of PubMed, Embase, OpenAlex, and Semantic Scholar for automated single-database searches in living guideline evidence surveillance: a case study of the international polycystic ovary syndrome guidelines 2023

OBJECTIVES

Living guideline maintenance is underpinned by manual approaches toward evidence retrieval, limiting long-term sustainability. Our study aimed to evaluate the feasibility of using only PubMed, Embase, OpenAlex, or Semantic Scholar in automatically retrieving articles that were included in a high-quality international guideline - the 2023 international polycystic ovary syndrome (PCOS) guidelines.

METHODS

The digital object identifiers (DOIs) and PubMed ID (PMIDs) of articles included after full-text screening in the 2023 international PCOS guidelines were extracted. These IDs were used to automatically retrieve article metadata from all tested databases. A title-only search was then conducted on articles that were not initially retrievable. The extent of coverage, and overlap of coverage, was determined for each database. An exploratory analysis of the risk of bias (RoB) of articles that were unretrievable was then conducted for each database.

RESULTS

OpenAlex had the best coverage (98.6%), followed by Semantic Scholar (98.3%), Embase (96.8%), and PubMed (93.0%). However, 90.5% of all articles were retrievable from all four databases. All articles that were not retrievable from OpenAlex and Semantic Scholar were either assessed as medium or high RoB. In contrast, both Embase and PubMed missed articles that were of high quality (low RoB).

CONCLUSION

OpenAlex should be considered a single source for automated evidence retrieval in living guideline development, due to high coverage, and low risk of missing high-quality articles. These insights are being leveraged as part of transitioning the 2023 international PCOS guidelines toward a living format.

A living critical interpretive synthesis to yield a framework on the production and dissemination of living evidence syntheses for decision-making

BACKGROUND

The COVID-19 pandemic has had an unprecedented impact in the global research production and has also increased research waste. Living evidence syntheses (LESs) seek to regularly update a body of evidence addressing a specific question. During the COVID-19 pandemic, the production and dissemination of LESs emerged as a cornerstone of the evidence infrastructure. This critical interpretive synthesis answers the questions: What constitutes an LES to support decision-making?; when should one be produced, updated, and discontinued?; and how should one be disseminated?

METHODS

Searches included the Cochrane Library, EMBASE (Ovid), Health Systems Evidence, MEDLINE (Ovid), PubMed, and Web of Science up to 23 April 2024 and included articles that provide any insights on addressing the compass questions on LESs. Articles were selected and appraised, and their insights extracted. An interpretive and iterative coding process was used to identify relevant thematic categories and create a conceptual framework.

RESULTS

Among the 16,630 non-duplicate records identified, 208 publications proved eligible. Most were non-empirical articles, followed by actual LESs. Approximately one in three articles were published in response to the COVID-19 pandemic. The conceptual framework addresses six thematic categories: (1) what is an LES; (2) what methodological approaches facilitate LESs production; (3) when to produce an LES; (4) when to update an LES; (5) how to make available the findings of an LES; and (6) when to discontinue LES updates.

CONCLUSION

LESs can play a critical role in reducing research waste and ensuring alignment with advisory and decision-making processes. This critical interpretive synthesis provides relevant insights on how to better organize the global evidence architecture to support their production.

TRIAL REGISTRATION

PROSPERO registration: CRD42021241875.

Evidence surveillance for a living clinical guideline: Case study of the Australian stroke guidelines

BACKGROUND

Continual evidence surveillance is an integral feature of living guidelines. The Australian Stroke Guidelines include recommendations on 100 clinical topics and have been 'living' since 2018.

OBJECTIVES

To describe the approach for establishing and evaluating an evidence surveillance system for the living Australian Stroke Guidelines.

METHODS

We developed a pragmatic surveillance system based on an analysis of the searches for the 2017 Stroke Guidelines and evaluated its reliability by assessing the potential impact on guideline recommendations. Search retrieval and screening workload are monitored monthly, together with the frequency of changes to the guideline recommendations.

RESULTS

Evidence surveillance was guided by practical considerations of efficiency and sustainability. A single PubMed search covering all guideline topics, limited to systematic reviews and randomised trials, is run monthly. The search retrieves about 400 records a month of which a sixth are triaged to the guideline panels for further consideration. Evaluations with Epistemonikos and the Cochrane Stroke Trials Register demonstrated the robustness of adopting this more restrictive approach. Collaborating with the guideline team in designing, implementing and evaluating the surveillance is essential for optimising the approach.

CONCLUSION

Monthly evidence surveillance for a large living guideline is feasible and sustainable when applying a pragmatic approach.

What could health technology assessment learn from living clinical practice guidelines?

A "living" approach to clinical practice guidelines is when the identification, appraisal and synthesis of evidence is maintained and repeated at an agreed frequency, with a clear process for when and how new evidence is to be incorporated. The value of a living approach to guidelines was emphasised during the COVID-19 pandemic when health professionals and policymakers needed to make decisions regarding patient care in the context of a nascent but rapidly evolving evidence base. In this perspective, we draw on our recent experience developing Australian and international living guidelines and reflect on the feasibility of applying living guideline methods and processes to a lifecycle approach to health technology assessment (HTA). We believe the opportunities and challenges of adopting a living approach in HTA fall into five key themes: identification, appraisal and synthesis of evidence; optimising the frequency of updates; embedding ongoing multi-stakeholder engagement; linking the emergence of new evidence to reimbursement; and system capacity to support a living approach. We acknowledge that the suitability of specific living approaches to HTA will be heavily influenced by the type of health technology, its intended use in the health system, local reimbursement pathways, and other policy settings. But we believe that the methods and processes applied successfully to guideline development to manage evidentiary uncertainty could be applied in the context of HTA and reimbursement decision-making to help manage similar sources of uncertainty.

Weekly updating of guideline recommendations was feasible: the Australian National COVID-19 clinical evidence Taskforce

OBJECTIVES

To investigate how quickly evidence was incorporated into the Australian living guidelines for COVID-19 during the first 12 months of the pandemic.

STUDY DESIGN AND SETTING

For each study concerning drug therapies included in the guideline from April 3, 2020 to April 1, 2021, we extracted the publication date of the study, and the guideline version the study was included in. We analyzed two subgroups of studies as follows: those published in high impact factor journals and those with 100 or more participants.

RESULTS

In the first year, we published 37 major versions of the guidelines, incorporating 129 studies that investigated 48 drug therapies informing 115 recommendations. The median time from first publication of a study to incorporation in the guideline was 27 days (interquartile range [IQR], 16 to 44), ranging from 9 to 234 days. For the 53 studies in the highest impact factor journals, the median was 20 days (IQR 15 to 30), and for the 71 studies with 100 or more participants the median was 22 days (IQR 15 to 36).

CONCLUSION

Developing and sustaining living guidelines where evidence is rapidly incorporated is a resource- and time-intensive undertaking; however, this study demonstrates that it is feasible, even over a long period.

Methods for living guidelines: early guidance based on practical experience. Paper 3: selecting and prioritizing questions for living guidelines

OBJECTIVES

This article is part of a series on methods for living guidelines, consolidating practical experiences from developing living guidelines. It focuses on methods for identification, selection, and prioritization of clinical questions for a living approach to guideline development.

STUDY DESIGN AND SETTING

Members of the Australian Living Evidence Consortium, the National Institute of Health and Care Excellence and the US Grading of Recommendations, Assessment, Development and Evaluations Network, convened a working group. All members have expertize and practical experience in the development of living guidelines. We collated methods, documents on prioritization from each organization's living guidelines, conducted interviews and held working group discussions. We consolidated these to form best practice principles which were then edited and agreed on by the working group members.

RESULTS

We developed best practice principles for (1) identification, (2) selection, and (3) prioritization, of questions for a living approach to guideline development. Several different strategies for undertaking prioritizing questions are explored.

CONCLUSION

The article provides guidance for prioritizing questions in living guidelines. Subsequent articles in this series explore consumer involvement, search decisions, and methods decisions that are appropriate for questions with different priority levels.