A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

Performance of two large language models for data extraction in evidence synthesis

Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT-4) for extracting pre-specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT-4. GPT-4 required use of a third-party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT-4 with the plug-in was lower (68.8%); however, most of the errors were due to the plug-in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT-4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open-source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Data extraction for evidence synthesis using a large language model: A proof-of-concept study

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

What are the best methods for rapid reviews of the research evidence? A systematic review of reviews and primary studies

Rapid review methodology aims to facilitate faster conduct of systematic reviews to meet the needs of the decision-maker, while also maintaining quality and credibility. This systematic review aimed to determine the impact of different methodological shortcuts for undertaking rapid reviews on the risk of bias (RoB) of the results of the review. Review stages for which reviews and primary studies were sought included the preparation of a protocol, question formulation, inclusion criteria, searching, selection, data extraction, RoB assessment, synthesis, and reporting. We searched 11 electronic databases in April 2022, and conducted some supplementary searching. Reviewers worked in pairs to screen, select, extract data, and assess the RoB of included reviews and studies. We included 15 systematic reviews, 7 scoping reviews, and 65 primary studies. We found that several commonly used shortcuts in rapid reviews are likely to increase the RoB in the results. These include restrictions based on publication date, use of a single electronic database as a source of studies, and use of a single reviewer for screening titles and abstracts, selecting studies based on the full-text, and for extracting data. Authors of rapid reviews should be transparent in reporting their use of these shortcuts and acknowledge the possibility of them causing bias in the results. This review also highlights shortcuts that can save time without increasing the risk of bias. Further research is needed for both systematic and rapid reviews on faster methods for accurate data extraction and RoB assessment, and on development of more precise search strategies.

Data extraction error in pharmaceutical versus non-pharmaceutical interventions for evidence synthesis: Study protocol for a crossover trial

Background

Data extraction is the foundation for research synthesis evidence, while data extraction errors frequently occur in the literature. An interesting phenomenon was observed that data extraction error tend to be more common in trials of pharmaceutical interventions compared to non-pharmaceutical ones. The elucidation of which would have implications for guidelines, practice, and policy.

Methods and analyses

We propose a crossover, multicenter, investigator-blinded trial to elucidate the potential variants on the data extraction error rates. Eligible 90 participants would be 2nd year or above post-graduate students (e.g., masters, doctoral program). Participants will be randomized to one of the two groups to complete pre-defined data extraction tasks: 1) group A will contain 10 randomized controlled trials (RCTs) of pharmaceutical interventions; 2) group B will contain 10 RCTs of non-pharmaceutical interventions. Participants who finish the data extraction would then be assigned to the alternative group for another round of data extraction after a 30 min washout period. Finally, those participants assigned to A or B group will be further 1:1 randomly matched based on a random-sequenced number for the double-checking process on the extracted data. The primary outcome will be the data extract error rates of the pharmaceutical intervention group and non-pharmaceutical group, before the double-checking process, in terms of the cell level, study level, and participant level. The secondary outcome will be the data error rates of the pharmaceutical intervention group and non-pharmaceutical group after the double-checking process, again, in terms of the cell level, study level, and participant level. A generalized linear mixed effects model (based on the above three levels) will be used to estimate the potential differences in the error rates, with a log link function for binomial data. Subgroup analyses will account for the experience of individuals on systematic reviews and the time used for the data extraction.

Discussion

This trial will provide useful evidence for further systematic review of data extraction practices, improved data extraction strategies, and better guidelines.

Trial registration

Chinese Clinical Trial Register Center (Identifier: ChiCTR2200062206).

Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey

OBJECTIVE

Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs.

STUDY DESIGN AND SETTING

We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis.

RESULTS

162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%).

CONCLUSION

Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.

TFOS Lifestyle - Evidence quality report: Advancing the evaluation and synthesis of research evidence

Evidence-based practice is a dominant paradigm in healthcare that emphasizes the importance of ensuring the translation of the best available, relevant research evidence into practice. An Evidence Quality Subcommittee was established to provide specialized methodological support and expertise to promote rigorous and evidence-based approaches for the Tear Film and Ocular Surface Society (TFOS) Lifestyle Epidemic reports. The present report describes the purpose, scope, and activity of the Evidence Quality Subcommittee in the undertaking of high-quality narrative-style literature reviews, and leading prospectively registered, reliable systematic reviews of high priority research questions, using standardized methods for each topic area report. Identification of predominantly low or very low certainty evidence across the eight systematic reviews highlights a need for further research to define the efficacy and/or safety of specific lifestyle interventions on the ocular surface, and to clarify relationships between certain lifestyle factors and ocular surface disease. To support the citation of reliable systematic review evidence in the narrative review sections of each report, the Evidence Quality Subcommittee curated topic-specific systematic review databases and relevant systematic reviews underwent standardized reliability assessment. Inconsistent methodological rigor was noted in the published systematic review literature, emphasizing the importance of internal validity assessment. Based on the experience of implementing the Evidence Quality Subcommittee, this report makes suggestions for incorporation of such initiatives in future international taskforces and working groups. Content areas broadly relevant to the activity of the Evidence Quality Subcommittee, including the critical appraisal of research, clinical evidence hierarchies (levels of evidence), and risk of bias assessment, are also outlined.

Design and Conduct of Randomized Clinical Trials Evaluating Surgical Innovations in Ophthalmology: A Systematic Review

PURPOSE

Surgical innovations are necessary to improve patient care. After an initial exploratory phase, novel surgical technique should be compared with alternative options or standard care in randomized controlled trials (RCTs). However, surgical RCTs have unique methodological challenges. Our study sought to investigate key aspects of the design, conduct, and reporting of RCTs of novel surgeries.

DESIGN

Systematic review.

METHODS

The protocol was prospectively registered in PROSPERO (CRD42021253297). RCTs evaluating novel surgeries for cataract, vitreoretinal, glaucoma, and corneal diseases were included. Medline, EMBASE, Cochrane Library, and Clinicaltrials.gov were searched. The search period was January 1, 2016, to June 16, 2021.

RESULTS

A total of 52 ophthalmic surgery RCTs were identified in the fields of glaucoma (n = 12), vitreoretinal surgery (n = 5), cataract (n = 19), and cornea (n = 16). A description defining the surgeon's experience or level of expertise was reported in 30 RCTs (57%) and was presented in both control and intervention groups in 11 (21%). Specification of the number of cases performed in the particular surgical innovation being assessed prior to the trial was reported in 10 RCTs (19%) and an evaluation of quality of the surgical intervention in 7 (13%). Prospective trial registration was recorded in 12 RCTs (23%) and retrospective registration in 13 (25%); and there was no registration record in the remaining 28 (53%) studies.

CONCLUSIONS

Important aspects of the study design such as the surgical learning curve, surgeon's previous experience, quality assurance, and trial registration details were often missing in novel ophthalmic surgical procedures. The Idea, Development, Exploration, Assessment, Long-term follow-up (IDEAL) framework aims to improve the quality of study design.

Methods for living guidelines: early guidance based on practical experience. Article 5: decisions on methods for evidence synthesis and recommendation development for living guidelines

OBJECTIVES

Producing living guidelines requires making important decisions about methods for evidence identification, appraisal, and integration to allow the living mode to function. Clarifying what these decisions are and the trade-offs between options is necessary. This article provides living guideline developers with a framework to enable them to choose the most suitable model for their living guideline topic, question, or context.

STUDY DESIGN AND SETTING

We developed this guidance through an iterative process informed by interviews, feedback, and a consensus process with an international group of living guideline developers.

RESULTS

Several key decisions need to be made both before commencing and throughout the continual process of living guideline development and maintenance. These include deciding what approach is taken to the systematic review process; decisions about methods to be applied for the evidence appraisal process, including the use of unpublished data; and selection of "triggers" to incorporate new studies into living guideline recommendations. In each case, there are multiple options and trade-offs.

CONCLUSION

We identify trade-offs and important decisions to be considered throughout the living guideline development process. The most appropriate, and most sustainable, mode of development and updating will be dependent on the choices made in each of these areas.

Data reproducibility issues and their potential impact on conclusions from evidence syntheses of randomized controlled trials in sleep medicine

In this study, we examined the data reproducibility issues in systematic reviews in sleep medicine. We searched for systematic reviews of randomized controlled trials published in sleep medicine journals. The metadata in meta-analyses among the eligible systematic reviews were collected. The original sources of the data were reviewed to see if the components used in the meta-analyses were correctly extracted or estimated. The impacts of the data reproducibility issues were investigated. We identified 48 systematic reviews with 244 meta-analyses of continuous outcomes and 54 of binary outcomes. Our results suggest that for continuous outcomes, 20.03% of the data used in meta-analyses cannot be reproduced at the trial level, and 43.44% of the data cannot be reproduced at the meta-analysis level. For binary outcomes, the proportions were 14.14% and 40.74%. In total, 83.33% of the data cannot be reproduced at the systematic review level. Our further analysis suggested that these reproducibility issues would lead to as much as 6.52% of the available meta-analyses changing the direction of the effects, and 9.78% changing the significance of the P-values. Sleep medicine systematic reviews and meta-analyses face serious issues in terms of data reproducibility, and further efforts are urgently needed to improve this situation.

Effectiveness of interventions for dry eye: a protocol for an overview of systematic reviews

INTRODUCTION

Dry eye is a leading cause of ocular morbidity and economic and societal burden for patients and healthcare systems. There are several treatment options available for dry eye and high-quality systematic reviews synthesise the evidence for their effectiveness and potential harms.

METHODS AND ANALYSIS

We will search the Cochrane Eyes and Vision US satellite (CEV@US) database of eyes and vision systematic reviews for systematic reviews on interventions for dry eye. CEV@US conducted an initial search of PubMed and Embase to populate the CEV@US database of eyes and vision systematic reviews in 2007, which was updated most recently in August 2021. We will search the database for systematic reviews published since 1 January 2016 because systematic reviews more than 5 years are unlikely to be up to date. We will consider Cochrane and non-Cochrane systematic reviews eligible for inclusion. Two authors will independently screen articles. We will include studies that evaluate interventions for dry eye and/or meibomian gland dysfunction with no restriction on types of participants or review language. We will select reliable systematic reviews (ie, those meeting pre-established methodological criteria) for inclusion, assessed by one investigator and verified by a second investigator. We will extract ratings of the certainty of evidence from within each review. We will report the degree of overlap for systematic reviews that answer similar questions and include overlapping primary studies. We will present results of the overview in alignment with guidelines in the Cochrane Handbook of Systematic Reviews of Interventions (Online Chapter 5: Overviews of Reviews), the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement, and an overview of reviews quality and transparency checklist. The anticipated start and completion dates for this overview are 1 May 2021 and 30 April 2022, respectively.

ETHICS AND DISSEMINATION

This overview will not require the approval of an Ethics Committee because it will use published studies. We will publish results in a peer-reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42021279880.

Harms in Systematic Reviews Paper 2: Methods used to assess harms are neglected in systematic reviews of gabapentin

OBJECTIVE

We compared methods used with current recommendations for synthesizing harms in systematic reviews and meta-analyses (SRMAs) of gabapentin.

STUDY DESIGN & SETTING

We followed recommended systematic review practices. We selected reliable SRMAs of gabapentin (i.e., met a pre-defined list of methodological criteria) that assessed at least one harm. We extracted and compared methods in four areas: pre-specification, searching, analysis, and reporting. Whereas our focus in this paper is on the methods used, Part 2 examines the results for harms across reviews.

RESULTS

We screened 4320 records and identified 157 SRMAs of gabapentin, 70 of which were reliable. Most reliable reviews (51/70; 73%) reported following a general guideline for SRMA conduct or reporting, but none reported following recommendations specifically for synthesizing harms. Across all domains assessed, review methods were designed to address questions of benefit and rarely included the additional methods that are recommended for evaluating harms.

CONCLUSION

Approaches to assessing harms in SRMAs we examined are tokenistic and unlikely to produce valid summaries of harms to guide decisions. A paradigm shift is needed. At a minimal, reviewers should describe any limitations to their assessment of harms and provide clearer descriptions of methods for synthesizing harms.

Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews

BACKGROUND

Previous research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews.

METHODS

We included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively.

RESULTS

Of the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%).

CONCLUSIONS

Reporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses.

Resource use during systematic review production varies widely: a scoping review

OBJECTIVE

We aimed to map the resource use during systematic review (SR) production and reasons why steps of the SR production are resource intensive to discover where the largest gain in improving efficiency might be possible.

STUDY DESIGN AND SETTING

We conducted a scoping review. An information specialist searched multiple databases (e.g., Ovid MEDLINE, Scopus) and implemented citation-based and grey literature searching. We employed dual and independent screenings of records at the title/abstract and full-text levels and data extraction.

RESULTS

We included 34 studies. Thirty-two reported on the resource use-mostly time; four described reasons why steps of the review process are resource intensive. Study selection, data extraction, and critical appraisal seem to be very resource intensive, while protocol development, literature search, or study retrieval take less time. Project management and administration required a large proportion of SR production time. Lack of experience, domain knowledge, use of collaborative and SR-tailored software, and good communication and management can be reasons why SR steps are resource intensive.

CONCLUSION

Resource use during SR production varies widely. Areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.

What Do We Really Know about the Effectiveness of Glaucoma Interventions?: An Overview of Systematic Reviews

PURPOSE

To identify systematic reviews of interventions for glaucoma conditions and to assess their reliability, thereby generating a list of potentially reliable reviews for updating glaucoma practice guidelines.

DESIGN

Cross-sectional study.

PARTICIPANTS

Systematic reviews of interventions for glaucoma conditions.

METHODS

We used a database of systematic reviews and meta-analyses in vision research and eye care maintained by the Cochrane Eyes and Vision United States Satellite. We examined all Cochrane systematic reviews of interventions for glaucoma conditions published before August 7, 2019, and all non-Cochrane systematic reviews of interventions for glaucoma conditions published between January 1, 2014, and August 7, 2019.

MAIN OUTCOME MEASURES

We assessed eligible reviews for reliability, extracted characteristics, and summarized key findings from reviews classified as reliable.

RESULTS

Of the 4451 systematic reviews in eyes and vision identified, 129 met our eligibility criteria and were assessed for reliability. Of these, we classified 49 (38%) as reliable. We found open-angle glaucoma (22/49) to be the condition with the most reviews and medical management (17/49) and intraocular pressure (IOP; 43/49) to be the most common interventions and outcomes studied. Most reviews found a high degree of uncertainty in the evidence, which hinders the possibility of making strong recommendations in guidelines. These reviews found high-certainty evidence about a few topics: reducing IOP helps to prevent glaucoma and its progression, prostaglandin analogs are the most effective medical treatment for lowering IOP, laser trabeculoplasty is as effective as medical treatment as a first-line therapy in controlling IOP, the use of IOP-lowering medications in the perioperative or postoperative periods to accompany laser (e.g., trabeculoplasty) reduces the risk of postoperative IOP spikes, conventional surgery (i.e., trabeculectomy) is more effective than medications in reducing IOP, and antimetabolites and β-radiation improve IOP control after trabeculectomy. The evidence is weak regarding the effectiveness of minimally invasive glaucoma surgeries.

CONCLUSIONS

Most systematic reviews evaluating interventions for glaucoma are of poor reliability. Even among those that may be considered reliable, important limitations exist in the value of information because of the uncertainty of the evidence as well as small and sometimes unimportant clinical differences between interventions.

Methodological options of the nominal group technique for survey item elicitation in health research: A scoping review

OBJECTIVE

To conduct a scoping review that identifies different nominal group technique (NGT) methods used to elicit items for health surveys, and their advantages and disadvantages.

STUDY DESIGN AND SETTING

We conducted a comprehensive search process from database inception to July 22, 2019 in Medline, EMBASE, PsychInfo, CINAHL, Cochrane Central and Scopus without language restriction. We screened titles and abstracts. Data from potentially relevant articles were extracted by one reviewer and verified by a second reviewer, with disagreements resolved by consensus or a third reviewer.

RESULTS

We included 57 studies, which used between 1 and 41 nominal groups that included between 2 and 30 participants per group. We grouped the 30 identified decision points for the NGT process into two stages common to most qualitative group methods [Research objectives; Group characteristics] and three stages related to the nominal groups themselves [Eliciting survey items; Refining survey elicited items from stage 3; Evaluating and selecting final survey items]. We discuss the advantages and disadvantages of each option in relation to specific study contexts.

CONCLUSION

Investigators should carefully consider their options for each of the identified decision points and document the reasons for their choices in their protocol to maximize validity and transparency.

A Scoping Review of Process Indicators for Measuring Quality of Care in Glaucoma

PRCIS

There are no standardized process quality indicators (QIs) in glaucoma care. Although they can be inferred from guidelines and trials, they should be designed and standardized to allow better assessment of the quality of care.

PURPOSE

QIs are crucial for assessing the performance of any health care system. To allow efficiency, effectiveness, and patient-centeredness, there is a need for prompt acquisition of up-to-date information. Among the available QIs, process indicators have the highest sensitivity to frequent changes and could better reflect the implementation outcomes of novel ideas and technology. This study aimed to map the available information regarding process QIs in glaucoma care, identify the current development stage of these indicators, and systematically synthesize them.

MATERIALS AND METHODS

We performed a scoping review of 4 electronic bibliographic databases for studies reporting on process QIs in glaucoma. We retrieved 7502 references and created a domain list reflecting the core idea underlying each indicator.

RESULTS

We summarized information from 18 documents and listed 20 domains. The most mentioned domains were follow-up, optic nerve head assessment, visual field test, and intraocular pressure. Indicators regarding the quality of life assessment, patient assistance, or presence of written protocols were less frequently mentioned.

CONCLUSIONS

There are notable variations among process QIs in glaucoma and significant heterogeneity in their descriptions in published studies. Although novel indicators can be inferred from guidelines and trials, they should be designed and standardized for better assessment of performance in health systems to improve their quality.

PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews

The methods and results of systematic reviews should be reported in sufficient detail to allow users to assess the trustworthiness and applicability of the review findings. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement was developed to facilitate transparent and complete reporting of systematic reviews and has been updated (to PRISMA 2020) to reflect recent advances in systematic review methodology and terminology. Here, we present the explanation and elaboration paper for PRISMA 2020, where we explain why reporting of each item is recommended, present bullet points that detail the reporting recommendations, and present examples from published reviews. We hope that changes to the content and structure of PRISMA 2020 will facilitate uptake of the guideline and lead to more transparent, complete, and accurate reporting of systematic reviews.

Optical Coherence Tomography Imaging of the Lamina Cribrosa: Structural Biomarkers in Nonglaucomatous Diseases

The lamina cribrosa (LC) is an active structure that responds to the strain by changing its morphology. Abnormal changes in LC morphology are usually associated with, and indicative of, certain pathologies such as glaucoma, intraocular hypertension, and myopia. Recent developments in optical coherence tomography (OCT) have enabled detailed in vivo studies about the architectural characteristics of the LC. Structural characteristics of the LC have been widely explored in glaucoma management. However, information about which LC biomarkers could be useful for the diagnosis, and follow-up, of other diseases besides glaucoma is scarce. Hence, this literature review aims to summarize the role of the LC in nonophthalmic and ophthalmic diseases other than glaucoma. PubMed was used to perform a systematic review on the LC features that can be extracted from OCT images. All imaging features are presented and discussed in terms of their importance and applicability in clinical practice. A total of 56 studies were included in this review. Overall, LC depth (LCD) and thickness (LCT) have been the most studied features, appearing in 75% and 45% of the included studies, respectively. These biomarkers were followed by the prelaminar tissue thickness (21%), LC curvature index (5.4%), LC global shape index (3.6%), LC defects (3.6%), and LC strains/deformations (1.8%). Overall, the disease groups showed a thinner LC (smaller LCT) and a deeper ONH cup (larger LCD), with some exceptions. A large variability between approaches used to compute LC biomarkers has been observed, highlighting the importance of having automated and standardized methodologies in LC analysis. Moreover, further studies are needed to identify the pathologies where LC features have a diagnostic and/or prognostic value.

Adjudication rather than experience of data abstraction matters more in reducing errors in abstracting data in systematic reviews

BACKGROUND

During systematic reviews, "data abstraction" refers to the process of collecting data from reports of studies. The data abstractors' level of experience may affect the accuracy of data abstracted. Using data from a randomized crossover trial in which different data abstraction approaches were compared, we examined the association between abstractors' level of experience and accuracy of data abstraction.

METHODS

We classified abstractors as "more experienced" if they had authored three or more published systematic reviews, and "less experienced" otherwise. Each abstractor abstracted data related to study design, baseline characteristics, and outcomes/results from six articles. We considered two types of errors: incorrect abstraction and errors of omission. We estimated the proportion of errors by level of experience using a binomial generalized linear mixed model.

RESULTS

We used data from 25 less experienced and 25 more experienced data abstractors. Overall error proportions were similar for less experienced abstractors (21%) and more experienced abstractors (19%). Compared with less experienced abstractors, more experienced abstractors had a lower odds of errors for data items related to outcomes/results (adjusted odds ratio [OR] = 0.53; 95% CI, 0.34-0.82) and potentially for data items related to study design (adjusted OR = 0.83; 95% CI, 0.64-1.09) but a potentially higher odds of errors for items related to baseline characteristics (adjusted OR = 1.42; 95% CI, 0.97-2.06).

CONCLUSION

Experience of data abstraction matters little. Errors are reduced by adjudication but still remain high for data items related to outcomes/results.

The Systematic Review Data Repository (SRDR): descriptive characteristics of publicly available data and opportunities for research

BACKGROUND

Conducting systematic reviews ("reviews") requires a great deal of effort and resources. Making data extracted during reviews available publicly could offer many benefits, including reducing unnecessary duplication of effort, standardizing data, supporting analyses to address secondary research questions, and facilitating methodologic research. Funded by the US Agency for Healthcare Research and Quality (AHRQ), the Systematic Review Data Repository (SRDR) is a free, web-based, open-source, data management and archival platform for reviews. Our specific objectives in this paper are to describe (1) the current extent of usage of SRDR and (2) the characteristics of all projects with publicly available data on the SRDR website.

METHODS

We examined all projects with data made publicly available through SRDR as of November 12, 2019. We extracted information about the characteristics of these projects. Two investigators extracted and verified the data.

RESULTS

SRDR has had 2552 individual user accounts belonging to users from 80 countries. Since SRDR's launch in 2012, data have been made available publicly for 152 of the 735 projects in SRDR (21%), at a rate of 24.5 projects per year, on average. Most projects are in clinical fields (144/152 projects; 95%); most have evaluated interventions (therapeutic or preventive) (109/152; 72%). The most frequent health areas addressed are mental and behavioral disorders (31/152; 20%) and diseases of the eye and ocular adnexa (23/152; 15%). Two-thirds of the projects (104/152; 67%) were funded by AHRQ, and one-sixth (23/152; 15%) are Cochrane reviews. The 152 projects each address a median of 3 research questions (IQR 1-5) and include a median of 70 studies (IQR 20-130).

CONCLUSIONS

Until we arrive at a future in which the systematic review and broader research communities are comfortable with the accuracy of automated data extraction, re-use of data extracted by humans has the potential to help reduce redundancy and costs. The 152 projects with publicly available data through SRDR, and the more than 15,000 studies therein, are freely available to researchers and the general public who might be working on similar reviews or updates of reviews or who want access to the data for decision-making, meta-research, or other purposes.