Dual computer monitors to increase efficiency of conducting systematic reviews

What are the best methods for rapid reviews of the research evidence? A systematic review of reviews and primary studies

Rapid review methodology aims to facilitate faster conduct of systematic reviews to meet the needs of the decision-maker, while also maintaining quality and credibility. This systematic review aimed to determine the impact of different methodological shortcuts for undertaking rapid reviews on the risk of bias (RoB) of the results of the review. Review stages for which reviews and primary studies were sought included the preparation of a protocol, question formulation, inclusion criteria, searching, selection, data extraction, RoB assessment, synthesis, and reporting. We searched 11 electronic databases in April 2022, and conducted some supplementary searching. Reviewers worked in pairs to screen, select, extract data, and assess the RoB of included reviews and studies. We included 15 systematic reviews, 7 scoping reviews, and 65 primary studies. We found that several commonly used shortcuts in rapid reviews are likely to increase the RoB in the results. These include restrictions based on publication date, use of a single electronic database as a source of studies, and use of a single reviewer for screening titles and abstracts, selecting studies based on the full-text, and for extracting data. Authors of rapid reviews should be transparent in reporting their use of these shortcuts and acknowledge the possibility of them causing bias in the results. This review also highlights shortcuts that can save time without increasing the risk of bias. Further research is needed for both systematic and rapid reviews on faster methods for accurate data extraction and RoB assessment, and on development of more precise search strategies.

Title-plus-abstract versus title-only first-level screening approach: a case study using a systematic review of dietary patterns and sarcopenia risk to compare screening performance

BACKGROUND

Conducting a systematic review is a time- and resource-intensive multi-step process. Enhancing efficiency without sacrificing accuracy and rigor during the screening phase of a systematic review is of interest among the scientific community.

METHODS

This case study compares the screening performance of a title-only (Ti/O) screening approach to the more conventional title-plus-abstract (Ti + Ab) screening approach. Both Ti/O and Ti + Ab screening approaches were performed simultaneously during first-level screening of a systematic review investigating the relationship between dietary patterns and risk factors and incidence of sarcopenia. The qualitative and quantitative performance of each screening approach was compared against the final results of studies included in the systematic review, published elsewhere, which used the standard Ti + Ab approach. A statistical analysis was conducted, and contingency tables were used to compare each screening approach in terms of false inclusions and false exclusions and subsequent sensitivity, specificity, accuracy, and positive predictive power.

RESULTS

Thirty-eight citations were included in the final analysis, published elsewhere. The current case study found that the Ti/O first-level screening approach correctly identified 22 citations and falsely excluded 16 citations, most often due to titles lacking a clear indicator of study design or outcomes relevant to the systematic review eligibility criteria. The Ti + Ab approach correctly identified 36 citations and falsely excluded 2 citations due to limited population and intervention descriptions in the abstract. Our analysis revealed that the performance of the Ti + Ab first-level screening was statistically different compared to the average performance of both approaches (Chi-squared: 5.21, p value 0.0225) while the Ti/O approach was not (chi-squared: 2.92, p value 0.0874). The predictive power of the first-level screening was 14.3% and 25.5% for the Ti/O and Ti + Ab approaches, respectively. In terms of sensitivity, 57.9% of studies were correctly identified at the first-level screening stage using the Ti/O approach versus 94.7% by the Ti + Ab approach.

CONCLUSIONS

In the current case study comparing two screening approaches, the Ti + Ab screening approach captured more relevant studies compared to the Ti/O approach by including a higher number of accurately eligible citations. Ti/O screening may increase the likelihood of missing evidence leading to evidence selection bias.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO Protocol Number: CRD42020172655.

Resource use during systematic review production varies widely: a scoping review

OBJECTIVE

We aimed to map the resource use during systematic review (SR) production and reasons why steps of the SR production are resource intensive to discover where the largest gain in improving efficiency might be possible.

STUDY DESIGN AND SETTING

We conducted a scoping review. An information specialist searched multiple databases (e.g., Ovid MEDLINE, Scopus) and implemented citation-based and grey literature searching. We employed dual and independent screenings of records at the title/abstract and full-text levels and data extraction.

RESULTS

We included 34 studies. Thirty-two reported on the resource use-mostly time; four described reasons why steps of the review process are resource intensive. Study selection, data extraction, and critical appraisal seem to be very resource intensive, while protocol development, literature search, or study retrieval take less time. Project management and administration required a large proportion of SR production time. Lack of experience, domain knowledge, use of collaborative and SR-tailored software, and good communication and management can be reasons why SR steps are resource intensive.

CONCLUSION

Resource use during SR production varies widely. Areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.

Does Using Multiple Computer Monitors for Office Tasks Affect User Experience? : A Systematic Review

OBJECTIVE

To compare the impact of multiple computer monitor configurations on health and performance outcomes compared to the use of a single monitor.

BACKGROUND

Multiple monitor configurations are used in office settings to promote increased productivity by providing more screen space; however, it is unknown if there are health-related trade-offs to increased productivity.

METHOD

A systematic review was conducted according to the PRISMA statement guidelines and adapted the best evidence synthesis.

RESULTS

Eighteen studies were included in our review. There was strong evidence that implementing dual monitors is in line with users' preference. There was also moderate evidence for controlled laboratory studies demonstrating that multiple monitors may increase task efficiency with decreased desktop interaction; however, implementing multiple monitors may also result in nonneutral neck postures for users.

CONCLUSION

More research needs to be conducted on biomechanical exposures when using larger displays. Longitudinal field studies should be conducted to determine the influence of monitor interventions on health, productivity, and well-being. All studies must consider task complexity and user positioning and should measure health and productivity outcomes together. Researchers must also consider up-to-date purchasing trends when choosing the monitor configurations and sizes for their studies.

APPLICATION

Regulatory bodies and practitioners can use the results to develop evidence-based monitor guidelines and inform decision-making in practice, respectively. Researchers can use this information to design future studies on monitor configurations that incorporate current purchasing trends.

An evaluation of DistillerSR's machine learning-based prioritization tool for title/abstract screening - impact on reviewer-relevant outcomes

BACKGROUND

Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings.

METHODS

Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented.

RESULTS

Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%).

CONCLUSIONS

The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.

Error rates of human reviewers during abstract screening in systematic reviews

Background

Automated approaches to improve the efficiency of systematic reviews are greatly needed. When testing any of these approaches, the criterion standard of comparison (gold standard) is usually human reviewers. Yet, human reviewers make errors in inclusion and exclusion of references.

Objectives

To determine citation false inclusion and false exclusion rates during abstract screening by pairs of independent reviewers. These rates can help in designing, testing and implementing automated approaches.

Methods

We identified all systematic reviews conducted between 2010 and 2017 by an evidence-based practice center in the United States. Eligible reviews had to follow standard systematic review procedures with dual independent screening of abstracts and full texts, in which citation inclusion by one reviewer prompted automatic inclusion through the next level of screening. Disagreements between reviewers during full text screening were reconciled via consensus or arbitration by a third reviewer. A false inclusion or exclusion was defined as a decision made by a single reviewer that was inconsistent with the final included list of studies.

Results

We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions from 86 unique reviewers. The final systematic reviews included 5.48% of the potential references identified through bibliographic database search (95% confidence interval (CI): 2.38% to 8.58%). After abstract screening, the total error rate (false inclusion and false exclusion) was 10.76% (95% CI: 7.43% to 14.09%).

Conclusions

This study suggests important false inclusion and exclusion rates by human reviewers. When deciding the validity of a future automated study selection algorithm, it is important to keep in mind that the gold standard is not perfect and that achieving error rates similar to humans may be adequate and can save resources and time.

Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review

OBJECTIVES

The aim of the article was to identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews.

STUDY DESIGN AND SETTING

A systematic review was conducted, searching MEDLINE, EMBASE, and the Cochrane Library from inception to September 1, 2016. Quality appraisal of included studies was undertaken using a modified Quality Assessment of Diagnostic Accuracy Studies 2, and key results on accuracy, reliability, efficiency of a methodology, or impact on results and conclusions were extracted.

RESULTS

After screening 5,600 titles and abstracts and 245 full-text articles, 37 studies were included. For screening, studies supported the involvement of two independent experienced reviewers and the use of Google Translate when screening non-English articles. For data abstraction, studies supported involvement of experienced reviewers (especially for continuous outcomes) and two independent reviewers, use of dual monitors, graphical data extraction software, and contacting authors. For quality appraisal, studies supported intensive training, piloting quality assessment tools, providing decision rules for poorly reported studies, contacting authors, and using structured tools if different study designs are included.

CONCLUSION

Few studies exist documenting common systematic review practices. Included studies support several systematic review practices. These results provide an updated evidence-base for current knowledge synthesis guidelines and methods requiring further research.

Extracting data from figures with software was faster, with higher interrater reliability than manual extraction

OBJECTIVES

To compare speed and accuracy of graphical data extraction using manual estimation and open source software.

STUDY DESIGN AND SETTING

Data points from eligible graphs/figures published in randomized controlled trials (RCTs) from 2009 to 2014 were extracted by two authors independently, both by manual estimation and with the Plot Digitizer, open source software. Corresponding authors of each RCT were contacted up to four times via e-mail to obtain exact numbers that were used to create graphs. Accuracy of each method was compared against the source data from which the original graphs were produced.

RESULTS

Software data extraction was significantly faster, reducing time for extraction for 47%. Percent agreement between the two raters was 51% for manual and 53.5% for software data extraction. Percent agreement between the raters and original data was 66% vs. 75% for the first rater and 69% vs. 73% for the second rater, for manual and software extraction, respectively.

CONCLUSIONS

Data extraction from figures should be conducted using software, whereas manual estimation should be avoided. Using software for data extraction of data presented only in figures is faster and enables higher interrater reliability.