How does evidence affect clinical decision-making?

Reporting Bias is Highly Prevalent in Systematic Reviews and Meta-Analyses of Platelet Rich Plasma Injections for Hip Osteoarthritis

Purpose

To describe the incidence and types of spin in systematic reviews of platelet-rich plasma (PRP) injections for hip osteoarthritis (OA) and to determine whether patterns in study characteristics could be identified among studies with identifiable spin.

Methods

The PubMed, Scopus, and SPORTDiscus databases were queried. Inclusion criteria were systematic reviews or meta-analyses that included an assessment of intra-articular PRP injections as a stand-alone treatment for hip OA. Two authors independently assessed the presence of spin in the included studies and recorded general study characteristics. The prevalence of the 15 different categories of spin was quantified using descriptive statistics.

Results

Fifteen studies met inclusion criteria for this study. All studies contained at least two types of spin (range 2-9), with a median of 2. The most common type of spin was type 14 ("Failure to report a wide confidence interval of estimates"), which was observed in 10 studies. The second most common type of spin was type 13 ("Failure to specify the direction of the effect when it favors the control intervention"), found in 6 studies.

Conclusions

Spin is highly prevalent in abstracts of systematic reviews of PRP in the treatment of hip OA. Several associations were found between spin types and the study characteristics of AMSTAR 2 rating, Scopus CiteScore, journal impact factor, and PROSPERO preregistration. When present, spin in the abstracts of reviewed studies tended to favor the use of PRP in hip osteoarthritis.

Clinical Relevance

It is important to understand the prevalence of spin in published abstracts, especially in areas of great impact or interest, so authors and readers can have a greater awareness of this potential form of bias.

Evaluation of spin in reviews of biodegradable balloon spacers for massive irreparable rotator cuff tears

HYPOTHESIS

Clinical studies are often at risk of spin, a form of bias where beneficial claims are overstated while negative findings are minimized or dismissed. Spin is often more problematic in abstracts given their brevity and can result in the misrepresentation of a study's actual findings. The goal of this study is to aggregate primary and secondary studies reporting the clinical outcomes of the use of subacromial balloon spacers in the treatment of massive irreparable rotator cuff tears to identify the incidence of spin and find any significant association with study design parameters.

MATERIALS AND METHODS

This study was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Independent searches were completed on 2 databases (PubMed and Embase) for primary studies, systematic and current concepts reviews, and meta-analyses and the results were compiled. Two authors independently screened the studies using a predetermined inclusion criteria and aggregated data including titles, publication journals and years, authors, study design, etc. Each study was independently assessed for the presence of 15 different types of spin. Statistical analysis was conducted to identify associations between study characteristics and spin.

RESULTS

Twenty-nine studies met the inclusion criteria for our analysis, of which 10 were reviews or meta-analyses and 19 were primary studies. Spin was identified in every study except for 2 (27/29, 93.1%). Type 3 spin, "Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention" and type 9 spin, "Conclusion claims the beneficial effect of the experimental treatment despite reporting bias" were most frequently noted in our study, both observed in 12/29 studies (41.4%). Date of publication, and adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses or "The International Prospective Register of Systematic Reviews" were study characteristics associated with a higher rate of certain types of spin. There was a statistically significant association between disclosure of external study funding source and the presence of spin type 4, but none of the other forms of spin.

CONCLUSION

Spin is highly prevalent in the abstracts of primary studies, systematic reviews, and meta-analyses discussing the use of subacromial balloon spacer technology in the treatment of massive irreparable rotator cuff tears. Our findings revealed that spin in the abstract tended to favor the balloon spacer intervention. Further efforts are required in the future to mitigate spin within the abstracts of published manuscripts.

PubMed4Hh: A Point-of-Care Mobile App for Evidence-Based Clinical Decision Support for Nurse Residents in Maryland

Evidence-based clinical decision-making is considered a core competency for professional nurses. However, barriers such as lack of time in clinical practice; information overload; restricted access to resources; lack of evidence appraisal skills, awareness, and knowledge; and unreliable resources prevent them from adopting evidence-based practice. This study examined the usefulness of the PubMed4Hh (PubMed for Handhelds) mobile application for clinical decision-making at the point of care among Maryland newly licensed RNs. Using iOS and Android smartphones, 178 newly licensed RNs participated in a 6-month trial from November 2019 to April 2020. Nurses manually entered free-text queries or used voice inputs on the "Patient, Intervention, Comparison, Outcome" or "askMEDLINE" search engines. The results retrieved were presented as journal article abstracts or short summaries called "the bottom line," designed for quick reading at the point of care. Both Patient, Intervention, Comparison, Outcome and askMEDLINE were rated highly for their usefulness, and participants said they would continue using PubMed4Hh and recommend it to others. Newly licensed RNs had a significantly higher perception of the usefulness of PubMed4Hh when the results of "the bottom line" or abstracts confirmed, led, or modified their nursing skills, knowledge, or the patient's care plan.

Evaluation of spin in systematic reviews and meta-analyses of superior capsular reconstruction

BACKGROUND

Small, preliminary studies and the systematic reviews on superior capsular reconstruction (SCR) that collate data are at increased risk spin. This study's primary objective was to identify, describe, and account for the incidence of spin in systematic reviews of SCR. This study's secondary objective was to characterize the studies in which spin was identified to determine whether identifiable patterns of characteristics exist among studies with spin.

METHODS

This study was conducted per Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines using a predetermined protocol. A search was conducted on the PubMed and Embase databases for systematic reviews and meta-analyses on SCR. Screening and data extraction were conducted independently by 2 authors. Each included study's abstract was assessed for the presence of the 15 most common types of spin, with full texts reviewed during cases of disagreement or for clarification. General data that were extracted included study title, authors, publication year, journal, level of evidence, study design, funding source, reported adherence to PRISMA guidelines, preregistration of the study protocol, and primary and secondary outcome measures. Full texts were used in the assessment of study quality per AMSTAR 2.

RESULTS

We identified 53 studies during our search, of which 17 met the inclusion criteria. At least 1 form of spin was observed in all 17 studies. The most common types of spin were type 5 ("The conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies") and type 9 ("Conclusion claims the beneficial effect of the experimental treatment despite reporting bias"), both of which were observed in 11 studies (11 of 17, 65%). A statistically significant association between lower level of evidence and type 5 ("The conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies") was observed (P = .0175). A statistically significant association was also found between more recent year of publication and the spin category misleading interpretation (P = .0398), and between lower AMSTAR 2 score and type 13 ("Failure to specify the direction of the effect when it favors the control intervention") (P = .0260). No other statistical associations between other study characteristics were observed.

CONCLUSION

Spin is highly prevalent in abstracts of SCR systematic reviews and meta-analyses. An association was found between the presence of spin and lower level of evidence, year of publication, and AMSTAR 2 ratings.

Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on Tinnitus

HYPOTHESIS

The objective was to investigate the prevalence of spin in abstracts of systematic reviews and meta-analyses covering the treatment of tinnitus. We hypothesized that spin would be present in these articles and a significant relationship would exist between spin usage and extracted study characteristics.

BACKGROUND

Spin, the misrepresentation of study findings, can alter a clinician's interpretation of a study's results, potentially affecting patient care. Previous work demonstrates that spin is present in abstracts of randomized clinical trials.

METHODS

Using a cross-sectional analysis, we conducted a systematic search using MEDLINE and Embase databases on June 2, 2020, for systematic reviews focused on tinnitus treatment. Investigators performed screening and data extraction in a masked, duplicate fashion.

RESULTS

Forty systematic reviews met inclusion criteria, and spin was identified in four of them. Spin in abstracts most frequently occurred when conclusions claimed the beneficial effect of the experimental treatment despite high risk of bias in primary studies (n = 3). The other form of spin found was the conclusion claims safety based on nonstatistically significant results with a wide confidence interval (n = 1). There was no significant association between spin and any of our extracted study characteristics.

CONCLUSION

Spin was observed in 10% of abstracts of systematic reviews and meta-analyses covering the treatment of tinnitus. Although this percentage may be small, we recommend that medical journals provide a more detailed framework for abstract structure and require the inclusion of risk of bias assessment results in abstracts to prevent the incorporation of spin.

Analyzing Spin in Abstracts of Orthopaedic Randomized Controlled Trials with Statistically Insignificant Primary Endpoints

PURPOSE

Our primary aim of our study is to evaluate the prevalence of spin among abstracts in orthopaedic randomized controlled trials (RCTs) with nonsignificant primary endpoints.

METHODS

This study was conducted in accordance with a previously written protocol publicly available via the Open Science Framework. PubMed (which includes Medline) was searched for RCTs in orthopaedic surgery. The articles that were identified were then uploaded to Rayyan, and the abstracts were screened for inclusion. To be included, a trial had to have randomized the patients for intervention, statistically compare multiple groups, and had a primary endpoint that was not significant. Odds ratios and summary statistics (frequencies and proportions) were then calculated for spin in the abstracts.

RESULTS

Of the 780 articles retrieved from our search string, 250 articles met the inclusion criteria. Analysis resulted in 112/250 (44.80%; [95% CI, 38.64-50.96]) RCTs that containing spin within the abstract. Of the 112 RCTs, 52 (46.43%; [95% CI, 37.19-55.66]) had spin in the results, and 89 (79.46%; [95% CI, 71.98-86.95]) had spin in the conclusion of the abstract. The Journal of Bone and Joint Surgery was found to have the highest prevalence of spin (21/37, 56.76% [95% CI, 40.79%-72.72%]) while Arthroscopy: The Journal of Arthroscopic & Related Surgery had the lowest prevalence of spin (5/15, 33.33% [95% CI, 9.48%-57.19%]). No correlation was found between industry funding and increased odds of spin in the abstract (uOR, 1.10; 95% CI, 0.45-2.63). Discrepancies for our primary endpoint, prevalence of spin among abstracts, were analyzed with Gwet's AC1 inter-rater statistic and found to be 81% [95% CI, 0.75-0.87].

CONCLUSION

Spin was found in 44.8% of the abstracts within our sample of orthopaedic RCTs. Nonsignificant primary data were often represented to seem significant, many orthopaedic RCTs did not indicate primary endpoints, and orthopaedic RCTs infrequently reported trial registration.

Clinicians' perceptions of usefulness of the PubMed4Hh mobile device application for clinical decision making at the point of care: a pilot study

BACKGROUND

Although evidence-based practice in healthcare has been facilitated by Internet access through wireless mobile devices, research on the effectiveness of clinical decision support for clinicians at the point of care is lacking. This study examined how evidence as abstracts and the bottom-line summaries, accessed with PubMed4Hh mobile devices, affected clinicians' decision making at the point of care.

METHODS

Three iterative steps were taken to evaluate the usefulness of PubMed4Hh tools at the NIH Clinical Center. First, feasibility testing was conducted using data collected from a librarian. Next, usability testing was carried out by a postdoctoral research fellow shadowing clinicians during rounds for one month in the inpatient setting. Then, a pilot study was conducted from February, 2016 to January, 2017, with clinicians using a mobile version of PubMed4Hh. Invitations were sent via e-mail lists to clinicians (physicians, physician assistants and nurse practitioners) along with periodic reminders. Participants rated the usefulness of retrieved bottom-line summaries and abstracts and indicated their usefulness on a 7-point Likert scale. They also indicated location of use (office, rounds, etc.).

RESULTS

Of the 166 responses collected in the feasibility phase, more than half of questions (57%, n = 94) were answerable by both the librarian using various resources and by the postdoctoral research fellow using PubMed4Hh. Sixty-six questions were collected during usability testing. More than half of questions (60.6%) were related to information about medication or treatment, while 21% were questions regarding diagnosis, and 12% were specific to disease entities. During the pilot study, participants reviewed 34 abstracts and 40 bottom-line summaries. The abstracts' usefulness mean scores were higher (95% CI [6.12, 6.64) than the scores of the bottom-line summaries (95% CI [5.25, 6.10]). The most frequent reason given was that it confirmed current or tentative diagnostic or treatment plan. The bottom-line summaries were used more in the office (79.3%), and abstracts were used more at point of care (51.9%).

CONCLUSIONS

Clinicians reported that retrieving relevant health information from biomedical literature using the PubMed4Hh was useful at the point of care and in the office.