Is quality and completeness of reporting of systematic reviews and meta-analyses published in high impact radiology journals associated with citation rates?

Currently Available Radiology-Specific Reporting Guidelines

The aim of this paper is to contextualize and review reporting guidelines available at the EQUATOR Network that are most relevant to radiology-specific investigations. Eight EQUATOR Network reporting guidelines for the clinical area of radiology, not including the subspecialized areas of imaging of the cardiovascular, neurologic, and oncologic diseases are reviewed and discussed. The reporting guidelines are for diagnostic and therapeutic clinical research. Why the reporting guideline was development, by whom, their aims and what they hope to achieve are discussed. A table summarizes what the reporting guideline is provided for; an acronym if present is given; a full bibliographic reference with PMID number; the reporting guideline website URL or link; the study design and section of the report that the guideline applies to; and the date that the reporting guideline was last updated.

Evaluation of Imaging Research Adherence to the STARD 2015 Reporting Guideline: Update 9 Years After Implementation and Baseline Assessment

Background: Adherence of diagnostic accuracy imaging research to the STARD 2015 reporting guideline was assessed at baseline in 2016; on average, only 55% of 30 items were reported. Several knowledge translation strategies have since been implemented by the STARD group. Purpose: The purpose of this study was to evaluate the adherence of diagnostic accuracy studies recently published in imaging journals to STARD 2015, to assess for changes in the level of adherence relative to the baseline study. Methods: We performed an electronic search on MEDLINE for diagnostic accuracy studies, published between May and June of 2024, from a select group of imaging journals. The timespan was modulated to achieve a sample size of 100 to 150 included studies. Overall and item-specific adherence to STARD 2015 was evaluated, in addition to associations with journal of publication, imaging modality, study design, country of corresponding author, imaging subspecialty area, journal impact factor, and journal STARD adoption. Statistical comparison to the baseline study from 2016 was also performed. Poisson Regression and two-tailed student's tests were used to compare STARD adherence relative to variables included in subgroup analysis. Results: In the 126 included studies, average adherence to STARD 2015 was 61% (18.3/30 items; SD = 3.1), improved compared to the baseline study (55%; 16.6/30 items; SD = 2.2; P < .0001). Studies published in higher impact factor journals reported more items than those in lower impact factor journals (20.6 vs 18.4 items, P-value <.0001). There was no significant association between reporting completeness and journal of publication (P = .7), imaging modality (P = .21), country of corresponding author (P = .46), imaging subspecialty (P = .31), and journal STARD adoption status (P = .55). Conclusion: Recently published diagnostic accuracy studies reported more STARD 2015 items than studies published in 2016, but completeness of reporting is still not optimal.

Concordance between humans and GPT-4 in appraising the methodological quality of case reports and case series using the Murad tool

BACKGROUND

Assessing the methodological quality of case reports and case series is challenging due to human judgment variability and time constraints. We evaluated the agreement in judgments between human reviewers and GPT-4 when applying a standard methodological quality assessment tool designed for case reports and series.

METHODS

We searched Scopus for systematic reviews published in 2023-2024 that cited the appraisal tool by Murad et al. A GPT-4 based agent was developed to assess the methodological quality using the 8 signaling questions of the tool. Observed agreement and agreement coefficient were estimated comparing published judgments of human reviewers to GPT-4 assessment.

RESULTS

We included 797 case reports and series. The observed agreement ranged between 41.91% and 80.93% across the eight questions (agreement coefficient ranged from 25.39 to 79.72%). The lowest agreement was noted in the first signaling question about selection of cases. The agreement was similar in articles published in journals with impact factor < 5 vs. ≥ 5, and when excluding systematic reviews that did not use 3 causality questions. Repeating the analysis using the same prompts demonstrated high agreement between the two GPT-4 attempts except for the first question about selection of cases.

CONCLUSIONS

The study demonstrates a moderate agreement between GPT-4 and human reviewers in assessing the methodological quality of case series and reports using the Murad tool. The current performance of GPT-4 seems promising but unlikely to be sufficient for the rigor of a systematic review and pairing the model with a human reviewer is required.

Evaluation of reporting quality of cohort studies using real-world data based on RECORD: systematic review

OBJECTIVE

Real-world data (RWD) and real-world evidence (RWE) have been paid more and more attention in recent years. We aimed to evaluate the reporting quality of cohort studies using real-world data (RWD) published between 2013 and 2021 and analyze the possible factors.

METHODS

We conducted a comprehensive search in Medline and Embase through the OVID interface for cohort studies published from 2013 to 2021 on April 29, 2022. Studies aimed at comparing the effectiveness or safety of exposure factors in the real-world setting were included. The evaluation was based on the REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. Agreement for inclusion and evaluation was calculated using Cohen's kappa. Pearson chi-square test or Fisher's exact test and Mann-Whitney U test were used to analyze the possible factors, including the release of RECORD, journal IFs, and article citations. Bonferroni's correction was conducted for multiple comparisons. Interrupted time series analysis was performed to display the changes in report quality over time.

RESULTS

187 articles were finally included. The mean ± SD of the percentage of adequately reported items in the 187 articles was 44.7 ± 14.3 with a range of 11.1-87%. Of 23 items, the adequate reporting rate of 10 items reached 50%, and the reporting rate of some vital items was inadequate. After Bonferroni's correction, the reporting of only one item significantly improved after the release of RECORD and there was no significant improvement in the overall report quality. For interrupted time series analysis, there were no significant changes in the slope (p = 0.42) and level (p = 0.12) of adequate reporting rate. The journal IFs and citations were respectively related to 2 areas and the former significantly higher in high-reporting quality articles.

CONCLUSION

The endorsement of the RECORD cheklist was generally inadequate in cohort studies using RWD and has not improved in recent years. We encourage researchers to endorse relevant guidelines when utilizing RWD for research.

TFOS Lifestyle - Evidence quality report: Advancing the evaluation and synthesis of research evidence

Evidence-based practice is a dominant paradigm in healthcare that emphasizes the importance of ensuring the translation of the best available, relevant research evidence into practice. An Evidence Quality Subcommittee was established to provide specialized methodological support and expertise to promote rigorous and evidence-based approaches for the Tear Film and Ocular Surface Society (TFOS) Lifestyle Epidemic reports. The present report describes the purpose, scope, and activity of the Evidence Quality Subcommittee in the undertaking of high-quality narrative-style literature reviews, and leading prospectively registered, reliable systematic reviews of high priority research questions, using standardized methods for each topic area report. Identification of predominantly low or very low certainty evidence across the eight systematic reviews highlights a need for further research to define the efficacy and/or safety of specific lifestyle interventions on the ocular surface, and to clarify relationships between certain lifestyle factors and ocular surface disease. To support the citation of reliable systematic review evidence in the narrative review sections of each report, the Evidence Quality Subcommittee curated topic-specific systematic review databases and relevant systematic reviews underwent standardized reliability assessment. Inconsistent methodological rigor was noted in the published systematic review literature, emphasizing the importance of internal validity assessment. Based on the experience of implementing the Evidence Quality Subcommittee, this report makes suggestions for incorporation of such initiatives in future international taskforces and working groups. Content areas broadly relevant to the activity of the Evidence Quality Subcommittee, including the critical appraisal of research, clinical evidence hierarchies (levels of evidence), and risk of bias assessment, are also outlined.

Checklist for Artificial Intelligence in Medical Imaging Reporting Adherence in Peer-Reviewed and Preprint Manuscripts With the Highest Altmetric Attention Scores: A Meta-Research Study

Purpose: To establish reporting adherence to the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) in diagnostic accuracy AI studies with the highest Altmetric Attention Scores (AAS), and to compare completeness of reporting between peer-reviewed manuscripts and preprints. Methods: MEDLINE, EMBASE, arXiv, bioRxiv, and medRxiv were retrospectively searched for 100 diagnostic accuracy medical imaging AI studies in peer-reviewed journals and preprint platforms with the highest AAS since the release of CLAIM to June 24, 2021. Studies were evaluated for adherence to the 42-item CLAIM checklist with comparison between peer-reviewed manuscripts and preprints. The impact of additional factors was explored including body region, models on COVID-19 diagnosis and journal impact factor. Results: Median CLAIM adherence was 48% (20/42). The median CLAIM score of manuscripts published in peer-reviewed journals was higher than preprints, 57% (24/42) vs 40% (16/42), P < .0001. Chest radiology was the body region with the least complete reporting ( P = .0352), with manuscripts on COVID-19 less complete than others (43% vs 54%, P = .0002). For studies published in peer-reviewed journals with an impact factor, the CLAIM score correlated with impact factor, rho = 0.43, P = .0040. Completeness of reporting based on CLAIM score had a positive correlation with a study’s AAS, rho = 0.68, P < .0001. Conclusions: Overall reporting adherence to CLAIM is low in imaging diagnostic accuracy AI studies with the highest AAS, with preprints reporting fewer study details than peer-reviewed manuscripts. Improved CLAIM adherence could promote adoption of AI into clinical practice and facilitate investigators building upon prior works.

The association between topic growth and citation impact of research publications

Citations are used for research evaluation, and it is therefore important to know which factors influence or associate with citation impact of articles. Several citation factors have been studied in the literature. In this study we propose a new factor, topic growth, that no previous study has studied empirically. The growth rate of topics may influence future citation counts because a high growth in a topic means there are more publications citing previous publications in that topic. We construct topics using community detection in a citation network and use a two-part regression model to study the association between topic growth and citation counts in eight broad disciplines. The first part of the model uses quantile regression to estimate the effect of growth ratio on citation counts for publications with more than three citations. The second part of the model uses logistic regression to model the influence of the explanatory variables on the probability of being lowly cited versus being modestly or highly cited. Both models control for three variables that may distort the association between the topic growth and citations: journal impact, number of references, and number of authors. The regression model clearly shows that publications in fast-growing topics have a citation advantage compared to publications in slow-growing or declining topics in all of the eight disciplines. Using citation indicators for research evaluation may give incentives for researchers to publish in fast-growing topics, but they may cause research to be less diversified. The results have also some implications for citation normalization.

Assessing journal author guidelines for systematic reviews and meta-analyses: findings from an institutional sample

OBJECTIVES

Systematic reviews and meta-analyses (SRs/MAs) are designed to be rigorous research methodologies that synthesize information and inform practice. An increase in their publication runs parallel to quality concerns and a movement toward standards to improve reporting and methodology. With the goal of informing the guidance librarians provide to SR/MA teams, this study assesses online journal author guidelines from an institutional sample to determine whether these author guidelines address SR/MA methodological quality.

METHODS

A Web of Science Core Collection (Clarivate) search identified SRs/MAs published in 2014-2019 by authors affiliated with a single institution. The AMSTAR 2 checklist was used to develop an assessment tool of closed questions specific to measures for SR/MA methodological quality in author guidelines, with questions added about author guidelines in general. Multiple reviewers completed the assessment.

RESULTS

The author guidelines of 141 journals were evaluated. Less than 20% addressed at least one of the assessed measures specific to SR/MA methodological quality. There was wide variation in author guidelines between journals from the same publisher apart from the American Medical Association, which consistently offered in-depth author guidelines. Normalized Eigenfactor and Article Influence Scores did not indicate author guideline breadth.

CONCLUSIONS

Most author guidelines in the institutional sample did not address SR/MA methodological quality. When consulting with teams embarking on SRs/MAs, librarians should not expect author guidelines to provide details about the requirements of the target journals. Librarians should advise teams to follow established SR/MA standards, contact journal staff, and review SRs/MAs previously published in the journal.

High variability in results and methodological quality among overlapping systematic reviews on the same topics in surgery: a meta-epidemiological study

BACKGROUND

Redundant publication of systematic reviews and meta-analyses (SRs/MAs) on the same topic presents an increasing burden for clinicians. The aim of this study was to describe variabilities in effect size and methodological quality of overlapping surgery-related SRs/MAs and to investigate factors associated with their postpublication citations.

METHODS

PubMed/MEDLINE was searched to identify SRs/MAs of RCTs on thoracoabdominal surgeries published in 2015. Previous SRs/MAs on the same topics published within the preceding 5 years (2011-2015) were identified and 5-year citation counts (through to 2020) were evaluated. Discrepancies in pooled effect sizes and their methodological quality using A Measurement Tool to Assess Systematic Reviews (AMSTAR) among overlapping SRs/MAs were assessed. The SR/MA-level factors associated with 5-year citation counts were explored, using a mixed-effects regression model with a random intercept for surgical topics.

RESULTS

A total of 57 surgery-related SRs/MAs (48 topics) published in 2015 were identified, and 146 SRs/MAs had overlapping publications on 29 topics (60.4 per cent of all topics) in the preceding 5 years. There was considerable variability in methodological quality of SRs/MAs and coverage probability for relevant RCTs, resulting in discrepant effect size estimates for the same topic. High quality (AMSTAR score 8-11) was independently associated with higher 5-year citation counts (coefficient = 32.82; 95 per cent c.i. 15.63 to 50.02; P < 0.001).

CONCLUSION

Overlapping SRs/MAs with high variability in results and methodological quality were common in surgery. A high-quality SR/MA score was an independent predictor of more frequent citations. Researchers and journal editors should concentrate their efforts on limiting publications to higher-quality reviews.

A bibliometric analysis of the top 100 most-cited case reports and case series in Endodontic journals

AIM

To identify the top 100 most-cited case reports and case series published in Endodontic journals and to analyse their bibliometric characteristics.

METHODOLOGY

The Clarivate Analytics' Web of Science (WoS), Scopus and PubMed databases were used to identify the top 100 most-cited case reports and case series in Endodontic journals. Complete bibliographic records of the selected case reports and case series were exported in plain text or BibTeX format and imported into the R environment for statistical computing and graphics. The following parameters were then analysed: names and affiliations of the authors, title, year of publication, journal of publication, first author, corresponding author, literature cited within reports, language, citation counts, impact factor of the journal, keywords, Keywords Plus and research topic.

RESULTS

In total, 88 case reports and 12 case series published in English between 1977 and 2016 were identified as the most-cited reports in the field of Endodontics. The terms "case report(s)" or "case series" were not included in the title of 57 articles. The number of authors per report ranged from one to seven, with the average number of co-authors per report being 3.14. The most-cited author was M Trope (University of Pennsylvania, USA). The University of Washington and Private Practice, Cetraro, Italy, were the most productive institutions. The country whose case reports received the largest total number of citations was the United States. The largest number of the most-cited reports appeared in 2002, 2004 and 2007 (n = 7, respectively). According to the WoS database, the total number of citations ranged from 42 to 453, with the average number of citations per report being 79.97. The majority of the top 100 most-cited articles were published in the Journal of Endodontics and the International Endodontic Journal. The most frequently used author keywords were revascularization and mineral trioxide aggregate. The majority of the case reports and case series dealt with topics related to pulp regeneration.

CONCLUSION

This bibliometric study provides a comprehensive overview on the progress, trends and current directions in clinical practice within the field of Endodontics.

A comparison of systematic reviews and guideline-based systematic reviews in medical studies

The question of how citation impact relates to academic quality accompanies every decade in bibliometric research. Although experts have employed more complex conceptions of research quality for responsible evaluation, detailed analyses of how impact relates to dimensions such as methodological rigor are lacking. But the increasing number of formal guidelines for biomedical research offer not only the potential to understand the social dynamics of standardization, but also their relations to scientific rewards. By using data from Web of Science and PubMed, this study focuses on systematic reviews from biomedicine and compares this genre with those systematic reviews that applied the PRISMA reporting standard. Besides providing an overview about growth and location, it was found that the latter, more standardized type of systematic review accumulates more citations. It is argued that instead of reinforcing the traditional conception that higher impact represents higher quality, highly prolific authors could be more inclined to develop and apply new standards than more average researchers. In addition, research evaluation would benefit from a more nuanced conception of scientific output which respects the intellectual role of various document types.

Methodological assessment of systematic reviews and meta-analyses on COVID-19: A meta-epidemiological study

RATIONALE, AIMS, AND OBJECTIVES

COVID-19 has caused an ongoing public health crisis. Many systematic reviews and meta-analyses have been performed to synthesize evidence for better understanding this new disease. However, some concerns have been raised about rapid COVID-19 research. This meta-epidemiological study aims to methodologically assess the current systematic reviews and meta-analyses on COVID-19.

METHODS

We searched in various databases for systematic reviews with meta-analyses published between 1 January 2020 and 31 October 2020. We extracted their basic characteristics, data analyses, evidence appraisal, and assessment of publication bias and heterogeneity.

RESULTS

We identified 295 systematic reviews on COVID-19. The median time from submission to acceptance was 33 days. Among these systematic reviews, 73.9% evaluated clinical manifestations or comorbidities of COVID-19. Stata was the most used software programme (43.39%). The odds ratio was the most used effect measure (34.24%). Moreover, 28.14% of the systematic reviews did not present evidence appraisal. Among those reporting the risk of bias results, 14.64% of studies had a high risk of bias. Egger's test was the most used method for assessing publication bias (38.31%), while 38.66% of the systematic reviews did not assess publication bias. The I2 statistic was widely used for assessing heterogeneity (92.20%); many meta-analyses had high values of I2 . Among the meta-analyses using the random-effects model, 75.82% did not report the methods for model implementation; among those meta-analyses reporting implementation methods, the DerSimonian-Laird method was the most used one.

CONCLUSIONS

The current systematic reviews and meta-analyses on COVID-19 might suffer from low transparency, high heterogeneity, and suboptimal statistical methods. It is recommended that future systematic reviews on COVID-19 strictly follow well-developed guidelines. Sensitivity analyses may be performed to examine how the synthesized evidence might depend on different methods for appraising evidence, assessing publication bias, and implementing meta-analysis models.

Determinants of societal and academic recognition: Evidence from randomised controlled trials

Given the increasing importance of recognition in academia and the vital role of randomised controlled trials (RCTs) in medical research and clinical decisions, this study verifies how RCTs’ academic and societal impacts are affected by visibility factors, subjects and methodological validity. This study concentrated on a sample of 446 RCTs indexed in Scopus and evaluated by Cochrane reviewers in terms of their methodological validity. The altmetrics, bibliometric and bibliographical information were extracted from Altmetric.com and Scopus, and the contributing countries’ development ranks were obtained from the United Nations Development report. The linear regression analyses revealed that citations and altmetrics depend on some subjects. They are also affected by publication year and journals’ previous reputation. Citations are also affected by keyword counts and reference counts. Keyword counts and contributing countries’ developmental rank also predict the tweet counts. While none of the methodological validity dimensions were found to predict citations, ‘Incomplete Outcome Data’ and ‘Random Sequence Generation’ significantly, though slightly, affect Mendeley Readership and tweets, respectively. By confirming the dependence of RCTs’ recognition on some methodological validity features and attention-inducing characteristics, the study provides further evidence on the interaction of quality and visibility dynamisms in the recognition network and the complementary role of societal mentions for academic citation.

Mapping the scientific research on radiology departments: Global trends in publication, collaboration and trending topics

PURPOSE

To characterize the global research trend in radiology departments based on bibliometric indicators.

MATERIAL AND METHOD

As a source of information, Science Citation Index Expanded and Journal Citation Reports from Web of Science Core Collection (WoSCC) were used. Annual trends, journals of publication, subject categories of journals, collaboration indexes between authors and institutions, network of cowords and most cited papers were identified and analysed. The period of study was 2009-2018.

RESULTS

283,587 downloaded papers were analysed. The number of articles was increasing, as well as the percentage of funded works. Papers were published in 7314 different journals, being the most productive Plos One (5077), followed by American Journal of Roentgenology (4602) and European Radiology (3644). Most productive subject categories of journals were Radiology, Nuclear Medicine & Medical Imaging (86,568 papers), Clinical Neurology (29,722) and Surgery (23,564). International collaboration has increased more than 5 points, from 15.2% in 2009 to 20.7% in 2018.

CONCLUSIONS

Most cited articles were published in high impact journals outside the scope of diagnostic imaging. Most influential topics included technical innovations within imaging modalities. MRI replaced conventional radiography and CT as the imaging technique of choice in imaging research.

A Bibliometric analysis of folate receptor research

BACKGROUND

The objective of this study was to conduct a bibliometric analysis of the entire field of folate receptor research. Folate receptor is expressed on a wide variety of cancers and certain immune cells.

METHODS

A Web of Science search was performed on folate receptor or folate binding protein (1969-to June 28, 2019). The following information was examined: publications per year, overall citations, top 10 authors, top 10 institutions, top 10 cited articles, top 10 countries, co-author collaborations and key areas of research.

RESULTS

In total, 3248 documents for folate receptor or folate binding protein were retrieved for the study years outlined in the methods section search query. The range was 1 per year in 1969 to 264 for the last full year studied (2018). A total of 123,720 citations for the 3248 documents retrieved represented a mean citation rate per article of 38.09 and range of 1667 citations (range 0 to 1667). Researchers in 71 countries authored publications analyzed in this study. The US was the leader in publications and had the highest ranking institution. The top 10 articles have been cited 7270 times during the time frame of this study. The top cited article had an average citation rate of 110 citations per year. Network maps revealed considerable co-authorship among several of the top 10 authors.

CONCLUSION

Our study presents several important insights into the features and impact of folate receptor research. To our knowledge, this is the first bibliometric analysis of folate receptor.

Adherence to the Standards for Reporting of Diagnostic Accuracy (STARD) 2015 Guidelines in Acute Point-of-Care Ultrasound Research

IMPORTANCE

Incomplete reporting of diagnostic accuracy research impairs assessment of risk of bias and limits generalizability. Point-of-care ultrasound has become an important diagnostic tool for acute care physicians, but studies assessing its use are of varying methodological quality.

OBJECTIVE

To assess adherence to the Standards for Reporting of Diagnostic Accuracy (STARD) 2015 guidelines in the literature on acute care point-of-care ultrasound.

EVIDENCE REVIEW

MEDLINE was searched to identify diagnostic accuracy studies assessing point-of-care ultrasound published in critical care, emergency medicine, or anesthesia journals from 2016 to 2019. Studies were evaluated for adherence to the STARD 2015 guidelines, with the following variables analyzed: journal, country, STARD citation, STARD-adopting journal, impact factor, patient population, use of supplemental material, and body region. Data analysis was performed in November 2019.

FINDINGS

Seventy-four studies were included in this systematic review for assessment. Overall adherence to STARD was moderate, with 66% (mean [SD], 19.7 [2.9] of 30 items) of STARD items reported. Items pertaining to imaging specifications, patient population, and readers of the index test were frequently reported (>66% of studies). Items pertaining to blinding of readers to clinical data and to the index or reference standard, analysis of heterogeneity, indeterminate and missing data, and time intervals between index and reference test were either moderately (33%-66%) or infrequently (<33%) reported. Studies in STARD-adopting journals (mean [SD], 20.5 [2.9] items in adopting journals vs 18.6 [2.3] items in nonadopting journals; P = .002) and studies citing STARD (mean [SD], 21.3 [0.9] items in citing studies vs 19.5 [2.9] items in nonciting studies; P = .01) reported more items. Variation by country and journal of publication were identified. No differences in STARD adherence were identified by body region imaged (mean [SD], abdominal, 20.0 [2.5] items; head and neck, 17.8 [1.6] items; musculoskeletal, 19.2 [3.1] items; thoracic, 20.2 [2.8] items; and other or procedural, 19.8 [2.7] items; P = .29), study design (mean [SD], prospective, 19.7 [2.9] items; retrospective, 19.7 [1.8] items; P > .99), patient population (mean [SD], pediatric, 20.0 [3.1] items; adult, 20.2 [2.7] items; mixed, 17.9 [1.9] items; P = .09), use of supplementary materials (mean [SD], yes, 19.2 [3.0] items; no, 19.7 [2.8] items; P = .91), or journal impact factor (mean [SD], higher impact factor, 20.3 [3.1] items; lower impact factor, 19.1 [2.4] items; P = .08).

CONCLUSIONS AND RELEVANCE

Overall, the literature on acute care point-of-care ultrasound showed moderate adherence to the STARD 2015 guidelines, with more complete reporting found in studies citing STARD and those published in STARD-adopting journals. This study has established a current baseline for reporting; however, future studies are required to understand barriers to complete reporting and to develop strategies to mitigate them.

Publication integrity in orthopaedic journals: the self-citation in orthopaedic research (SCOR) threshold

BACKGROUND

The impact factor (IF) is the most commonly used bibliometric method for rating academic journals. However, the practice of journals' self-citation may artificially elevate the IF. Additional bibliometric methods including Eigenfactor scale, SCImago Journal Ranking (SJR), and corrected IF (cIF) have been created. Comparing general-interest and specialized orthopaedic journals, the aims of this study were to assess: (1) the effect of journal´s self-citation on IF; (2) differences in bibliometric analysis; and (3) to determine thresholds for monitoring self-citation practices by defining the self-citation in orthopaedic research (SCOR) Threshold.

METHODS

The journal citation reports and SCImago Journal and Country Rank databases were queried for orthopaedic journals from 1997 to 2017. The following bibliometrics were compared between general-interest and specialized journals: IF, cIF, Eigenfactor, self-citation rates, and SJR. A novel metric, the cIF ratio, was proposed to represent the relationship between a journal's IF and cIF. Thresholds for cIF were based on statistical outliers of cIF ratio within general-interest and specialized journals were calculated. Outliers were defined as data points that were greater than the third quartile by 1.5 times the interquartile range using the last 10 years studied (2007-2017).

RESULTS

Specialized orthopaedic journals had a higher median self-citation rates compared to general-interest journals (11.85% vs. 6.36%, p < 0.001). Overall, cIF ratio declined over study period, and general-interest journals had a lower cIF ratio than specialized journals (8.77% vs. 19.54%, p < 0.001). Overall, general-interest journals had more favourable values for the bibliometric indices studied compared to specialized journals The SCOR threshold for cIF ratio was determined as 25.4% for general-interest journals and 53.3% for specialized journals.

CONCLUSION

Overall, self-citation occurs at a higher rate in specialized versus general-interest orthopaedic journals. We propose the use of a cIF ratio along with the SCOR threshold as a tool to evaluate and monitor journal self-citation practices in orthopaedic research.

Predictors of Citation Rate for Original Research Studies in the Canadian Association of Radiologists Journal

OBJECTIVE

This study is aimed to identify predictors of citation rate of original research published in the Canadian Association of Radiologists Journal (CARJ).

METHODS

A search of MEDLINE was conducted from January 1, 2000-June 30, 2013 to identify all studies published in the CARJ. Original research studies were included. Reviews, pictorial essays, guidelines, case studies, case series, and original studies with a sample size <10 were excluded. Variables assessed for association with citation rate included number of authors, study design, sample size, multi-institutional study, multi-national study, study type, presence of statistically significant result, presence of funding, and number of references. Statistical analysis was completed using linear regression and Pearson correlation coefficients (r).

RESULTS

A total of 714 studies were published in CARJ, of which 181 were original research publications that were cited a total of 1517 times. Twelve original research studies were uncited, while the most-cited one was cited 58 times. Sample size (r = 0.177, P = .017) and number of references (r = 0.164, P = .028) demonstrated statistically significant weak positive correlations with citation rate. Number of authors, study design, setting, statistically significant results, and funding were not associated with citation rate.

CONCLUSION

Only a very small number of original research studies published at the CARJ remained uncited 5 or more years after the publication. Sample size and number of references were identified as significant, but weak predictors of citation rate in CARJ.

Steps toward more complete reporting of systematic reviews of diagnostic test accuracy: Preferred Reporting Items for Systematic Reviews and Meta-Analyses of Diagnostic Test Accuracy (PRISMA-DTA)

Reporting standards in biomedical research have been shown to be suboptimal. The publication of the PRISMA statement has improved the completeness of reporting of systematic reviews, but several issues specific to diagnostic test accuracy are not included in the PRISMA statement. Therefore, a diagnostic test accuracy extension of the PRISMA statement, PRISMA-DTA, was created. This commentary addresses completeness of reporting in systematic reviews, the PRISMA-DTA statement, and strategies for optimal uptake of reporting guidelines.

Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review

BACKGROUND

From 2005 to 2010, we conducted 2 randomized studies on a journal (Medicina Clínica), where we took manuscripts received for publication and randomly assigned them to either the standard editorial process or to additional processes. Both studies were based on the use of methodological reviewers and reporting guidelines (RG). Those interventions slightly improved the items reported on the Manuscript Quality Assessment Instrument (MQAI), which assesses the quality of the research report. However, masked evaluators were able to guess the allocated group in 62% (56/90) of the papers, thus presenting a risk of detection bias. In this post-hoc study, we analyse whether those interventions that were originally designed for improving the completeness of manuscript reporting may have had an effect on the number of citations, which is the measured outcome that we used.

METHODS

Masked to the intervention group, one of us used the Web of Science (WoS) to quantify the number of citations that the participating manuscripts received up December 2016. We calculated the mean citation ratio between intervention arms and then quantified the uncertainty of it by means of the Jackknife method, which avoids assumptions about the distribution shape.

RESULTS

Our study included 191 articles (99 and 92, respectively) from the two previous studies, which all together received 1336 citations. In both studies, the groups subjected to additional processes showed higher averages, standard deviations and annual rates. The intervention effect was similar in both studies, with a combined estimate of a 43% (95% CI: 3 to 98%) increase in the number of citations.

CONCLUSIONS

We interpret that those effects are driven mainly by introducing into the editorial process a senior methodologist to find missing RG items. Those results are promising, but not definitive due to the exploratory nature of the study and some important caveats such as: the limitations of using the number of citations as a measure of scientific impact; and the fact that our study is based on a single journal. We invite journals to perform their own studies to ascertain whether or not scientific repercussion is increased by adhering to reporting guidelines and further involving statisticians in the editorial process.

Systematic Review of the Literature: Best Practices

Reviews of published scientific literature are a valuable resource that can underline best practices in medicine and clarify clinical controversies. Among the various types of reviews, the systematic review of the literature is ranked as the most rigorous since it is a high-level summary of existing evidence focused on answering a precise question. Systematic reviews employ a pre-defined protocol to identify relevant and trustworthy literature. Such reviews can accomplish several critical goals that are not easily achievable with typical empirical studies by allowing identification and discussion of best evidence, contradictory findings, and gaps in the literature. The Association of University Radiologists Radiology Research Alliance Systematic Review Task Force convened to explore the methodology and practical considerations involved in performing a systematic review. This article provides a detailed and practical guide for performing a systematic review and discusses its applications in radiology.

Appraising the Quality of Systematic Reviews for Age-Related Macular Degeneration Interventions: A Systematic Review

Importance

Age-related macular degeneration (AMD) is a leading cause of vision impairment. It is imperative that AMD care is timely, appropriate, and evidence-based. It is thus essential that AMD systematic reviews are robust; however, little is known about the quality of this literature.

Objectives

To investigate the methodological quality of systematic reviews of AMD intervention studies, and to evaluate their use for guiding evidence-based care.

Evidence Review

This systematic review adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. All studies that self-identified as a systematic review in their title or abstract or were categorized as a systematic review from a medical subject heading and investigated the safety, efficacy and/or effectiveness of an AMD intervention were included. Comprehensive electronic searches were performed in Ovid MEDLINE, Embase, and the Cochrane Library from inception to March 2017. Two reviewers independently assessed titles and abstracts, then full-texts for eligibility. Quality was assessed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool. Study characteristics (publication year, type of intervention, journal, citation rate, and funding source) were extracted.

Findings

Of 983 citations retrieved, 71 studies (7.6%) were deemed eligible. The first systematic review relating to an AMD intervention was published in 2003. More than half were published since 2014. Methodological quality was highly variable. The mean (SD) AMSTAR score was 5.8 (3.2) of 11.0, with no significant improvement over time (r = -0.03; 95% CI, -0.26 to 0.21; P = .83). Cochrane systematic reviews were overall of higher quality than reviews in other journals (mean [SD] AMSTAR score, 9.9 [1.2], n = 15 vs 4.7 [2.2], n = 56; P < .001). Overall, there was poor adherence to referring to an a priori design (22 articles [31%]) and reporting conflicts of interest in both the review and included studies (16 articles [23%]). Reviews funded by government grants and/or institutions were generally of higher quality than industry-sponsored reviews or where the funding source was not reported.

Conclusions and Relevance

There are gaps in the conduct of systematic reviews in the field of AMD. Enhanced endorsement of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement by refereed journals may improve review quality and improve the dissemination of reliable evidence relating to AMD interventions to clinicians.

H-index in medicine is driven by original research

Aim

To investigate the contribution of selected types of articles to h-indices of medical researchers.

Methods

We used the Web of Science to export the publication records of various members from 26 scientific medical societies (13 European, 13 North American) associated with 13 medical specialties. Those included were presidents (n = 26), heads of randomly chosen committees (n = 52), and randomly selected members of those committees (n = 52). Publications contributing to h-index were categorized as research articles, reviews, guidelines, meta-analyses, or other published work.

Results

Overall, 3259 items authored by 129 scholars were analyzed. The median h-index was 19.5. The median contribution of research articles to h-index was 84.4%. Researchers in the upper h-index tercile (≥28.5) had a larger share of research articles that contributed to h-index in comparison with those in the lower h-index tercile (≤12.5) (median 87.3% [1st-3rd quartile: 80.0%-93.1%] vs 80.0% [50.0%-88.9%], P = 0.015). We observed an analogous difference with regard to guidelines (1.1% [0%-3.7%] vs 0% [0%-0%], P = 0.007).

Conclusions

Original research drives h-indices in medicine. Although guidelines contribute to h-indices in medicine, their influence is low. The specific role of randomized controlled trials in building h-index in medicine remains to be assessed.

Systematic Reviews in Burns Care: Poor Quality and Getting Worse

The objective of this article is to investigate adherence to reporting standards and methodological quality in systematic reviews on burns care published in peer-reviewed journals to determine their utility for guiding evidence-based burns care. PubMed, Embase, Database of Abstracts of Reviews of Effects, Cochrane Database of Systematic Reviews, and the Joanna Briggs Institute (JBI) Database of Systematic Reviews and Implementation Reports were searched from 2009. Any systematic review on any question on therapeutic interventions in burns care was eligible for inclusion. Critical appraisal and data extraction were performed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) checklist by two independent reviewers. The overall quality of the 44 included burns care systematic reviews was low, with an average methodological quality of 55% and an average compliance with reporting guidelines of 70%. Correlation analysis showed that adherence to reporting guidelines has been relatively stable, but methodological quality has deteriorated (r = -.32, P < .05). Cochrane reviews had lower citation rates than reviews published in other journals, whereas reviews that included meta-analyses had more citations. Quality did not have a significant effect on citation rate. Health professionals working in burns should be able to expect that systematic reviews published in their field are of a high standard. Unfortunately, this is not the case. To address this problem, established guidelines on the conduct and reporting of systematic reviews should be adhered to by researchers and editors.

Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review

BACKGROUND

The PRISMA Statement is a reporting guideline designed to improve transparency of systematic reviews (SRs) and meta-analyses. Seven extensions to the PRISMA Statement have been published to address the reporting of different types or aspects of SRs, and another eight are in development. We performed a scoping review to map the research that has been conducted to evaluate the uptake and impact of the PRISMA Statement and extensions. We also synthesised studies evaluating how well SRs published after the PRISMA Statement was disseminated adhere to its recommendations.

METHODS

We searched for meta-research studies indexed in MEDLINE® from inception to 31 July 2017, which investigated some component of the PRISMA Statement or extensions (e.g. SR adherence to PRISMA, journal endorsement of PRISMA). One author screened all records and classified the types of evidence available in the studies. We pooled data on SR adherence to individual PRISMA items across all SRs in the included studies and across SRs published after 2009 (the year PRISMA was disseminated).

RESULTS

We included 100 meta-research studies. The most common type of evidence available was data on SR adherence to the PRISMA Statement, which has been evaluated in 57 studies that have assessed 6487 SRs. The pooled results of these studies suggest that reporting of many items in the PRISMA Statement is suboptimal, even in the 2382 SRs published after 2009 (where nine items were adhered to by fewer than 67% of SRs). Few meta-research studies have evaluated the adherence of SRs to the PRISMA extensions or strategies to increase adherence to the PRISMA Statement and extensions.

CONCLUSIONS

Many studies have evaluated how well SRs adhere to the PRISMA Statement, and the pooled result of these suggest that reporting of many items is suboptimal. An update of the PRISMA Statement, along with a toolkit of strategies to help journals endorse and implement the updated guideline, may improve the transparency of SRs.

Citation Analysis and Trends in review articles in dentistry

OBJECTIVE

This study aimed to describe the trends in dentistry article reviews as well as to compare citation patterns between systematic and narrative reviews.

METHODS

A search strategy was developed, in Scopus database, in order to identify all narrative and systematic reviews published between 2000 and 2015. Original research studies, letters to the editor, editorials, book chapters, and case reports were excluded. From the list of studies available, 30 reviews per year were randomly chosen. The review type, year of publication, number of authors, country of the first author, open access, language, main topic of interest, journal's H index, number of references, and number of citations were extracted by 2 researchers. The number of citations was extracted from the Scopus database. Multivariable regression analysis was used in order to detect the association between citation rate and the independent variables.

RESULTS

Overall, 118 and 362 systematic and narrative reviews were included in this study. Throughout the years, the number of systematic reviews has increased from 5.8% to 53.3%. However, the mean number of citations has significantly decreased, and this is affected by the review's year of publication. A trend for lower citation in systematic reviews (Relative risk [RR]: 0.79; 95% confidence interval: 0.75-0.84) has been demonstrated; however, the number of citations of narrative reviews has been increasing over the years (RR: 1.14; 95% confidence interval: 1.08-1.21).

CONCLUSION

From 2000 to 2015, the number of systematic reviews increased substantially. On the other hand, a trend for lower citations of these studies has been observed that is affected over time.

Characteristics, Trends, and Quality of Systematic Review and Meta-Analysis in General Radiology between 2007 and 2015

RATIONALE AND OBJECTIVES

To evaluate the trends, characteristics, and quality of systematic review and meta-analysis in general radiology journals.

MATERIALS AND METHODS

We performed a PubMed search to identify systematic reviews and meta-analyses that had been carried out in the field of radiology between 2007 and 2015. The following data were extracted: journal, impact factor, type of research, year of publication, radiological subspecialty, imaging modalities used, number of authors, affiliated department of the first and corresponding authors, presence of a radiologist and a statistician among the authors, discordance between the first and corresponding authors, funding, country of first author, methodological quality, methods used for quality assessment, and statistics.

RESULTS

Ultimately, we included 210 articles from nine general radiology journals. The European Journal of Radiology was the most common journal represented (47 of 210; 22.4%). Meta-analyses (n = 177; 84.3%) were published about five times more than systematic reviews without meta-analysis (n = 33; 15.7%). Radiology of the gastrointestinal tract was the most commonly represented subspecialty (n = 49, 23.3%). The first authors were most frequently located in China (n = 64; 30.3%). In terms of modality, magnetic resonance imaging was used most often (n = 59; 28.1%). The number of authors tended to progressively increase over time, and the ratio of discordance between the first and corresponding authors also increased significantly, as did the proportion of research that has received funding from an external source. The mean AMSTAR assessment score improved over time (5.87/11 in 2007-2009, 7.11/11 in 2010-2012, and 7.49/11 in 2013-2015). In this regard, the journal Radiology had the highest score (7.59/11).

CONCLUSIONS

The quantity and quality of radiological meta-analyses have significantly increased over the past 9 years; however, specific weak areas remain, providing the opportunity for quality improvement.

The reporting of studies using routinely collected health data was often insufficient

OBJECTIVES

To assess reporting quality of studies using routinely collected health data (RCD) to inform the REporting of studies Conducted using Observational Routinely collected health Data (RECORD) guideline development.

STUDY DESIGN AND SETTING

PubMed search for observational studies using RCD on any epidemiologic or clinical topic. Sample of studies published in 2012. Evaluation of five items based on the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guideline and eight newly developed items for RCD studies.

RESULTS

Of 124 included studies, 39 (31.5%) clearly described its design in title or abstract. Complete information to frame a focused research question, that is, on the population, intervention/exposure, and outcome, was provided for 51 studies (41.1%). In 44 studies where definitions of codes or classification algorithms would be necessary to operationalize such a research question, only nine (20.5%) reported all items adequately. In 81 studies describing multivariable analyses, 54 (66.7%) reported all variables used for modeling and 34 (42.0%) reported basic details required for replication. Database linkage was reported adequately in 12 of 41 studies (29.3%). Statements about data sharing/availability were rare (5/124; 4%).

CONCLUSION

Most RCD studies are insufficiently reported. Specific reporting guidelines and more awareness and education on their use are urgently needed.

Is the Best Evidence Good Enough: Quality Assessment and Factor Analysis of Meta-Analyses on Depression

BACKGROUND

The quality of meta-analyses (MAs) on depression remains uninvestigated.

OBJECTIVE

To assess the overall reporting and methodological qualities of MAs on depression and to explore potential factors influencing both qualities.

METHODS

MAs investigating epidemiology and interventions for depression published in the most recent year (2014-2015) were selected from PubMed, EMBASE, PsycINFO and Cochrane Library. The characteristics of the included studies were collected and the total and per-item quality scores of the included studies were calculated based on the two checklists. Univariate and multivariate linear regression analyses were used to explore the potential factors influencing the quality of the articles.

RESULTS

A total of 217 MAs from 74 peer-reviewed journals were included. The mean score of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was 23.0 of 27 and mean score of Assessment of Multiple Systematic Reviews (AMSTAR) was 8.3 of 11. Items assessing registration and protocol (14.2%, 37/217) in PRISMA and item requiring a full list of included and excluded studies (16.1%, 40/217) in AMSTAR had poorer adherences than other items. The MAs that included only RCTs, pre-registered, had five more authors or authors from Cochrane groups and the MAs found negative results had better reporting and methodological qualities.

CONCLUSIONS

The reporting and methodological qualities of MAs on depression remained to be improved. Design of included studies, characteristics of authors and pre-registration in PROSPERO database are important factors influencing quality of MAs in the field of depression.

Most Common Publication Types in Radiology Journals:: What is the Level of Evidence?

RATIONALE AND OBJECTIVES

This study aimed to assess the most common publication types in radiology journals, as well as temporal trends and association with citation frequency.

MATERIALS AND METHODS

PubMed was searched to extract all published articles having the following "Publication Type" indices: "validation studies," "meta-analysis," "clinical trial," "comparative study," "evaluation study," "guideline," "multicenter study," "randomized study," "review," "editorial," "case report," and "technical report." The percentage of articles within each category published within clinical radiology journals was computed. Normalized percentages for each category were also computed on an annual basis. Citation counts within a 2-year window following publication were obtained using Web of Science. Overall trends were assessed.

RESULTS

Publication types with the highest fraction in radiology journals were technical reports, evaluation studies, and case reports (4.8% to 5.8%). Publication types with the lowest fraction in radiology journals were randomized trials, multicenter studies, and meta-analyses (0.8% to 1.5%). Case reports showed a significant decrease since 1999, with accelerating decline since 2007 (P = 0.002). Publication types with highest citation counts were meta-analyses, guidelines, and multicenter studies (8.1 ± 10.7 to 12.9 ± 5.1). Publication types with lowest citation counts were case reports, editorials, and technical reports (1.4 ± 2.4 to 2.9 ± 4.3). The representation in radiology journals and citation frequency of the publication types showed weak inverse correlation (r = -0.372).

CONCLUSIONS

Radiology journals have historically had relatively greater representation of less frequently cited publication types. Various strategies, including methodological training, multidisciplinary collaboration, national support networks, as well as encouragement of higher level of evidence by funding agencies and radiology journals themselves, are warranted to improve the impact of radiological research.

Pitfalls of Systematic Reviews and Meta-Analyses in Imaging Research

Systematic reviews of imaging research represent a tool to better understand test accuracy or the efficacy of interventions. Like any type of research, appropriate methods must be applied to optimize quality. The purpose of this review is to outline common pitfalls in performing systematic reviews of imaging research, with a focus on challenges particular to performing reviews of diagnostic accuracy studies. The following challenges are highlighted: posing relevant review questions, conducting comprehensive literature searches, assessing for bias in included studies, testing for heterogeneity and publication bias, pooling results across studies, and forming appropriate conclusions. By guiding authors on how to overcome these, the hope is that published reviews in imaging research will be of higher quality and have a positive impact on clinical practice. In addition, the review aims to educate readers of reviews so they become aware of crucial elements of systematic reviews that could bias review results.