How do statistics in dental articles differ from those articles published in highly visible medical journals?

THE EFFECT DIRECTION SHOULD BE TAKEN INTO ACCOUNT WHEN ASSESSING SMALL-STUDY EFFECTS

OBJECTIVE

Studies with statistically significant results are frequently more likely to be published than those with non-significant results. This phenomenon leads to publication bias or small-study effects and can seriously affect the validity of the conclusion from systematic reviews and meta-analyses. Small-study effects typically appear in a specific direction, depending on whether the outcome of interest is beneficial or harmful, but this direction is rarely taken into account in conventional methods.

METHODS

We propose to use directional tests to assess potential small-study effects. The tests are built on a one-sided testing framework based on the existing Egger's regression test. We performed simulation studies to compare the proposed one-sided regression tests, conventional two-sided regression tests, as well as two other competitive methods (Begg's rank test and the trim-and-fill method). Their performance was measured by type I error rates and statistical power. Three real-world meta-analyses on measurements of infrabony periodontal defects were also used to examine the various methods' performance.

RESULTS

Based on simulation studies, the one-sided tests could have considerably higher statistical power than competing methods, particularly their two-sided counterparts. Their type I error rates were generally controlled well. In the case study of the three real-world meta-analyses, by accounting for the favored direction of effects, the one-sided tests could rule out potential false-positive conclusions about small-study effects. They also are more powerful in assessing small-study effects than the conventional two-sided tests when true small-study effects likely exist.

CONCLUSION

We recommend researchers incorporate the potential favored direction of effects into the assessment of small-study effects.

The Quality of Statistical Reporting and Data Presentation in Predatory Dental Journals Was Lower Than in Non-Predatory Journals

Proper peer review and quality of published articles are often regarded as signs of reliable scientific journals. The aim of this study was to compare whether the quality of statistical reporting and data presentation differs among articles published in 'predatory dental journals' and in other dental journals. We evaluated 50 articles published in 'predatory open access (OA) journals' and 100 clinical trials published in legitimate dental journals between 2019 and 2020. The quality of statistical reporting and data presentation of each paper was assessed on a scale from 0 (poor) to 10 (high). The mean (SD) quality score of the statistical reporting and data presentation was 2.5 (1.4) for the predatory OA journals, 4.8 (1.8) for the legitimate OA journals, and 5.6 (1.8) for the more visible dental journals. The mean values differed significantly (p < 0.001). The quality of statistical reporting of clinical studies published in predatory journals was found to be lower than in open access and highly cited journals. This difference in quality is a wake-up call to consume study results critically. Poor statistical reporting indicates wider general lower quality in publications where the authors and journals are less likely to be critiqued by peer review.

Use of Data Analysis Methods in Dental Publications: Is There Evidence of a Methodological Change?

Objectives: To evaluate how data analysis methods in dental studies have changed in recent years. Methods: A total of 400 articles published in 2010 and 2017 in five dental journals, Journal of Dental Research, Caries Research, Community Dentistry and Oral Epidemiology, Journal of Dentistry, and Acta Odontologica Scandinavica, were analyzed. The study characteristics and the reporting of data analysis techniques were systematically identified. Results: The statistical intensity of the dental journals did not change from 2010 to 2017. Dental researchers did not adopt the data mining, machine learning, or Bayesian approaches advocated in the computer-oriented methodological literature. The determination of statistical significance was the most generally used method for conducting research in both 2010 and 2017. Observational study designs were more common in 2017. Insufficient and incomplete descriptions of statistical methods were still a serious problem. Conclusion: The stabilization of statistical intensity in the literature suggests that papers applying highly computationally complex data analysis methods have not meaningfully contributed to dental research or clinical care. Greater rigor is required in reporting the methods in dental research articles, given the current pervasiveness of failure to describe the basic techniques used.

Reporting data analysis methods in high-impact respiratory journals

Data analysis methods play an important role in respiratory research. We evaluated the application and complexity of data analytical methods in high-impact respiratory journals and compared the statistical reporting in these respiratory articles with reports published in other eminent medical journals. This study involved a total of 160 papers published in 2015 in the European Respiratory Journal, American Journal of Respiratory and Critical Care Medicine, Chest and Thorax, and 680 papers published between 2007-2015 in other medical journals including the Lancet and New England Journal of Medicine. We manually reviewed the articles to determine the way in which they reported the methods applied in data analysis. The statistical intensity in the respiratory journals was equal to that in eminent medical journals. Traditional ways of testing statistical significance were widely used in respiratory articles. Statistical procedures were not always described in sufficient detail, and the prominent respiratory journals did not display different profiles with respect to their statistical content. Readers of the prominent respiratory journals need to possess a substantial level of statistical expertise if they wish to critically evaluate the design, methodology, data analysis and interpretation of the findings published in these journals.

An instrument to assess the statistical intensity of medical research papers

Background

There is widespread evidence that statistical methods play an important role in original research articles, especially in medical research. The evaluation of statistical methods and reporting in journals suffers from a lack of standardized methods for assessing the use of statistics. The objective of this study was to develop and evaluate an instrument to assess the statistical intensity in research articles in a standardized way.

Methods

A checklist-type measure scale was developed by selecting and refining items from previous reports about the statistical contents of medical journal articles and from published guidelines for statistical reporting. A total of 840 original medical research articles that were published between 2007–2015 in 16 journals were evaluated to test the scoring instrument. The total sum of all items was used to assess the intensity between sub-fields and journals. Inter-rater agreement was examined using a random sample of 40 articles. Four raters read and evaluated the selected articles using the developed instrument.

Results

The scale consisted of 66 items. The total summary score adequately discriminated between research articles according to their study design characteristics. The new instrument could also discriminate between journals according to their statistical intensity. The inter-observer agreement measured by the ICC was 0.88 between all four raters. Individual item analysis showed very high agreement between the rater pairs, the percentage agreement ranged from 91.7% to 95.2%.

Conclusions

A reliable and applicable instrument for evaluating the statistical intensity in research papers was developed. It is a helpful tool for comparing the statistical intensity between sub-fields and journals. The novel instrument may be applied in manuscript peer review to identify papers in need of additional statistical review.