Post-publication peer review and evidence appraisals in primary care

A comment-driven evidence appraisal approach to promoting research findings into practice when only uncertain evidence is available

BACKGROUND

Comments in PubMed are usually short papers for supporting or refuting claims, or discussing methods and findings in original articles. This study aims to explore whether they can be used as a quick and reliable evidence appraisal instrument for promoting research findings into practice, especially in emergency situations such as COVID-19 in which only missing, incomplete or uncertain evidence is available.

METHODS

Evidence-comment networks (ECNs) were constructed by linking COVID-19-related articles to the commentaries (letters, editorials or brief correspondence) they received. PubTator Central was used to extract entities with a high volume of comments from the titles and abstracts of the articles. Among them, six drugs were selected, and their evidence assertions were analysed by exploring the structural information in the ECNs as well as the sentiment of the comments (positive, negative, neutral). Recommendations in WHO guidelines were used as the gold standard control to validate the consistency, coverage and efficiency of comments in reshaping clinical knowledge claims.

RESULTS

The overall positive/negative sentiments of comments were aligned with recommendations for/against the corresponding treatments in the WHO guidelines. Comment topics covered all significant points of evidence appraisal and beyond. Furthermore, comments may indicate the uncertainty regarding drug use for clinical practice. Half of the critical comments emerged 4.25 months earlier on average than the guideline release.

CONCLUSIONS

Comments have the potential as a support tool for rapid evidence appraisal as they have a selection effect by appraising the benefits, limitations and other clinical practice issues of concern in existing evidence. We suggest as a future direction an appraisal framework based on the comment topics and sentiment orientations to leverage the potential of scientific commentaries supporting evidence appraisal and decision-making.

Altmetric analysis of the contemporary scientific literature in Endodontology

AIM

To analyse and visualize the knowledge structure of scientific articles in the field of Endodontology with high altmetric attention scores to discover hot topics, active researchers and the journals involved.

METHODOLOGY

On 5 June 2019, the altmetric database (Altmetric LLP, London, UK) was searched using the titles of 11 endodontic journals. Bibliometric data from endodontic articles and journals with an altmetric score >5 (top 5%) were retrieved from PubMed and analysed using the VOSviewer. Science mapping of articles with an altmetric score >5 at two levels was created: author keywords co-occurrence and co-authorship network analysis.

RESULTS

Of the 2197 articles in the field of Endodontology identified with altmetrics, 192 had altmetric scores >5 (top 5%). Considering the total mentions amongst all altmetric resources, the Journal of Endodontics had the highest rank followed by the International Endodontic Journal and Australian Endodontic Journal. Twitter was the most popular altmetric data resource followed by patents and Facebook. Meta-analysis, systematic review and pulpitis were the hot topics. At the author level, Dummer P.M.H had the greatest influence on the network. There was no significant correlation between altmetric score and citations count (P > 0.05). Mendeley mentions correlated with citations (P < 0.05).

CONCLUSIONS

Overall, the altmetric scores of topics within Endodontology were low, possibly due to the specific and specialized nature of the specialty, as well as the difficulty members of the public probably have in understanding endodontic research. Journals and researchers with a focus on Endodontology would have more influence if they were to set-up their own social media profiles and thus enhance their visibility and social impact by immediately sharing research findings and communicating with their network and audience.

Analysis of evidence appraisals for interventional studies in family medicine using an informatics approach

This study reports the first assessment of published comments in the family medicine literature using structured codes, which produced commentary annotations that will be the foundation of a knowledge base of appraisals of family medicine trials. Evidence appraisal occurs in a variety of formats and serves to shed light on the quality of research. However, scientific discourse generally and evidence appraisal in particular has not itself been analyzed for insights. A search strategy was devised to identify all journal comments indexed in PubMed linked to controlled intervention studies published in a recent 15-year period in major family medicine journals. A previously developed structured representation in the form of a list of appraisal concepts was used to formally annotate and categorize the journal comments through an iterative process. Trends in family medicine evidence appraisal were then analyzed. A total of 93 comments on studies from five journals over 15 years were included in the analysis. Two thirds of extracted appraisals were negative criticisms. All appraisals of measurement instruments were negative (100%). The participants baseline characteristics, the author discussions, and the design of the interventions were also criticized (respectively 91.7%, 84.6% and 83.3% negative). In contrast, appraisals of the scientific basis of the studies were positive (81.8%). The categories with the most appraisals were, most generally, those focused on the study design, and most specifically, those focused on the scientific basis. This study provides a new data-driven approach to review scientific discourse regarding the strengths and limitations of research within academic family medicine. This methodology can potentially generalize to other medical domains. Structured appraisal data generated here will enable future clinical, scientific, and policy decision-making and broader meta-research in family medicine.