Meta-research: bird's eye views of primary care research

The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study

BACKGROUND

Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables.

METHODS

This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis.

RESULTS

We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals.

CONCLUSIONS

The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.

Health outcomes in primary care: a 20-year evidence map of randomized controlled trials

OBJECTIVE

To quantify the different types of health outcomes assessed as primary outcomes in randomized controlled trials (RCTs) in the primary care (PC) setting during the last 20 years and identify whether potential gaps exist in specific types of health care and types of intervention.

METHODS

We systematically searched PubMed, Scopus, and Cochrane Central Register of Controlled Trials, from January 2000 to September 2020 for published RCTs in PC. We recorded characteristics of eligible studies and mapped evidence by health outcome category (patient health outcomes, health services outcomes); and for each outcome category, by types of health care (preventive, acute, chronic, palliative), and by types of intervention (drug, behavioural, on structure, and on process). For RCTs assessing patient health outcomes as primary outcomes, we further mapped using the quality-of-care dimensions, that is, effectiveness, safety, and patient-centredness.

RESULTS

Of the 518 eligible RCTs in PC, 357 (68.9%) evaluated a patient health outcome as the primary outcome, and 161 (31.1%) evaluated only health services outcomes as primary outcomes. Many focused on population with chronic illness (224 trials; 43.2%) and evaluated interventions on processes of health care (239 trials; 46.1%). Research gaps identified include preventive and palliative care, behavioural interventions, and safety and patient-centredness outcomes as primary outcomes.

CONCLUSION

Our evidence map showed research gaps in certain types of health care and interventions. It also showed research gaps in assessing safety and measures to place patient at the centre of health care delivery as primary outcomes.

Evidence reversals in primary care research: a study of randomized controlled trials

BACKGROUND

Evidence-Based Medicine is built on the premise that clinicians can be more confident when their decisions are grounded in high-quality evidence. Furthermore, evidence from studies involving patient-oriented outcomes is preferred when making decisions about tests or treatments. Ideally, the findings of relevant and valid trials should be stable over time, that is, unlikely to be reversed in subsequent research.

OBJECTIVE

To evaluate the stability of evidence from trials relevant to primary healthcare and to identify study characteristics associated with their reversal.

METHODS

We studied synopses of randomized controlled trials (RCTs) published from 2002 to 2005 as "Daily POEMs" (Patient Oriented Evidence that Matters). The initial evidence (E1) from these POEMs (2002-2005) was compared with the updated evidence (E2) on that same topic in a summary resource (DynaMed 2019). Two physician-raters independently categorized each POEM-RCT as (i) reversed when E1 ≠ E2, or as (ii) not reversed, when E1 = E2. For all "Evidence Reversals" (E1 ≠ E2), we assessed the direction of change in the evidence.

RESULTS

We evaluated 408 POEMs on RCTs. Of those, 35 (9%; 95% confidence interval [6-12]) were identified as reversed, 359 (88%) were identified as not reversed, and 14 (3%) were indeterminate. On average, this represents about 2 evidence reversals per annum for POEMs about RCTs.

CONCLUSIONS

Over 12-17 years, 9% of RCTs summarized as POEMs are reversed. Information alerting services that apply strict criteria for relevance and validity of clinical information are likely to identify RCTs whose findings are stable over time.