A search filter to identify pragmatic trials in MEDLINE was highly specific but lacked sensitivity

The majority of clinical trials assessing mobilization and manipulation for neck pain lack a pragmatic approach: a systematic review of 174 trials

BACKGROUND

Disorders of the cervical spine are some of the costliest musculoskeletal conditions to manage globally. Joint mobilization and manipulation have been shown to be an effective treatment for neck pain. However, the generalizability and clinical translation depends on the nature of the trial designs that inform its use. The extent to which randomized control trials (RCTs) assessing manual therapy treatments for cervical spine disorders fall on the efficacy (explanatory) -effectiveness (pragmatic) spectrum often informs how the findings are translated into clinical practice.

OBJECTIVE

The aim of this systematic review was to determine where RCTs of manual therapy for neck disorders fall on the efficacy-effectiveness spectrum.

METHODS

A search of three electronic databases including PubMed, CINAHL, and CENTRAL were completed for trials published from inception to May 2023. RCTs in which joint mobilization or manipulation were used to treat cervical spine disorders were assessed on the effectiveness-efficacy spectrum using the Rating of Included Trials on the Efficacy-Effectiveness Spectrum (RITES) tool and risk of bias using the Revised Cochrane Risk of Bias tool.

RESULTS

A total of 174 trials met eligibility. RITES domain two trial setting (71.3% vs 16.1%), domain three flexibility of intervention(s) (62.1% vs 23%), and domain four clinical relevance of experimental and comparison intervention(s) (51.7% vs 29.3%) all favored efficacy over effectiveness. Domain one participant characteristic(s) had a slightly greater emphasis on effectiveness compared to efficacy (36.8% vs 44.8%). Most studies (96%) had at least some risk of bias.

CONCLUSION

Over half of the RCTs assessing the treatment effect of joint mobilization and manipulation for neck pain favor efficacy (explanatory) over effectiveness (pragmatic) designs. Future RCTs on this topic should consider a greater emphasis on pragmatic trial design components in order to better reflect real-world translation to clinical practice.

Development of a search filter to retrieve reports of interrupted time series studies from MEDLINE and PubMed

BACKGROUND

Interrupted time series (ITS) studies contribute importantly to systematic reviews of population-level interventions. We aimed to develop and validate search filters to retrieve ITS studies in MEDLINE and PubMed.

METHODS

A total of 1017 known ITS studies (published 2013-2017) were analysed using text mining to generate candidate terms. A control set of 1398 time-series studies were used to select differentiating terms. Various combinations of candidate terms were iteratively tested to generate three search filters. An independent set of 700 ITS studies was used to validate the filters' sensitivities. The filters were test-run in Ovid MEDLINE and the records randomly screened for ITS studies to determine their precision. Finally, all MEDLINE filters were translated to PubMed format and their sensitivities in PubMed were estimated.

RESULTS

Three search filters were created in MEDLINE: a precision-maximising filter with high precision (78%; 95% CI 74%-82%) but moderate sensitivity (63%; 59%-66%), most appropriate when there are limited resources to screen studies; a sensitivity-and-precision-maximising filter with higher sensitivity (81%; 77%-83%) but lower precision (32%; 28%-36%), providing a balance between expediency and comprehensiveness; and a sensitivity-maximising filter with high sensitivity (88%; 85%-90%) but likely very low precision, useful when combined with specific content terms. Similar sensitivity estimates were found for PubMed versions.

CONCLUSION

Our filters strike different balances between comprehensiveness and screening workload and suit different research needs. Retrieval of ITS studies would be improved if authors identified the ITS design in the titles.

Use of pragmatic randomized trials in multiple sclerosis: A systematic overview

BACKGROUND

Pragmatic trials are increasingly recognized for providing real-world evidence on treatment choices.

OBJECTIVE

The objective of this study is to investigate the use and characteristics of pragmatic trials in multiple sclerosis (MS).

METHODS

Systematic literature search and analysis of pragmatic trials on any intervention published up to 2022. The assessment of pragmatism with PRECIS-2 (PRagmatic Explanatory Continuum Indicator Summary-2) is performed.

RESULTS

We identified 48 pragmatic trials published 1967-2022 that included a median of 82 participants (interquartile range (IQR) = 42-160) to assess typically supportive care interventions (n = 41; 85%). Only seven trials assessed drugs (15%). Only three trials (6%) included >500 participants. Trials were mostly from the United Kingdom (n = 18; 38%), Italy (n = 6; 13%), the United States and Denmark (each n = 5; 10%). Primary outcomes were diverse, for example, quality-of-life, physical functioning, or disease activity. Only 1 trial (2%) used routinely collected data for outcome ascertainment. No trial was very pragmatic in all design aspects, but 14 trials (29%) were widely pragmatic (i.e. PRECIS-2 score ⩾ 4/5 in all domains).

CONCLUSION

Only few and mostly small pragmatic trials exist in MS which rarely assess drugs. Despite the widely available routine data infrastructures, very few trials utilize them. There is an urgent need to leverage the potential of this pioneering study design to provide useful randomized real-world evidence.

Patient and public involvement in pragmatic trials: online survey of corresponding authors of published trials

BACKGROUND

There are few data on patient and public involvement (PPI) in pragmatic trials. We aimed to describe the prevalence and nature of PPI within pragmatic trials, describe variation in prevalence of PPI by trial characteristics and compare prevalence of PPI reported by trial authors to that reported in trial publications.

METHODS

We applied a search filter to identify pragmatic trials published from 2014 to 2019 in MEDLINE. We invited the corresponding authors of pragmatic trials to participate in an online survey about their specific trial.

RESULTS

Of 3163 authors invited, 2585 invitations were delivered, 710 (27.5%) reported on 710 unique trials and completed the survey; 334 (47.0%) conducted PPI. Among those who conducted PPI, for many the aim was to increase the research relevance (86.3%) or quality (76.5%). Most PPI partners were engaged at protocol development stages (79.1%) and contributed to the co-design of interventions (70.9%) or recruitment or retention strategies (60.5%). Patient and public involvement was more common among trials involving children, trials conducted in the United Kingdom, cluster randomized trials, those explicitly labelled as "pragmatic" in the study manuscript, and more recent trials. Less than one-quarter of trials (22.8%) that reported PPI in the survey also reported PPI in the trial manuscript.

INTERPRETATION

Nearly half of trialists in this survey reported conducting PPI and listed several benefits of doing so, but researchers who did not conduct PPI often cited a lack of requirement for it. Patient and public involvement appears to be significantly underreported in trial publications. Consistent and standardized reporting is needed to promote transparency about PPI methods, outcomes, challenges and benefits.

Meta-research on pragmatism of randomized trials: rationale and design of the PragMeta database

BACKGROUND

Pragmatic trials provide decision-oriented, real-world evidence that is highly applicable and generalizable. The interest in real-world evidence is fueled by the assumption that effects in the "real-world" are different to effects obtained under artificial, controlled, research conditions as often used for traditional explanatory trials. However, it is unknown which features of pragmatism, generalizability, and applicability would be responsible for such differences. There is a need to provide empirical evidence and promote meta-research to answer these fundamental questions on the pragmatism of randomized trials and real-world evidence. Here, we describe the rationale and design of the PragMeta database which pursues this goal ( www.PragMeta.org ).

METHODS

PragMeta is a non-commercial, open data platform and infrastructure to facilitate research on pragmatic trials. It collects and shares data from published randomized trials that either have a specific design feature or other characteristic related to pragmatism or they form clusters of trials addressing the same research question but having different aspects of pragmatism. This lays the foundation to determine the relationship of various features of pragmatism, generalizability, and applicability with intervention effects or other trial characteristics. The database contains trial data actively collected for PragMeta but also allows to import and link existing datasets of trials collected for other purposes, forming a large-scale meta-database. PragMeta captures data on (1) trial and design characteristics (e.g., sample size, population, intervention/comparison, outcome, longitudinal structure, blinding), (2) effects estimates, and (3) various determinants of pragmatism (e.g., the use of routinely collected data) and ratings from established tools used to determine pragmatism (e.g., the PRagmatic-Explanatory Continuum Indicator Summary 2; PRECIS-2). PragMeta is continuously provided online, inviting the meta-research community to collaborate, contribute, and/or use the database. As of April 2023, PragMeta contains data from > 700 trials, mostly with assessments on pragmatism.

CONCLUSIONS

PragMeta will inform a better understanding of pragmatism and the generation and interpretation of real-world evidence.

Analysis of the characteristics and the degree of pragmatism exhibited by pragmatic-labelled trials of antineoplastic treatments

BACKGROUND

Pragmatic clinical trials (PCTs) are designed to reflect how an investigational treatment would be applied in clinical practice. As such, unlike their explanatory counterparts, they measure therapeutic effectiveness and are capable of generating high-quality real-world evidence. However, the conduct of PCTs remains extremely rare. The scarcity of such studies has contributed to the emergence of the efficacy-effectiveness gap and has led to calls for launching more of them, including in the field of oncology. This analysis aimed to identify self-labelled pragmatic trials of antineoplastic interventions and to evaluate whether their use of this label was justified.

METHODS

We searched PubMed® and Embase® for publications corresponding with studies that investigated antitumor therapies and that were tagged as pragmatic in their titles, abstracts and/or index terms. Subsequently, we consulted all available source documents for the included trials and extracted relevant information from them. The data collected were then used to appraise the degree of pragmatism displayed by the PCTs with the help of the validated PRECIS-2 tool.

RESULTS

The literature search returned 803 unique records, of which 46 were retained upon conclusion of the screening process. This ultimately resulted in the identification of 42 distinct trials that carried the 'pragmatic' label. These studies examined eight different categories of neoplasms and were mostly randomized, open-label, multicentric, single-country trials sponsored by non-commercial parties. On a scale of one (very explanatory) to five (very pragmatic), the median PCT had a PRECIS-2 score per domain of 3.13 (interquartile range: 2.57-3.53). The most and least pragmatic studies in the sample had a score of 4.44 and 1.57, respectively. Only a minority of trials were described in sufficient detail to allow them to be graded across all domains of the PRECIS-2 instrument. Many of the studies examined also had features that arguably precluded them from being pragmatic altogether, such as being monocentric or placebo-controlled in nature.

CONCLUSION

PCTs of antineoplastic treatments are generally no more pragmatic than they are explanatory.

A scoping review described diversity in methods of randomization and reporting of baseline balance in stepped-wedge cluster randomized trials

OBJECTIVES

In stepped-wedge cluster randomized trials (SW-CRTs), clusters are randomized not to treatment and control arms but to sequences dictating the times of crossing from control to intervention conditions. Randomization is an essential feature of this design but application of standard methods to promote and report on balance at baseline is not straightforward. We aimed to describe current methods of randomization and reporting of balance at baseline in SW-CRTs.

STUDY DESIGN AND SETTING

We used electronic searches to identify primary reports of SW-CRTs published between 2016 and 2022.

RESULTS

Across 160 identified trials, the median number of clusters randomized was 11 (Q1-Q3: 8-18). Sixty-three (39%) used restricted randomization-most often stratification based on a single cluster-level covariate; 12 (19%) of these adjusted for the covariate(s) in the primary analysis. Overall, 50 (31%) and 134 (84%) reported on balance at baseline on cluster- and individual-level characteristics, respectively. Balance on individual-level characteristics was most often reported by condition in cross-sectional designs and by sequence in cohort designs. Authors reported baseline imbalances in 72 (45%) trials.

CONCLUSION

SW-CRTs often randomize a small number of clusters using unrestricted allocation. Investigators need guidance on appropriate methods of randomization and assessment and reporting of balance at baseline.

Health equity considerations in pragmatic trials in Alzheimer's and dementia disease: Results from a methodological review

Introduction

To improve dementia care delivery for persons across all backgrounds, it is imperative that health equity is integrated into pragmatic trials.

Methods

We reviewed 62 pragmatic trials of people with dementia published 2014 to 2019. We assessed health equity in the objectives; design, conduct, analysis; and reporting using PROGRESS-Plus which stands for Place of residence, Race/ethnicity, Occupation, Gender/sex, Religion, Education, Socioeconomic status, Social capital, and other factors such as age and disability.

Results

Two (3.2%) trials incorporated equity considerations into their objectives; nine (14.5%) engaged with communities; 4 (6.5%) described steps to increase enrollment from equity-relevant groups. Almost all trials (59, 95.2%) assessed baseline balance for at least one PROGRESS-Plus characteristic, but only 10 (16.1%) presented subgroup analyses across such characteristics. Differential recruitment, attrition, implementation, adherence, and applicability across PROGRESS-Plus were seldom discussed.

Discussion

Ongoing and future pragmatic trials should more rigorously integrate equity considerations in their design, conduct, and reporting.

Highlights

Few pragmatic trials are explicitly designed to inform equity-relevant objectives.Few pragmatic trials take steps to increase enrollment from equity-relevant groups.Disaggregated results across equity-relevant groups are seldom reported.Adherence to existing tools (e.g., IMPACT Best Practices, CONSORT-Equity) is key.

Reporting of and explanations for under-recruitment and over-recruitment in pragmatic trials: a secondary analysis of a database of primary trial reports published from 2014 to 2019

OBJECTIVES

To describe the extent to which pragmatic trials underachieved or overachieved their target sample sizes, examine explanations and identify characteristics associated with under-recruitment and over-recruitment.

STUDY DESIGN AND SETTING

Secondary analysis of an existing database of primary trial reports published during 2014-2019, registered in ClinicalTrials.gov, self-labelled as pragmatic and with target and achieved sample sizes available.

RESULTS

Of 372 eligible trials, the prevalence of under-recruitment (achieving <90% of target sample size) was 71 (19.1%) and of over-recruitment (>110% of target) was 87 (23.4%). Under-recruiting trials commonly acknowledged that they did not achieve their targets (51, 71.8%), with the majority providing an explanation, but only 11 (12.6%) over-recruiting trials acknowledged recruitment excess. The prevalence of under-recruitment in individually randomised versus cluster randomised trials was 41 (17.0%) and 30 (22.9%), respectively; prevalence of over-recruitment was 39 (16.2%) vs 48 (36.7%), respectively. Overall, 101 025 participants were recruited to trials that did not achieve at least 90% of their target sample size. When considering trials with over-recruitment, the total number of participants recruited in excess of the target was a median (Q1-Q3) 319 (75-1478) per trial for an overall total of 555 309 more participants than targeted. In multinomial logistic regression, cluster randomisation and lower journal impact factor were significantly associated with both under-recruitment and over-recruitment, while using exclusively routinely collected data and educational/behavioural interventions were significantly associated with over-recruitment; we were unable to detect significant associations with obtaining consent, publication year, country of recruitment or public engagement.

CONCLUSIONS

A clear explanation for under-recruitment or over-recruitment in pragmatic trials should be provided to encourage transparency in research, and to inform recruitment to future trials with comparable designs. The issues and ethical implications of over-recruitment should be more widely recognised by trialists, particularly when designing cluster randomised trials.

An analysis of published trials found that current use of pragmatic trial labels is uninformative

OBJECTIVES

Randomized trials labelled as "pragmatic" are attractive to funders, patients, and clinicians as the label implies that the results are directly applicable to clinical care. We examined how authors justify use of the label (e.g., by referring to one or more PRECIS [PRagmatic Explanatory Continuum Indicator Summary]-2 domains).

STUDY DESIGN AND SETTING

We reviewed primary trial reports published 2014-2019, registered in ClinicalTrials.gov and using the pragmatic label anywhere in the report.

RESULTS

Among 415 trials, the label was justified by reference to at least one design element in 282 (68.0%); of these, 240 (85.1%) referenced trial characteristics that can be mapped to one or more of the PRECIS-2 domains, most commonly eligibility (91, 32.3%), setting (90, 31.9%), flexibility delivery (89, 31.6%), and organization (75, 26.6%); 42 (14.9%) referenced characteristics that are not PRECIS-2 domains, most commonly type of intervention/comparator (48, 17%), recruitment without consent (22, 7.8%), routinely collected data (22, 7.8%), and cluster randomization (20, 7.1%). Most reports referenced only one or two design elements. Overall, 9/415 (2.2%) provided PRECIS wheels.

CONCLUSION

Current use of pragmatic labels is uninformative. Authors should clarify the decision the trial is intended to support and include a PRECIS-2 table to make the design transparent.

Machine learning algorithms to identify cluster randomized trials from MEDLINE and EMBASE

BACKGROUND

Cluster randomized trials (CRTs) are becoming an increasingly important design. However, authors of CRTs do not always adhere to requirements to explicitly identify the design as cluster randomized in titles and abstracts, making retrieval from bibliographic databases difficult. Machine learning algorithms may improve their identification and retrieval. Therefore, we aimed to develop machine learning algorithms that accurately determine whether a bibliographic citation is a CRT report.

METHODS

We trained, internally validated, and externally validated two convolutional neural networks and one support vector machine (SVM) algorithm to predict whether a citation is a CRT report or not. We exclusively used the information in an article citation, including the title, abstract, keywords, and subject headings. The algorithms' output was a probability from 0 to 1. We assessed algorithm performance using the area under the receiver operating characteristic (AUC) curves. Each algorithm's performance was evaluated individually and together as an ensemble. We randomly selected 5000 from 87,633 citations to train and internally validate our algorithms. Of the 5000 selected citations, 589 (12%) were confirmed CRT reports. We then externally validated our algorithms on an independent set of 1916 randomized trial citations, with 665 (35%) confirmed CRT reports.

RESULTS

In internal validation, the ensemble algorithm discriminated best for identifying CRT reports with an AUC of 98.6% (95% confidence interval: 97.8%, 99.4%), sensitivity of 97.7% (94.3%, 100%), and specificity of 85.0% (81.8%, 88.1%). In external validation, the ensemble algorithm had an AUC of 97.8% (97.0%, 98.5%), sensitivity of 97.6% (96.4%, 98.6%), and specificity of 78.2% (75.9%, 80.4%)). All three individual algorithms performed well, but less so than the ensemble.

CONCLUSIONS

We successfully developed high-performance algorithms that identified whether a citation was a CRT report with high sensitivity and moderately high specificity. We provide open-source software to facilitate the use of our algorithms in practice.

Over Half of Clinical Trials of Mobilization and Manipulation for Patients With Low Back Pain May Have Limited Real-World Applicability: A Systematic Review of 132 Clinical Trials

OBJECTIVE

To evaluate the existing body of trials assessing manual therapy for low back pain (LBP) to determine where it falls on the efficacyeffectiveness continuum.

DESIGN

Methodology systematic review.

LITERATURE SEARCH

PubMed, CINAHL (Cumulative Index to Nursing and Allied Health Literature), CENTRAL (Cochrane Central Register of Controlled Trials), and PEDro (Physiotherapy Evidence Database) were searched for trials published between January 1, 2000, and April 30, 2021.

STUDY SELECTION CRITERIA

We included randomized clinical trials investigating joint mobilization and manipulation for adults with nonspecific LBP that were available in English.

DATA SYNTHESIS

We used the Rating of Included Trials on the Efficacy-Effectiveness Spectrum (RITES) tool to score included trials across 4 domains: participant characteristics, trial setting, flexibility of intervention(s), and clinical relevance of experimental and comparison intervention(s). Proportions of trials with greater emphasis on efficacy or effectiveness were calculated for each domain.

RESULTS

Of the 132 included trials, a greater proportion emphasized efficacy than effectiveness for domains participant characteristics (50% vs 38%), trial setting (71% vs 20%), and flexibility of intervention(s) (61% vs 25%). The domain clinical relevance of experimental and comparison intervention(s) had lower emphasis on efficacy (41% vs 50%).

CONCLUSION

Most trials investigating manual therapy for LBP lack pragmatism across the RITES domains (ie, they emphasize efficacy). To improve real-world implementation, more research emphasizing effectiveness is needed. This could be accomplished by recruiting from more diverse participant pools, involving multiple centers that reflect common clinical practice settings, involving clinicians with a variety of backgrounds/experience, and allowing flexibility in how interventions are delivered. J Orthop Sports Phys Ther 2022;52(8):532-545. Epub: 19 June 2022. doi:10.2519/jospt.2022.10962.

Ethical considerations within pragmatic randomized controlled trials in dementia: Results from a literature survey

Introduction

This review aims to describe the landscape of pragmatic randomized controlled trials (RCTs) in the context of Alzheimer's disease (AD) and related dementias with respect to ethical considerations.

Methods

Searches of MEDLINE were performed from January 2014 until April 2019. Extracted information included: trial setting, interventions, data collection, study population, and ethical protections (including ethics approvals, capacity assessment, and informed consent).

Results

We identified 62 eligible reports. More than two-thirds (69%) included caregivers or health-care professionals as research participants. Fifty-eight (94%) explicitly identified at least one vulnerable group. Two studies did not report ethics approval. Of 57 studies in which patients were participants, 55 (96%) reported that consent was obtained but in 37 studies (67%) no mention was made regarding assessment of the patients' capacity to consent to research participation.

Discussion

Few studies reported protections implemented when vulnerable participants were included. Shortcomings remain when reporting consent approaches and capacity assessment.

A review identified challenges distinguishing primary reports of randomized trials for meta-research: A proposal for improved reporting

Meta-research is the discipline of studying research itself. A core investigative tool in meta-research is the use of systematic or scoping reviews to study the characteristics, methods and reporting of primary research studies. In the context of identifying eligible publications for methodological reviews of randomized controlled trials (RCTs), a challenge is to efficiently distinguish the primary trial report - which reports results for the primary outcome - from other types of reports, including design papers and secondary or supplementary analyses, or what we collectively refer to as non-primary reports. This may not be a straightforward task and may contribute to inefficiencies in the review process. Here, we draw on our recent methodological review of over 13,000 records to identify primary reports of pragmatic RCTs. We offer recommendations to improve the reporting of RCTs to facilitate more efficient identification of primary trial reports. We suggest that future updates to existing CONSORT guidelines include consideration of multiple trial reports and recommendations to clarify the primary or non-primary nature of each report. Our recommendations, together with improved adherence to inclusion of the trial registration number in the abstract and citation of a protocol or previously published primary report, would facilitate the conduct of methodological reviews.

Patient-reported outcomes and target effect sizes in pragmatic randomized trials in ClinicalTrials.gov: A cross-sectional analysis

BACKGROUND

Use of patient-reported outcomes (PROs) and patient and public engagement are critical ingredients of pragmatic trials, which are intended to be patient centered. Engagement of patients and members of the public in selecting the primary trial outcome and determining the target difference can better ensure that the trial is designed to inform the decisions of those who ultimately stand to benefit. However, to the best of our knowledge, the use and reporting of PROs and patient and public engagement in pragmatic trials have not been described. The objectives of this study were to review a sample of pragmatic trials to describe (1) the prevalence of reporting patient and public engagement; (2) the prevalence and types of PROs used; (3) how its use varies across trial characteristics; and (4) how sample sizes and target differences are determined for trials with primary PROs.

METHODS AND FINDINGS

This was a methodological review of primary reports of pragmatic trials. We used a published electronic search filter in MEDLINE to identify pragmatic trials, published in English between January 1, 2014 and April 3, 2019; we identified the subset that were registered in ClinicalTrials.gov and explicitly labeled as pragmatic. Trial descriptors were downloaded from ClinicalTrials.gov; information about PROs and sample size calculations were extracted from the manuscript. Chi-squared, Cochran-Armitage, and Wilcoxon rank sum tests were used to examine associations between trial characteristics and use of PROs. Among 4,337 identified primary trial reports, 1,988 were registered in CT.gov, of which 415 were explicitly labeled as pragmatic. Use of patient and public engagement was identified in 39 (9.4%). PROs were measured in 235 (56.6%): 144 (34.7%) used PROs as primary outcomes and 91 (21.9%) as only secondary outcomes. Primary PROs were symptoms (64; 44%), health behaviors (36; 25.0%), quality of life (17; 11.8%), functional status (16; 11.1%), and patient experience (10; 6.9%). Trial characteristics with lower prevalence of use of PROs included being conducted exclusively in children or adults over age 65 years, cluster randomization, recruitment in low- and middle-income countries, and primary purpose of prevention; trials conducted in Europe had the highest prevalence of PROs. For the 144 trials with a primary PRO, 117 (81.3%) reported a sample size calculation for that outcome; of these, 71 (60.7%) justified the choice of target difference, most commonly, using estimates from pilot studies (31; 26.5%), standardized effect sizes (20; 17.1%), or evidence reviews (16; 13.7%); patient or stakeholder opinions were used to justify the target difference in 8 (6.8%). Limitations of this study are the need for trials to be registered in ClinicalTrials.gov, which may have reduced generalizability, and extracting information only from the primary trial report.

CONCLUSIONS

In this study, we observed that pragmatic trials rarely report patient and public engagement and do not commonly use PROs as primary outcomes. When provided, target differences are often not justified and rarely informed by patients and stakeholders. Research funders, scientific journals, and institutions should support trialists to incorporate patient engagement to fulfill the mandate of pragmatic trials to be patient centered.

Review of pragmatic trials found that multiple primary outcomes are common but so too are discrepancies between protocols and final reports

OBJECTIVES

To describe prevalence of multiple primary outcomes, changes in primary outcomes and target sample sizes between protocols and final reports, and how issues of multiplicity are addressed in pragmatic trials.

STUDY DESIGN AND SETTING

Individually randomised trials labelled as pragmatic, published 2014-2019 in MEDLINE and registered with ClinicalTrials.gov.

RESULTS

We identified 262 final reports and located protocols for 159 (61%); primary outcomes were clearly reported in 145 (91%) protocols and 256 (98%) final reports. Thirty (19%) protocols and 38 (15%) final reports had multiple primary outcomes. Primary outcomes were present and identical in 128 (81%) matched protocol-final reports. Among 140 pairs with target sample sizes reported, 28 (20.0%) reduced their target sample size (mean 543 fewer participants per trial) and 16 (11.4%) increased it (mean 192 more participants per trial). Thirteen (29.5%) provided an explanation. Only 2/30 (7%) protocols and 4/38 (11%) final reports with co-primary outcomes explained how results would be interpreted in light of multiplicity; 21/30 (70%) protocols and 20/38 (53%) final reports accounted for co-primary outcomes in power calculations.

CONCLUSION

Co-primary outcomes are common in pragmatic trials; improved transparency around design and analysis decisions involving co-primary outcomes is required.

Methodological challenges in pragmatic trials in Alzheimer's disease and related dementias: Opportunities for improvement

BACKGROUND AND AIMS

We need more pragmatic trials of interventions to improve care and outcomes for people living with Alzheimer's disease and related dementias. However, these trials present unique methodological challenges in their design, analysis, and reporting-often, due to the presence of one or more sources of clustering. Failure to account for clustering in the design and analysis can lead to increased risks of Type I and Type II errors. We conducted a review to describe key methodological characteristics and obtain a "baseline assessment" of methodological quality of pragmatic trials in dementia research, with a view to developing new methods and practical guidance to support investigators and methodologists conducting pragmatic trials in this field.

METHODS

We used a published search filter in MEDLINE to identify trials more likely to be pragmatic and identified a subset that focused on people living with Alzheimer's disease or other dementias or included them as a defined subgroup. Pairs of reviewers extracted descriptive information and key methodological quality indicators from each trial.

RESULTS

We identified N = 62 eligible primary trial reports published across 36 different journals. There were 15 (24%) individually randomized, 38 (61%) cluster randomized, and 9 (15%) individually randomized group treatment designs; 54 (87%) trials used repeated measures on the same individual and/or cluster over time and 17 (27%) had a multivariate primary outcome (e.g. due to measuring an outcome on both the patient and their caregiver). Of the 38 cluster randomized trials, 16 (42%) did not report sample size calculations accounting for the intracluster correlation and 13 (34%) did not account for intracluster correlation in the analysis. Of the 9 individually randomized group treatment trials, 6 (67%) did not report sample size calculations accounting for intracluster correlation and 8 (89%) did not account for it in the analysis. Of the 54 trials with repeated measurements, 45 (83%) did not report sample size calculations accounting for repeated measurements and 19 (35%) did not utilize at least some of the repeated measures in the analysis. No trials accounted for the multivariate nature of their primary outcomes in sample size calculation; only one did so in the analysis.

CONCLUSION

There is a need and opportunity to improve the design, analysis, and reporting of pragmatic trials in dementia research. Investigators should pay attention to the potential presence of one or more sources of clustering. While methods for longitudinal and cluster randomized trials are well developed, accessible resources and new methods for dealing with multiple sources of clustering are required. Involvement of a statistician with expertise in longitudinal and clustered designs is recommended.

Informed consent in pragmatic trials: results from a survey of trials published 2014-2019

OBJECTIVES

To describe reporting of informed consent in pragmatic trials, justifications for waivers of consent and reporting of alternative approaches to standard written consent. To identify factors associated with (1) not reporting and (2) not obtaining consent.

METHODS

Survey of primary trial reports, published 2014-2019, identified using an electronic search filter for pragmatic trials implemented in MEDLINE, and registered in ClinicalTrials.gov.

RESULTS

Among 1988 trials, 132 (6.6%) did not include a statement about participant consent, 1691 (85.0%) reported consent had been obtained, 139 (7.0%) reported a waiver and 26 (1.3%) reported consent for one aspect (eg, data collection) but a waiver for another (eg, intervention). Of the 165 trials reporting a waiver, 76 (46.1%) provided a justification. Few (53, 2.9%) explicitly reported use of alternative approaches to consent. In multivariable logistic regression analyses, lower journal impact factor (p=0.001) and cluster randomisation (p<0.0001) were significantly associated with not reporting on consent, while trial recency, cluster randomisation, higher-income country settings, health services research and explicit labelling as pragmatic were significantly associated with not obtaining consent (all p<0.0001).

DISCUSSION

Not obtaining consent seems to be increasing and is associated with the use of cluster randomisation and pragmatic aims, but neither cluster randomisation nor pragmatism are currently accepted justifications for waivers of consent. Rather than considering either standard written informed consent or waivers of consent, researchers and research ethics committees could consider alternative consent approaches that may facilitate the conduct of pragmatic trials while preserving patient autonomy and the public's trust in research.

Secondary electronic sources demonstrated very good sensitivity for identifying studies evaluating interventions for COVID-19

Objectives

To assess the sensitivity of two secondary electronic sources of COVID-19 studies: 1) the Cochrane COVID-19 Study Register (https://covid-19.cochrane.org/); and, 2) the Living Overview of the Evidence (L•OVE) COVID-19 platform (https://iloveevidence.com/).

Study design and setting

We identified reports of randomized controlled trials (RCTs) and observational studies (OS) assessing preventive interventions or treatment for COVID-19.

The reference standard comprised all reports included in the COVID-NMA platform (covid-nma.com), in two major living systematic reviews of RCTs assessing pharmacologic treatment of COVID-19, or identified in either of the two secondary sources evaluated. The search for all sources was conducted through September 7, 2020.

Our primary outcome was the proportion of the reports included in the reference standard that were identified by each secondary source.

Results

We identified 680 reports, 91 RCT reports, 97 RCT protocols, and 492 OS reports. The Cochrane COVID-19 Study Register identified 88% [95% confidence interval, 79–94] of the RCT reports, 90% [82–95] of the RCT protocols, and 82% [78–85] of the OS reports. The L•OVE platform identified 100% [97–100] of the RCT reports and RCT protocols and 100% [99–100] of the OS reports.

Conclusion

These platforms proved to be a viable screening alternative to searching every individual source.

A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and trial registry data, and poor indexing

OBJECTIVE

We established a large database of trials to serve as a resource for future methodological and ethical analyses. Here, we use meta-data to describe the broad landscape of pragmatic trials including research areas, identification as pragmatic, quality of trial registry data and enrolment.

STUDY DESIGN AND SETTING

Trials were identified by a validated search filter and included if a primary report of a health-related randomized trial published January 2014-April 2019. Data were collated from MEDLINE, Web of Science, ClinicalTrials.gov, and full text.

RESULTS

4337 eligible trials were identified from 13,065 records, of which 1988 were registered in ClinicalTrials.gov. Research areas were diverse, with the most common being general and internal medicine; public, environmental and occupational health; and health care sciences and services. The term "pragmatic" was seldom used in titles or abstracts. Several domains in ClinicalTrials.gov had questionable data quality. We estimated that one-fifth of trials under-accrued by at least 15%.

CONCLUSION

There is a need to improve reporting of pragmatic trials and quality of trial registry data. Under accrual remains a challenge in pragmatic RCTs despite calls for more streamlined recruitment approaches. The diversity of pragmatic trials should be reflected in future ethical analyses.