Risk-adapted approaches to the management of clinical trials: guidance from the Department of Health (DH)/Medical Research Council(MRC)/Medicines and Healthcare Products Regulatory Agency (MHRA) Clinical Trials Working Group

Determining a risk-proportionate approach to the validation of statistical programming for clinical trials

BACKGROUND

The contribution of the statistician to the design and analysis of a clinical trial is acknowledged as essential. Ability to reconstruct the statistical contribution to a trial requires rigorous and transparent documentation as evidenced by the reproducibility of results. The process of validating statistical programmes is a key requirement. While guidance relating to software development and life cycle methodologies details steps for validation by information systems developers, there is no guidance applicable to programmes written by statisticians. We aimed to develop a risk-based approach to the validation of statistical programming that would support scientific integrity and efficient resource use within clinical trials units.

METHODS

The project was embedded within the Information Systems Operational Group and the Statistics Operational Group of the UK Clinical Research Collaboration Registered Clinical Trials Unit network. Members were asked to share materials relevant to validation of statistical programming. A review of the published literature, regulatory guidance and knowledge of relevant working groups was undertaken. Surveys targeting the Information Systems Operational Group and Statistics Operational Group were developed to determine current practices across the Registered Clinical Trials Unit network. A risk-based approach was drafted and used as a basis for a workshop with representation from statisticians, information systems developers and quality assurance managers (n = 15). The approach was subsequently modified and presented at a second, larger scale workshop (n = 47) to gain a wider perspective, with discussion of content and implications for delivery. The approach was revised based on the discussions and suggestions made. The workshop was attended by a member of the Medicines for Healthcare products Regulatory Agency Inspectorate who also provided comments on the revised draft.

RESULTS

Types of statistical programming were identified and categorised into six areas: generation of randomisation lists; programmes to explore/understand the data; data cleaning, including complex checks; derivations including data transformations; data monitoring; or interim and final analysis. The risk-based approach considers each category of statistical programme against the impact of an error and its likelihood, whether the programming can be fully prespecified, the need for repeated use and the need for reproducibility. Approaches to the validation of programming within each category are proposed.

CONCLUSION

We have developed a risk-based approach to the validation of statistical programming. It endeavours to facilitate the implementation of targeted quality assurance measures while making efficient use of limited resources.

Monitoring strategies for clinical intervention studies

BACKGROUND

Trial monitoring is an important component of good clinical practice to ensure the safety and rights of study participants, confidentiality of personal information, and quality of data. However, the effectiveness of various existing monitoring approaches is unclear. Information to guide the choice of monitoring methods in clinical intervention studies may help trialists, support units, and monitors to effectively adjust their approaches to current knowledge and evidence.

OBJECTIVES

To evaluate the advantages and disadvantages of different monitoring strategies (including risk-based strategies and others) for clinical intervention studies examined in prospective comparative studies of monitoring interventions.

SEARCH METHODS

We systematically searched CENTRAL, PubMed, and Embase via Ovid for relevant published literature up to March 2021. We searched the online 'Studies within A Trial' (SWAT) repository, grey literature, and trial registries for ongoing or unpublished studies.

SELECTION CRITERIA

We included randomized or non-randomized prospective, empirical evaluation studies of different monitoring strategies in one or more clinical intervention studies. We applied no restrictions for language or date of publication.

DATA COLLECTION AND ANALYSIS

We extracted data on the evaluated monitoring methods, countries involved, study population, study setting, randomization method, and numbers and proportions in each intervention group. Our primary outcome was critical and major monitoring findings in prospective intervention studies. Monitoring findings were classified according to different error domains (e.g. major eligibility violations) and the primary outcome measure was a composite of these domains. Secondary outcomes were individual error domains, participant recruitment and follow-up, and resource use. If we identified more than one study for a comparison and outcome definitions were similar across identified studies, we quantitatively summarized effects in a meta-analysis using a random-effects model. Otherwise, we qualitatively summarized the results of eligible studies stratified by different comparisons of monitoring strategies. We used the GRADE approach to assess the certainty of the evidence for different groups of comparisons.

MAIN RESULTS

We identified eight eligible studies, which we grouped into five comparisons. 1. Risk-based versus extensive on-site monitoring: based on two large studies, we found moderate certainty of evidence for the combined primary outcome of major or critical findings that risk-based monitoring is not inferior to extensive on-site monitoring. Although the risk ratio was close to 'no difference' (1.03 with a 95% confidence interval [CI] of 0.81 to 1.33, below 1.0 in favor of the risk-based strategy), the high imprecision in one study and the small number of eligible studies resulted in a wide CI of the summary estimate. Low certainty of evidence suggested that monitoring strategies with extensive on-site monitoring were associated with considerably higher resource use and costs (up to a factor of 3.4). Data on recruitment or retention of trial participants were not available. 2. Central monitoring with triggered on-site visits versus regular on-site visits: combining the results of two eligible studies yielded low certainty of evidence with a risk ratio of 1.83 (95% CI 0.51 to 6.55) in favor of triggered monitoring intervention. Data on recruitment, retention, and resource use were not available. 3. Central statistical monitoring and local monitoring performed by site staff with annual on-site visits versus central statistical monitoring and local monitoring only: based on one study, there was moderate certainty of evidence that a small number of major and critical findings were missed with the central monitoring approach without on-site visits: 3.8% of participants in the group without on-site visits and 6.4% in the group with on-site visits had a major or critical monitoring finding (odds ratio 1.7, 95% CI 1.1 to 2.7; P = 0.03). The absolute number of monitoring findings was very low, probably because defined major and critical findings were very study specific and central monitoring was present in both intervention groups. Very low certainty of evidence did not suggest a relevant effect on participant retention, and very low certainty evidence indicated an extra cost for on-site visits of USD 2,035,392. There were no data on recruitment. 4. Traditional 100% source data verification (SDV) versus targeted or remote SDV: the two studies assessing targeted and remote SDV reported findings only related to source documents. Compared to the final database obtained using the full SDV monitoring process, only a small proportion of remaining errors on overall data were identified using the targeted SDV process in the MONITORING study (absolute difference 1.47%, 95% CI 1.41% to 1.53%). Targeted SDV was effective in the verification of source documents, but increased the workload on data management. The other included study was a pilot study, which compared traditional on-site SDV versus remote SDV and found little difference in monitoring findings and the ability to locate data values despite marked differences in remote access in two clinical trial networks. There were no data on recruitment or retention. 5. Systematic on-site initiation visit versus on-site initiation visit upon request: very low certainty of evidence suggested no difference in retention and recruitment between the two approaches. There were no data on critical and major findings or on resource use.

AUTHORS' CONCLUSIONS

The evidence base is limited in terms of quantity and quality. Ideally, for each of the five identified comparisons, more prospective, comparative monitoring studies nested in clinical trials and measuring effects on all outcomes specified in this review are necessary to draw more reliable conclusions. However, the results suggesting risk-based, targeted, and mainly central monitoring as an efficient strategy are promising. The development of reliable triggers for on-site visits is ongoing; different triggers might be used in different settings. More evidence on risk indicators that identify sites with problems or the prognostic value of triggers is needed to further optimize central monitoring strategies. In particular, approaches with an initial assessment of trial-specific risks that need to be closely monitored centrally during trial conduct with triggered on-site visits should be evaluated in future research.

An evaluation of risk-based monitoring in pragmatic trials in UK Clinical Trials Units

Background

Good Clinical Practice guidelines issued in 2016 encourage risk-based approaches to monitoring clinical trials. This study compared current risk assessment and monitoring approaches in UK Clinical Trials Units (CTUs) with the published guidance and makes recommendations for risk-based monitoring in pragmatic trials.

Methods

An online survey of UK Clinical Research Collaboration registered CTUs was administered via email invitation. Forty-nine units were invited, and 23 responded. Respondents were also invited to share copies of risk assessment templates.

Results

Most CTUs reported using remote combined with on-site monitoring. All reported undertaking a risk assessment for Clinical Trials of Investigational Medicinal Products (CTIMPs) and 21 units did so for non-CTIMPs. Most CTIMP risk assessments used MHRA (Medicines and Healthcare products Regulatory Agency) classifications, although some also employed staff judgement. Almost all units based their monitoring on perceived risk level; this number was higher for CTIMPs (n = 22) than for non-CTIMPs (n = 19). In most cases, monitoring plans were produced. More CTUs revisited risk assessments during trials in CTIMPs (n = 21) than in non-CTIMPs (n = 18). Small numbers of units reviewed the monitoring approach always (n = 4) or sometimes (n = 9) and few used the reflection to guide future monitoring.

Conclusions

A high proportion of UK CTUs are using risk-based monitoring in the UK, as recommended by guidelines, for both CTIMPs and non-CTIMPs. This has the potential to make trials more efficient and reduce costs. However, there appears to be a lack of reflection on the value of these revised approaches. There may be a benefit in CTUs collaborating nationally to improve processes for reflection and making changes during the life course of a trial.

Electronic supplementary material

The online version of this article (10.1186/s13063-019-3619-6) contains supplementary material, which is available to authorized users.

Risk-Based Monitoring (RBM) Implementation: Challenges and Potential Solutions

Risk-based monitoring (RBM) has disrupted the clinical trial industry, challenging conventional monitoring norms, business processes, and organizational structures. Endorsed by regulators and leading industry forums, and further driven by escalating drug development costs and enabling technology shifts making data available real time, the industry is moving from a mode of recalcitrance to acceptance. The effective implementation of RBM requires delicately interweaving changes in technology, processes, people, and perspectives. This article deliberates upon the multiple challenges that exist and proposes potential solutions.