Reproducibility in the absence of selective reporting: An illustration from large-scale brain asymmetry research

The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p-hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multisite collaborative study, we mapped brain anatomical left-right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we revisited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and significance threshold. In this sense, the results within each dataset were viewed as coming from separate studies in an "ideal publishing environment," that is, free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.

Mega-analysis methods in ENIGMA: The experience of the generalized anxiety disorder working group

The ENIGMA group on Generalized Anxiety Disorder (ENIGMA-Anxiety/GAD) is part of a broader effort to investigate anxiety disorders using imaging and genetic data across multiple sites worldwide. The group is actively conducting a mega-analysis of a large number of brain structural scans. In this process, the group was confronted with many methodological challenges related to study planning and implementation, between-country transfer of subject-level data, quality control of a considerable amount of imaging data, and choices related to statistical methods and efficient use of resources. This report summarizes the background information and rationale for the various methodological decisions, as well as the approach taken to implement them. The goal is to document the approach and help guide other research groups working with large brain imaging data sets as they develop their own analytic pipelines for mega-analyses.

Common functional localizers to enhance NHP & cross-species neuroscience imaging research

Functional localizers are invaluable as they can help define regions of interest, provide cross-study comparisons, and most importantly, allow for the aggregation and meta-analyses of data across studies and laboratories. To achieve these goals within the non-human primate (NHP) imaging community, there is a pressing need for the use of standardized and validated localizers that can be readily implemented across different groups. The goal of this paper is to provide an overview of the value of localizer protocols to imaging research and we describe a number of commonly used or novel localizers within NHPs, and keys to implement them across studies. As has been shown with the aggregation of resting-state imaging data in the original PRIME-DE submissions, we believe that the field is ready to apply the same initiative for task-based functional localizers in NHP imaging. By coming together to collect large datasets across research group, implementing the same functional localizers, and sharing the localizers and data via PRIME-DE, it is now possible to fully test their robustness, selectivity and specificity. To do this, we reviewed a number of common localizers and we created a repository of well-established localizer that are easily accessible and implemented through the PRIME-RE platform.

Strategies for the use of Ginkgo biloba extract, EGb 761® , in the treatment and management of mild cognitive impairment in Asia: Expert consensus

BACKGROUND

Mild cognitive impairment (MCI) is a neurocognitive state between normal cognitive aging and dementia, with evidence of neuropsychological changes but insufficient functional decline to warrant a diagnosis of dementia. Individuals with MCI are at increased risk for progression to dementia; and an appreciable proportion display neuropsychiatric symptoms (NPS), also a known risk factor for dementia. Cerebrovascular disease (CVD) is thought to be an underdiagnosed contributor to MCI/dementia. The Ginkgo biloba extract, EGb 761® , is increasingly being used for the symptomatic treatment of cognitive disorders with/without CVD, due to its known neuroprotective effects and cerebrovascular benefits.

AIMS

To present consensus opinion from the ASian Clinical Expert group on Neurocognitive Disorders (ASCEND) regarding the role of EGb 761® in MCI.

MATERIALS & METHODS

The ASCEND Group reconvened in September 2019 to present and critically assess the current evidence on the general management of MCI, including the efficacy and safety of EGb 761® as a treatment option.

RESULTS

EGb 761® has demonstrated symptomatic improvement in at least four randomized trials, in terms of cognitive performance, memory, recall and recognition, attention and concentration, anxiety, and NPS. There is also evidence that EGb 761® may help delay progression from MCI to dementia in some individuals.

DISCUSSION

EGb 761® is currently recommended in multiple guidelines for the symptomatic treatment of MCI. Due to its beneficial effects on cerebrovascular blood flow, it is reasonable to expect that EGb 761® may benefit MCI patients with underlying CVD.

CONCLUSION

As an expert group, we suggest it is clinically appropriate to incorporate EGb 761® as part of the multidomain intervention for MCI.

Development and reliability of questionnaires for the assessment of diet and physical activity behaviors in a multi-country sample in Europe the Feel4Diabetes Study

BACKGROUND

Assessment of diet and physical activity and their determinants still remains a demanding task, especially when the objective is to evaluate the efficacy of lifestyle interventions. In the context of the Feel4Diabetes study (a European community based intervention study in families with school aged children and at high risk of developing diabetes), we aimed to develop questionnaires for the assessment of food-frequency and eating behaviors, and physical activity and sedentary behaviors in both parents and school-aged children and a questionnaire for overall family's energy balance-related behaviors.

METHODS

Questionnaires were developed to be used in 6 countries under standardized harmonization procedures and included questions regarding not only food intake and physical activity, but also questions of their determinants. A reliability study was conducted in 191 pairs of parents and their children (N = 191). Parents completed the questionnaires on two occasions, within a 1-2 week interval. Reliability was tested by the intra-class correlation coefficients (ICC) of test-retest.

RESULTS

Most of the questions in all questionnaires had excellent reliability, assessed as an ICC of > 0.810. Mean ICCs for food-frequency and eating behaviors questionnaires were 0.838 and 0.787, and for physical activity and sedentary behaviors questionnaires were 0.734 and 0.793, in adults and children respectively. Mean ICC for overall family's energy balance-related behaviors and their determinants was 0.659.

CONCLUSION

The developed questionnaires showed acceptable reliability and may be valuable tools in the assessment of children's and parents' behaviors related to diet, physical activity, sedentary behavior and overall energy balance in school- and community-based interventions.

International multicenter examination of MOG antibody assays

OBJECTIVE

To compare the reproducibility of 11 antibody assays for immunoglobulin (Ig) G and IgM myelin oligodendrocyte glycoprotein antibodies (MOG-IgG and MOG-IgM) from 5 international centers.

METHODS

The following samples were analyzed: MOG-IgG clearly positive sera (n = 39), MOG-IgG low positive sera (n = 39), borderline negative sera (n = 13), clearly negative sera (n = 40), and healthy blood donors (n = 30). As technical controls, 18 replicates (9 MOG-IgG positive and 9 negative) were included. All samples and controls were recoded, aliquoted, and distributed to the 5 testing centers, which performed the following antibody assays: 5 live and 1 fixed immunofluorescence cell-based assays (CBA-IF, 5 MOG-IgG, and 1 MOG-IgM), 3 live flow cytometry cell-based assays (CBA-FACS, all MOG-IgG), and 2 ELISAs (both MOG-IgG).

RESULTS

We found excellent agreement (96%) between the live CBAs for MOG-IgG for samples previously identified as clearly positive or negative from 4 different national testing centers. The agreement was lower with fixed CBA-IF (90%), and the ELISA showed no concordance with CBAs for detection of human MOG-IgG. All CBAs showed excellent interassay reproducibility. The agreement of MOG-IgG CBAs for borderline negative (77%) and particularly low positive (33%) samples was less good. Finally, most samples from healthy blood donors (97%) were negative for MOG-IgG in all CBAs.

CONCLUSIONS

Live MOG-IgG CBAs showed excellent agreement for high positive and negative samples at 3 international testing centers. Low positive samples were more frequently discordant than in a similar comparison of aquaporin-4 antibody assays. Further research is needed to improve international standardization for clinical care.

Towards Precision Medicine in Psychosis: Benefits and Challenges of Multimodal Multicenter Studies-PSYSCAN: Translating Neuroimaging Findings From Research into Clinical Practice

In the last 2 decades, several neuroimaging studies investigated brain abnormalities associated with the early stages of psychosis in the hope that these could aid the prediction of onset and clinical outcome. Despite advancements in the field, neuroimaging has yet to deliver. This is in part explained by the use of univariate analytical techniques, small samples and lack of statistical power, lack of external validation of potential biomarkers, and lack of integration of nonimaging measures (eg, genetic, clinical, cognitive data). PSYSCAN is an international, longitudinal, multicenter study on the early stages of psychosis which uses machine learning techniques to analyze imaging, clinical, cognitive, and biological data with the aim of facilitating the prediction of psychosis onset and outcome. In this article, we provide an overview of the PSYSCAN protocol and we discuss benefits and methodological challenges of large multicenter studies that employ neuroimaging measures.

International Clinical Research and Justice in the Belmont Report

The Belmont Report was written by a US Commission charged by the US Congress to advise on research supported by the US government. Its focus was understandably domestic. In the 40 years since its publication, clinical research has become increasingly international. Many clinical trials have sites in multiple countries, and many of the host countries are relatively impoverished. Such research raises some distinctive ethical issues. This paper outlines some of the key ethical challenges that have been raised by clinical research conducted in low- and middle-income countries (LMICs) and sponsored by high-income country (HIC) institutions. It then considers whether the Belmont Report has the resources to address these problems and argues that it does not. The article closes by noting some parallels between this international research and domestic US research, which suggest that the US might benefit from the discussions abroad.

Standardised quantitative radioiodine SPECT/CT Imaging for multicentre dosimetry trials in molecular radiotherapy

The SEL-I-METRY trial (EudraCT No 2015-002269-47) is the first multicentre trial to investigate the role of 123I and 131I SPECT/CT-based tumour dosimetry to predict response to radioiodine therapy. Standardised dosimetry methodology is essential to provide a robust evidence-base for absorbed dose-response thresholds for molecular radiotherapy (MRT). In this paper a practical standardised protocol is used to establish the first network of centres with consistent methods of radioiodine activity quantification. Nine SPECT/CT systems at eight centres were set-up for quantitative radioiodine imaging. The dead-time of the systems was characterised for up to 2.8 GBq 131I. Volume dependent calibration factors were measured on centrally reconstructed images of 123I and 131I in six (0.8-196 ml) cylinders. Validation of image quantification using these calibration factors was performed on three systems, by imaging a 3D-printed phantom mimicking a patient's activity distribution. The percentage differences between the activities measured in the SPECT/CT image and those measured by the radionuclide calibrator were calculated. Additionally uncertainties on the SPECT/CT-based activities were calculated to indicate the limit on the quantitative accuracy of this method. For systems set-up to image high 131I count rates, the count rate versus activity did not peak below 2.8 GBq and fit a non-paralysable model. The dead-times and volume-dependent calibration factors were comparable between systems of the same model and crystal thickness. Therefore a global calibration curve could be fitted to each. The errors on the validation phantom activities' were comparable to the measurement uncertainties derived from uncertainty analysis, at 10% and 16% on average for 123I and 131I respectively in a 5 cm sphere. In conclusion, the dead-time and calibration factors varied between centres, with different models of system. However, global calibration factors may be applied to the same system model with the same crystal thickness, to simplify set-up of future multi-centre MRT studies.

The LEADER trial in type 2 diabetes: Were the characteristics and outcomes of the participants representative?

AIMS

To compare the characteristics and outcomes of people with type 2 diabetes recruited to the LEADER trial to those of participants in the contemporaneous community-based Fremantle Diabetes Study Phase II (FDS2) who fulfilled LEADER entry criteria.

METHODS

Baseline characteristics of LEADER and LEADER-eligible FDS2 participants were compared using bivariate methods. Incidence rates of the primary (nonfatal myocardial infarction, nonfatal stroke, cardiovascular disease (CVD) death) and other outcomes in the LEADER placebo group were compared with those in LEADER-eligible FDS2 participants during 3.8 years after entry, the median LEADER follow-up.

RESULTS

Of 1551 FDS2 type 2 participants, 323 (20.8%) were LEADER-eligible. Compared with the LEADER sample, they were an average 6 years older, and were less likely to be male, obese and to have prior CVD. There were 3.9 and 2.9 primary outcomes/100 patient-years in LEADER placebo-treated and FDS2 LEADER-eligible patients, respectively. Incidence rates for first myocardial infarction and stroke were 1.9 and 2.1 events/100 patient-years and 1.1 and 1.0 events/100 patient-years, respectively. FDS2 LEADER-eligible patients had a lower CVD death rate of 0.8 versus 1.6/100 patient-years in the LEADER placebo group, but their non-CVD mortality was greater (2.1 versus 1.0/100 patient-years).

CONCLUSIONS

These data suggest recruitment bias in type 2 diabetes CVD outcome trials.

Detecting Anomalies Among Practice Sites Within Multicenter Trials

BACKGROUND

Recent multisite trials reveal striking heterogeneities in results between trial sites. These may be because of population differences indicating different treatment benefits among different types of participants or site anomalies, such as failures to adhere to study protocols that could negatively affect study validity. We sought to determine whether a new data analysis strategy-transportability methods-could suggest site anomalies not readily identified through standard methods.

METHODS AND RESULTS

We applied transportability methods to 2 large, multicenter cardiovascular disease treatment trials: the TOPCAT trial (Treatment of Preserved Cardiac Function Heart Failure With an Aldosterone Antagonist; n=3445) comparing spironolactone to placebo for heart failure (for which site anomalies were suspected) and the ACCORD BP trial (Action to Control Cardiovascular Risk in Diabetes-Blood Pressure; n=4733) comparing intensive-to-standard blood pressure treatment (for which site anomalies were not suspected). The transportability methods give expected results by standardizing from one site to another using data on participant covariates. The difference between the expected and observed results was assessed using calibration tests to identify whether treatment-effect differences between sites could be explained by participant population characteristics. Standard regression methods did not detect heterogeneities in TOPCAT between Russia/Georgia study sites suspected of study protocol violations and sites in the Americas ( P=0.12 for difference in primary cardiovascular outcome; P=0.20 for difference in total mortality). The transportability methods, however, detected the difference between Russia/Georgia sites and sites in the Americas ( P<0.001) and found that measured participant characteristics did not explain the between-site discrepancies. The transport methods found no such discrepancies between sites in ACCORD BP, suggesting participant characteristics explained between-site differences.

CONCLUSIONS

Transportability methods may be superior to standard approaches for detecting anomalies within multicenter randomized trials and assist data monitoring boards to determine whether important treatment-effect heterogeneities can be attributed to participant differences or potentially to site performance differences requiring further investigation.

Trial protocol: a multicentre randomised trial of first-line treatment pathways for newly diagnosed immune thrombocytopenia: standard steroid treatment versus combined steroid and mycophenolate. The FLIGHT trial

INTRODUCTION

Immune thrombocytopenia (ITP) is an autoimmune condition that may cause thrombocytopenia-related bleeding. Current first-line ITP treatment is with high-dose corticosteroids but frequent side effects, heterogeneous responses and high relapse rates are significant problems with only 20% remaining in sustained remission with this approach. Mycophenolate mofetil (MMF) is often used as the next treatment with efficacy in 50%-80% of patients and good tolerability but can take up to 2 months to work.

OBJECTIVE

To test the hypothesis that MMF combined with corticosteroid is a more effective first-line treatment for immune thrombocytopenia (ITP) than current standard of corticosteroid alone.

METHODS AND ANALYSIS

DesignMulticentre, UK-based, open-label, randomised controlled trial.

SETTING

Haematology departments in secondary care.

PARTICIPANTS

We plan to recruit 120 patients >16 years old with a diagnosis of ITP and a platelet count <30x10⁹/L who require first-line treatment. Patients will be followed up for a minimum of 12 months following randomisation.

PRIMARY OUTCOME

Time from randomisation to treatment failure defined as platelets <30x10⁹/L and a need for second-line treatment.

SECONDARY OUTCOMES

Side effects, bleeding events, remission rates, time to relapse, time to next therapy, cumulative corticosteroid dose, rescue therapy, splenectomy, socioeconomic costs, patient-reported outcomes (quality of life, fatigue, impact of bleeding, care costs).

ANALYSIS

The sample size of 120 achieves a 91.5% power to detect a doubling of the median time to treatment failure from 5 to 10 months. This will be expressed as an HR with 95% CI, median time to event if more than 50% have had an event and illustrated with Kaplan-Meier curves. Cost-effectiveness will be based on the first 12 months from diagnosis.

ETHICS AND DISSEMINATION

Ethical approval from NRES Committee South West (IRAS number 225959). EudraCT Number: 2017-001171-23. Results will be submitted for publication in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

NCT03156452.

Development of a standardised set of metrics for monitoring site performance in multicentre randomised trials: a Delphi study

BACKGROUND

Site performance is key to the success of large multicentre randomised trials. A standardised set of clear and accessible summaries of site performance could facilitate the timely identification and resolution of potential problems, minimising their impact. The aim of this study was to identify and agree a core set of key performance metrics for managing multicentre randomised trials.

METHODS

We used a mixed methods approach to identify potential metrics and to achieve consensus about the final set, adapting methods that are recommended by the COMET Initiative for developing core outcome sets in health care. We used performance metrics identified from our systematic search and focus groups to create an online Delphi survey. We invited respondents to score each metric for inclusion in the final core set, over three survey rounds. Metrics scored as critical by ≥70% and unimportant by <15% of respondents were taken forward to a consensus meeting of representatives from key UK-based stakeholders. Participants in the consensus meeting discussed and voted on each metric, using anonymous electronic voting. Metrics with >50% of participants voting for inclusion were retained.

RESULTS

Round 1 of the Delphi survey presented 28 performance metrics, and a further six were added in round 2. Of 294 UK-based stakeholders who registered for the Delphi survey, 211 completed all three rounds. At the consensus meeting, 17 metrics were discussed and voted on: 15 metrics were retained following survey round 3, plus two others that were preferred by consensus meeting participants. Consensus was reached on a final core set of eight performance metrics in three domains: (1) recruitment and retention, (2) data quality and (3) protocol compliance. A simple tool for visual reporting of the metrics is available from the Nottingham Clinical Trials Unit website.

CONCLUSIONS

We have established a core set of metrics for measuring the performance of sites in multicentre randomised trials. These metrics could improve trial conduct by enabling researchers to identify and address problems before trials are adversely affected. Future work could evaluate the effectiveness of using the metrics and reporting tool.

A prospective comparison of cancer clinical trial availability and enrollment among adolescents/young adults treated at an adult cancer hospital or affiliated children's hospital

BACKGROUND

Low cancer clinical trial (CCT) enrollment may contribute to survival disparities affecting adolescents and young adults (AYAs) (ages 15-39 years). The objective of this study was to evaluate whether differences in CCT availability related to treatment site could explain the low CCT enrollment.

METHODS

This prospective, observational cohort study was conducted at an academic children's hospital and its affiliated but geographically separated adult cancer hospital within a National Cancer Institute-designated Comprehensive Cancer Center. For consecutive, newly diagnosed AYA patients, it was determined whether an appropriate CCT existed nationally, was available at the treatment site, and was used for enrollment. Proportions of AYAs in these categories were compared between sites using the chi-square test.

RESULTS

One hundred fifty-two consecutive AYA patients were included from the children's hospital (n = 68; ages 15-20 years) and the adult cancer hospital (n = 84; ages 18-39 years). Although there was no difference in CCT existence for individual AYA patients by site (children's hospital [36 of 68 patients; 52.9%] vs adult cancer hospital [45 of 84 patients; 53.6%]; P = .938), CCT availability was significantly lower at the adult cancer hospital (14 of 84 patients [16.7%] vs 30 of 68 [44.1%] at the children's hospital; P < .001). The proportion of AYAs enrolled was low at both sites (8 of 68 patients [11.8%] vs 6 of 84 patients [7.1%], respectively; P = .327). Fewer existing CCTs were available at the adult cancer hospital (4 of 27 patients [14.8%] vs 8 of 14 patients [57.1%], respectively), and those were directed toward solid tumors and new agents.

CONCLUSIONS

Efforts to improve low CCT enrollment among AYAs should be differentiated by treatment site. In the adult setting, these efforts should be aimed at improving CCT availability by overcoming site-level barriers to opening existing CCTs.

An account of data entry inconsistencies and their impact on positron emission tomography quantification

Reproducibility is essential to clinical application of positron emission tomography (PET) quantification. Human lapses in data registration and protocol compliance are pervasive sources of intrasite quantification variability. Although rarely assessed or reported, these lapses are ultimately a limitation to harmonization in multicenter clinical trials. A comprehensive account of their possible extent is relayed here.This is a retrospective audit of errors in manual registration of study parameters and in protocol adherence across a sample of in-center research projects over one year (201 patients, 222 PET/CT scans). Discrepancies in patient height and weight; tracer type; dose; injection; and scan times were listed. Correspondent variances in standardized uptake values (SUVs) normalized by body weight, SUV (BW), and body surface area, SUV (BSA), were assessed.Manual misregistrations totalled 41.8%. These were mainly small, but with a few large deviations, and most significant in weight (range: -1-100 kg) and dose (-19 to 12 MBq). Errors were more frequent and generally larger in non-routine studies. This also applied to protocol compliance. A 50.7% noncompliance was found with significant deviations in dose (-106 to 208 MBq) and especially in early scan uptake times (-37 to 54 min). Although misregistrations did not overall translate into significant SUV variability, noncompliance did. These errors contributed a factor 0.02 to 1.45 and 0.71 to 3.09 SUV (BW) change, respectively. SUV (BSA) saw a significant 21% to 22% decrease with mistyped height and weight.Inconsistency was frequent but less prominent in data entry than in protocol compliance. As both caused some substantial SUV variances, intra-site assessments and data checking are required for clinical trials.

Longitudinally and inter-site consistent multi-atlas based parcellation of brain anatomy using harmonized atlases

As longitudinal and multi-site studies become increasingly frequent in neuroimaging, maintaining longitudinal and inter-scanner consistency of brain parcellation has become a major challenge due to variation in scanner models and/or image acquisition protocols across scanners and sites. We present a new automated segmentation method specifically designed to achieve a consistent parcellation of anatomical brain structures in such heterogeneous datasets. Our method combines a site-specific atlas creation strategy with a state-of-the-art multi-atlas anatomical label fusion framework. Site-specific atlases are computed such that they preserve image intensity characteristics of each site's scanner and acquisition protocol, while atlas pairs share anatomical labels in a way consistent with inter-scanner acquisition variations. This harmonization of atlases improves inter-study and longitudinal consistency of segmentations in the subsequent consensus labeling step. We tested this approach on a large sample of older adults from the Baltimore Longitudinal Study of Aging (BLSA) who had longitudinal scans acquired using two scanners that vary with respect to vendor and image acquisition protocol. We compared the proposed method to standard multi-atlas segmentation for both cross-sectional and longitudinal analyses. The harmonization significantly reduced scanner-related differences in the age trends of ROI volumes, improved longitudinal consistency of segmentations, and resulted in higher across-scanner intra-class correlations, particularly in the white matter.

Multicentre trials: a US regulatory perspective

Multicentre trials are very common in the field of drug development. In recent years, multicentre trials have taken on a multinational and multiregional aspect. We provide a conceptual framework for the use of multicentre trials in the context of drug development, from the perspective of drug regulation in the United States. In this paper, we review some regulatory history, milestones and standards as they relate to multicentre trials. Special attention is given to the similarities and differences in the approaches to multicentre trials in the following documents; Guideline for the Format and Content of the Clinical and Statistical Sections of New Drug Applications, International Conference on Harmonization, Draft Guideline on Statistical Principles for clinical trials and the Guidance for Industry Providing Clinical Evidence of Effectiveness for Human Drug and Biologic Products. The paper includes a consideration of some of the issues in the analysis of data from multicentre trials.

Implementation of the who Multicentre Growth Reference Study in Brazil

The World Health Organization (WHO) Multicentre Growth Reference Study (MGRS) African site was Accra, Ghana. Its sample was drawn from 10 affluent residential areas where earlier research had demonstrated the presence of a child subpopulation with unconstrained growth. This subpopulation could be identified on the basis of the father's education and household income. The subjects for the longitudinal study were enrolled from 25 hospitals and delivery facilities that accounted for 80% of the study area's births. The cross-sectional sample was recruited at 117 day-care centers used by more than 80% of the targeted subpopulation. Public relations efforts were mounted to promote the study in the community. The large number of facilities involved in the longitudinal and cross-sectional components, the relatively large geographic area covered by the study, and the difficulties of working in a densely populated urban area presented special challenges. Conversely, the high rates of breastfeeding and general support for this practice greatly facilitated the implementation of the MGRS protocol.

Managing Data for a Multicountry Longitudinal Study: Experience from the who Multicentre Growth Reference Study

The World Health Organization (WHO) Multicentre Growth Reference Study (MGRS) South American site was Pelotas, Brazil. The sample for the longitudinal component was drawn from three hospitals that account for approximately 90% of the city's deliveries. The cross-sectional sample was drawn from a community survey based on households that participated in the longitudinal sample. One of the criteria for site selection was the availability of a large, community based sample of children whose growth was unconstrained by socioeconomic conditions. Local work done in 1993 demonstrated that children of families with incomes at least six times the minimum wage had a stunting rate of 2.5%. Special public relations and implementation activities were designed to promote the acceptance of the study by the community and its successful completion. Among the major challenges of the site were serving as the MGRS pilot site, low baseline breastfeeding initiation and maintenance rates, and reluctance among pediatricians to acknowledge the relevance of current infant feeding recommendations to higher socioeconomic groups.

Implementation of the who Multicentre Growth Reference Study in Ghana

The World Health Organization (WHO) Multicentre Growth Reference Study (MGRS) Asian site was New Delhi, India. Its sample was drawn from 58 affluent neighborhoods in South Delhi. This community was selected to facilitate the recruitment of children who had at least one parent with 17 or more years of education, a key factor associated with unconstrained child growth in this setting. A door-to-door survey was conducted to identify pregnant women whose newborns were subsequently screened for eligibility for the longitudinal study, and children aged 18 to 71 months for the cross-sectional component of the study. A total of 111,084 households were visited over an 18-month period. Newborns were screened at birth at 73 sites. The large number of birthing facilities used by this community, the geographically extensive study area, and difficulties in securing support of pediatricians and obstetricians for the feeding recommendations of the study were among the unique challenges faced by the implementation of the MGRS protocol at this site.

Methods for a Multisite Randomized Trial to Investigate the Effect of Constraint-Induced Movement Therapy in Improving Upper Extremity Function among Adults Recovering from a Cerebrovascular Stroke

This article describes the study design, methodological considerations, and demographic characteristics of a phase III RCT to determine if 1) constraint-induced therapy (CI therapy) can be applied with therapeutic success 3 to 9 months after stroke across different sites, 2) gains that might occur persist over 2 years, 3) initial level of motor ability determines responsiveness to CI therapy, and 4) the treatment effect differs between those treated before 9 months and after 1 year. Six sites will screen and recruit poststroke survivors stratified on initial level of motor ability and after randomization allocate participants to immediate or delayed intervention. Primary outcomes include a laboratory-based measure of function (Wolf Motor Function Test [WMFT]) and a real-world participant-centered functional use measure (Motor Activity Log [MAI]). Secondary outcomes concern function, behavior, and compliance. This is the first multisite, single-blind RCT of a formal training intervention for upper extremity rehabilitation in subacute stroke in the United States.

Unique challenges of obtaining regulatory approval for a multicenter protocol to study the genetics of RRP and suggested remedies

Objective

Investigations that seek to generalize findings or to understand uncommon diseases must be conducted at multiple centers. This study describes the process of obtaining regulatory approval for a minimal risk genetic study in a multi-center setting as undertaken by the Recurrent Respiratory Papillomatosis (RRP) Task Force.

Study Design and Setting

Sequential cohort of American children's hospitals. A single protocol was submitted to each Institutional Review Board (IRB).

Results

Documentation was prepared for 14 IRBs over 2.5 years. The median time between enlistment and approval at the first 8 sites was 15 months. Institutions varied considerably in their requirements and in the issues that were raised. Protocols were submitted sequentially and accumulated experience was used in the preparation of applications to subsequent IRBs. Nevertheless, there was no correlation between the accumulated experience and the number of issues that were raised.

Conclusion

Despite uniform federal standards, all local IRBs required unique and individualized submissions. For multicenter studies, investigators should seriously consider the establishment of cooperative authorization agreements. On a simpler level, a standardized format for applications needs to be adopted nationwide. EBM rating: B-3b

Time required to start multicentre clinical trials within the Italian Medicine Agency programme of support for independent research

BACKGROUND AND AIM

Time allowed for independent ethics committees (IECs) and administrative offices to assess and activate clinical trials is regulated by law. This study aims to describe time spent activating two multicentre non-profit trials supported by the Italian Medicines Agency (AIFA). Five non-AIFA supported (NAS) trials were used as a benchmark.

METHODS

The two AIFA-supported trials were FATA-GIM3 (optimal adjuvant hormonal treatment for breast cancer) and TOSCA (duration of adjuvant FOLFOX in colorectal cancer). The five NAS trials focused on lung or ovarian cancer. The following were measured for all trials: date of submission of trial documentation to peripheral IEC, date of IEC opinion and date trial contracts were signed. Times are reported in months.

RESULTS

106 centres applied to participate in FATA-GIM3 and 137 in TOSCA. An IEC opinion was issued by 100/106 (1 negative opinion) and 137/137 (2 negative opinions) centres, with a median time from submission of 3.6 months (range 0.1-60.2). After a positive IEC opinion, the median time before signing the trial contract was 3.3 months (0.1-59.2). Contracts were signed with 93/99 and 135/135 centres, with a median time from submission of study documentation of 8.4 months (0.5-61.1). Times for NAS trials were not substantially different.

CONCLUSIONS

FATA-GIM3 and TOSCA centres were opened after a median of 8 months, consisting of nearly 4 months each for IEC opinion and administrative signature, similar to the NAS trials. The process of trial activation in Italy remains inefficient and takes far longer than legally allowed.

Maximizing Effectiveness Trials in PTSD and SUD Through Secondary Analysis: Benefits and Limitations Using the National Institute on Drug Abuse Clinical Trials Network "Women and Trauma" Study as a Case Example

Recent federal legislation and a renewed focus on integrative care models underscore the need for economical, effective, and science-based behavioral health care treatment. As such, maximizing the impact and reach of treatment research is of great concern. Behavioral health issues, including the frequent co-occurrence of substance use disorders (SUD) and posttraumatic stress disorder (PTSD), are often complex, with a myriad of factors contributing to the success of interventions. Although treatment guides for comorbid SUD/PTSD exist, most patients continue to suffer symptoms following the prescribed treatment course. Further, the study of efficacious treatments has been hampered by methodological challenges (e.g., overreliance on "superiority" designs (i.e., designs structured to test whether or not one treatment statistically surpasses another in terms of effect sizes) and short term interventions). Secondary analyses of randomized controlled clinical trials offer potential benefits to enhance understanding of findings and increase the personalization of treatment. This paper offers a description of the limits of randomized controlled trials as related to SUD/PTSD populations, highlights the benefits and potential pitfalls of secondary analytic techniques, and uses a case example of one of the largest effectiveness trials of behavioral treatment for co-occurring SUD/PTSD conducted within the National Drug Abuse Treatment Clinical Trials Network (NIDA CTN) and producing 19 publications. The paper concludes with implications of this secondary analytic approach to improve addiction researchers' ability to identify best practices for community-based treatment of these disorders. Innovative methods are needed to maximize the benefits of clinical studies and better support SUD/PTSD treatment options for both specialty and non-specialty healthcare settings. Moving forward, planning for and description of secondary analyses in randomized trials should be given equal consideration and care to the primary outcome analysis.

Progress in centralised ethics review processes: Implications for multi-site health evaluations

Increasingly, public sector programmes respond to complex social problems that intersect specific fields and individual disciplines. Such responses result in multi-site initiatives that can span nations, jurisdictions, sectors and organisations. The rigorous evaluation of public sector programmes is now a baseline expectation. For evaluations of large and complex multi-site programme initiatives, the processes of ethics review can present a significant challenge. However in recent years, there have been new developments in centralised ethics review processes in many nations. This paper provides the case study of an evaluation of a national, inter-jurisdictional, cross-sector, aged care health initiative and its encounters with Australian centralised ethics review processes. Specifically, the paper considers progress against the key themes of a previous five-year, five nation study (Fitzgerald and Phillips, 2006), which found that centralised ethics review processes would save time, money and effort, as well as contribute to more equitable workloads for researchers and evaluators. The paper concludes with insights for those charged with refining centralised ethics review processes, as well as recommendations for future evaluators of complex multi-site programme initiatives.

A new approach to standardize multicenter studies: mobile lab technology for the German Environmental Specimen Bank

Technical progress has simplified tasks in lab diagnosis and improved quality of test results. Errors occurring during the pre-analytical phase have more negative impact on the quality of test results than errors encountered during the total analytical process. Different infrastructures of sampling sites can highly influence the quality of samples and therewith of analytical results. Annually the German Environmental Specimen Bank (ESB) collects, characterizes, and stores blood, plasma, and urine samples of 120–150 volunteers each on four different sampling sites in Germany. Overarching goal is to investigate the exposure to environmental pollutants of non-occupational exposed young adults combining human biomonitoring with questionnaire data. We investigated the requirements of the study and the possibility to realize a highly standardized sampling procedure on a mobile platform in order to increase the required quality of the pre-analytical phase. The results lead to the development of a mobile epidemiologic laboratory (epiLab) in the project “Labor der Zukunft” (future’s lab technology). This laboratory includes a 14.7 m² reception area to record medical history and exposure-relevant behavior, a 21.1 m² examination room to record dental fillings and for blood withdrawal, a 15.5 m² biological safety level 2 laboratory to process and analyze samples on site including a 2.8 m² personnel lock and a 3.6 m² cryofacility to immediately freeze samples. Frozen samples can be transferred to their final destination within the vehicle without breaking the cold chain. To our knowledge, we herewith describe for the first time the implementation of a biological safety laboratory (BSL) 2 lab and an epidemiologic unit on a single mobile platform. Since 2013 we have been collecting up to 15.000 individual human samples annually under highly standardized conditions using the mobile laboratory. Characterized and free of alterations they are kept ready for retrospective analyses in their final archive, the German ESB.

Harmonizing international trials of early goal-directed resuscitation for severe sepsis and septic shock: methodology of ProCESS, ARISE, and ProMISe

PURPOSE

To describe and compare the design of three independent but collaborating multicenter trials of early goal-directed resuscitation for severe sepsis and septic shock.

METHODS

We reviewed the three current trials, one each in the USA (ProCESS: protocolized care for early septic shock), Australasia (ARISE: Australasian resuscitation in sepsis evaluation), and the UK (ProMISe: protocolised management in sepsis). We used the 2010 CONSORT (consolidated standards of reporting trials) statement and the 2008 CONSORT extension for trials assessing non-pharmacologic treatments to describe and compare the underlying rationale, commonalities, and differences.

RESULTS

All three trials conform to CONSORT guidelines, address the same fundamental questions, and share key design elements. Each trial is a patient-level, equal-randomized, parallel-group superiority trial that seeks to enroll emergency department patients with inclusion criteria that are consistent with the original early goal-directed therapy (EGDT) trial (suspected or confirmed infection, two or more systemic inflammatory response syndrome criteria, and refractory hypotension or elevated lactate), is powered to detect a 6–8 % absolute mortality reduction (hospital or 90-day), and uses trained teams to deliver EGDT. Design differences appear to primarily be driven by between-country variation in health care context. The main difference between the trials is the inclusion of a third, alternative resuscitation strategy arm in ProCESS.

CONCLUSIONS

Harmonization of study design and methods between severe sepsis trials is feasible and may facilitate pooling of data on completion of the trials.

Data quality assessment for comparative effectiveness research in distributed data networks

BACKGROUND

Electronic health information routinely collected during health care delivery and reimbursement can help address the need for evidence about the real-world effectiveness, safety, and quality of medical care. Often, distributed networks that combine information from multiple sources are needed to generate this real-world evidence.

OBJECTIVE

We provide a set of field-tested best practices and a set of recommendations for data quality checking for comparative effectiveness research (CER) in distributed data networks.

METHODS

Explore the requirements for data quality checking and describe data quality approaches undertaken by several existing multi-site networks.

RESULTS

There are no established standards regarding how to evaluate the quality of electronic health data for CER within distributed networks. Data checks of increasing complexity are often used, ranging from consistency with syntactic rules to evaluation of semantics and consistency within and across sites. Temporal trends within and across sites are widely used, as are checks of each data refresh or update. Rates of specific events and exposures by age group, sex, and month are also common.

DISCUSSION

Secondary use of electronic health data for CER holds promise but is complex, especially in distributed data networks that incorporate periodic data refreshes. The viability of a learning health system is dependent on a robust understanding of the quality, validity, and optimal secondary uses of routinely collected electronic health data within distributed health data networks. Robust data quality checking can strengthen confidence in findings based on distributed data network.

HLA-DR expression as a biomarker of inflammation for multicenter clinical trials of ocular surface disease

There are currently no validated minimally invasive objective metrics for the classification and evaluation of ocular surface diseases and/or for evaluating treatment efficacy. We thus sought to establish a standardized methodology for determining the relative amount of the inflammatory biomarker HLA-DR on the ocular surface and to evaluate the precision, reliability and repeatability of its use for large multicenter clinical trials and translational research studies of ocular surface disease. Multiple studies were conducted to establish a Standard Operating Procedure (SOP) for utilizing HLA-DR expression as a minimally invasive, objective, ocular surface inflammatory biomarker. The established SOPs provide specific guidelines for HLA-DR collection and analysis, in order to incorporate it reliably into multicenter clinical trials and/or translational research. Duplicate cell samples from impression cytology (IC) samples of both normal and dry eye individuals were collected and split to assess repeatability (between the splits and between the duplicate samples). To determine storage capability, one duplicate was stained immediately and the other after 30 days cold storage. To demonstrate the feasibility of the use of the SOP for a multicenter clinical trial, clinicians out-of-state were trained to collect IC samples, and the samples shipped to our Biomarker Laboratory, logged, processed and analyzed. Demonstration of the ability to incorporate of IC into a randomized double masked clinical trial of dry eye disease (DED) was performed. In all cases, processing and analyses were performed by a masked independent observer. The validity/viability of the SOPs was established by demonstrating that: 1) sufficient numbers of cells can be collected via IC; 2) the precision/repeatability of the relative biomarker expression quantified in samples; 3) personnel at distant sites can be taught to collect, store and ship samples successfully; 4) samples can be stored for up to 30 days (refrigeration) before processing without affecting results; 5) IC can be incorporated into a double blind randomized clinical trial (RCT) of DED; and 6) the Biomarker Laboratory can track a large number of masked samples reliably. In conclusion, our standard operating procedure for impression cytology analysis of HLA-DR expression appears to be repeatable and reproducible for use in multicenter clinical trials, providing a minimally invasive objective biomarker of inflammation of the ocular surface.

Harmonization of the practice of independent ethics committees in Italy: project E-submission

Aim

The high variability of “centre-specific” documentation required by Independent Ethics Committee (IEC) plays a role in the time required for activation of participating centres of multicentre clinical trials. This study (a) provides a picture of the different activities, structural requirements and resources dedicated to the operation of the local IEC in Italy; (b) defines a detailed list of “centre-specific” documents considered as essential by the IEC for issuing its opinion and (c) suggests a “single document” to reduce the variability of the “centre-specific” documents required by the IEC.

Methodology

Two surveys were conducted through the portal of National Monitoring Centre of Clinical Trials (https://oss-sper-clin.agenziafarmaco.it/). The first survey focused on the local IEC resources and on the “centre-specific” documentation that local IEC required from the Sponsor and local Principal Investigator (PI). The second focused on “single document” required in the form of statements from the Sponsor and the PI. Answers were discussed and extended during regular scheduled teleconferences and plenary meeting.

Principal Findings

From 22/07/2009 to 15/12/2009, and from 19/04/2010 to 14/05/2010, 131 and 125 IECs responded to the first and the second surveys, respectively. 67% and 51% of IECs consider the structural requirements and the staff dedicated to the activity of the IECs as sufficient, respectively. Most of the IECs consider the “centre-specific” documentation as necessary for issuing the opinion, and a high percentage of IECs consider the proposed documentation as acceptable in substitution to any other “centre-specific” documentation already in use.

Conclusions

The harmonization of IECs practice in Italy is the first step to facilitate multicentre clinical trials. Similar efforts should be directed to reduce the total number of IECs and to standardize clinical trials approval procedures, focusing on administrative procedures as well.

Redesigning radiotherapy quality assurance: opportunities to develop an efficient, evidence-based system to support clinical trials--report of the National Cancer Institute Work Group on Radiotherapy Quality Assurance

PURPOSE

In the context of national calls for reorganizing cancer clinical trials, the National Cancer Institute sponsored a 2-day workshop to examine challenges and opportunities for optimizing radiotherapy quality assurance (QA) in clinical trial design.

METHODS AND MATERIALS

Participants reviewed the current processes of clinical trial QA and noted the QA challenges presented by advanced technologies. The lessons learned from the radiotherapy QA programs of recent trials were discussed in detail. Four potential opportunities for optimizing radiotherapy QA were explored, including the use of normal tissue toxicity and tumor control metrics, biomarkers of radiation toxicity, new radiotherapy modalities such as proton beam therapy, and the international harmonization of clinical trial QA.

RESULTS

Four recommendations were made: (1) to develop a tiered (and more efficient) system for radiotherapy QA and tailor the intensity of QA to the clinical trial objectives (tiers include general credentialing, trial-specific credentialing, and individual case review); (2) to establish a case QA repository; (3) to develop an evidence base for clinical trial QA and introduce innovative prospective trial designs to evaluate radiotherapy QA in clinical trials; and (4) to explore the feasibility of consolidating clinical trial QA in the United States.

CONCLUSION

Radiotherapy QA can affect clinical trial accrual, cost, outcomes, and generalizability. To achieve maximum benefit, QA programs must become more efficient and evidence-based.

Site selection in community-based clinical trials for substance use disorders: strategies for effective site selection

BACKGROUND

The importance of conducting substance use disorder treatment research in real-world settings is now well recognized. While this approach to clinical trials research offers a variety of benefits, challenges also arise. Selecting high-quality sites to participate is critical to recruitment, retention, and overall trial performance when conducting multi-site, community-based clinical trials of treatments for substance use disorders.

OBJECTIVES

Over the past 10 years, the National Institute on Drug Abuse-sponsored National Drug Abuse Treatment Clinical Trials Network (CTN) has strived to conduct high-quality, well-managed clinical trials. This includes developing methods for site selection to be used by investigators conducting CTN trials.

METHODS

We review site selection strategies from two community-based multi-site clinical trials conducted under the auspices of the National Drug Abuse Treatment Clinical Trials Network.

RESULTS

Issues relevant to site selection include the clinical trial design, availability of appropriate clinical population, and organizational attributes of potential clinical research sites. Site selection strategies include reviewing regional epidemiologic data, collecting standard site selection surveys, evaluating clinic data on existing patient populations, and site selection interviews and visits.

CONCLUSION

This article describes considerations for selecting research sites and identifies specific strategies to employ when selecting community-based sites for participation in clinical trials.

Prospective registration and results disclosure of clinical trials in the Americas: a roadmap toward transparency

The objective of this article is to propose a roadmap toward transparency of clinical trials in the Americas by their prospective registration and results disclosure. This will broaden access to more complete and accurate data and facilitate evidence-informed decision-making and participation in research. Consequently, it should have a positive impact on people's health and should promote trust in health research. Existing initiatives were identified, registration of trials was analyzed following the World Health Organization (WHO) standards on trial registration, and a roadmap is proposed to address the gaps in advancing transparency. The analysis shows that, in spite of numerous regional and country initiatives, clinical trials taking place in nonEnglish-speaking parts of the Americas are underregistered. A roadmap is proposed to enhance research governance and good research practice by improving the transparency of clinical trials. The proposed roadmap includes strategies for implementing WHO international standards for trial registration, for developing international standards of public disclosure of trial results, and for a potential role of the Pan American Health Organization.

The FDA outlook of events reporting after ticagrelor or clopidogrel in the PLATO Trial: impact of sponsor censoring dates, drug discontinuation, and withdrawal of consent

BACKGROUND

Censoring by the study sponsor of clinical endpoint events in indication-seeking randomized trials represents a controversial approach since the reported data may be biased in favor of experimental agents due to the obvious conflict of interest. The frequency of drug discontinuation and rates of consent withdrawal may also impact the trial outcomes.

PURPOSE

To assess patterns of event reporting dependent on sponsor censoring dates, drug discontinuation and consent withdrawal in the PLATO trial.

METHODS

Analysis of the Food and Drug Administration Complete Response Review for ticagrelor.

RESULTS

Excluding adjudicated deaths, the distribution for clopidogrel appears more uniform while that for ticagrelor was skewed to the right, suggesting more events were reported after the sponsor censoring end date. PLATO investigators reported 16 unmatched primary endpoint events for ticagrelor immediately following the sponsor censoring date. Twenty-six out of 30 unreported events following early drug discontinuation occurred amongst patients using ticagrelor. More ticagrelor patients withdrew consent (Δ = 47), or were 'not willing' to complete the study (Δ = 87) when compared to clopidogrel.

CONCLUSIONS

Site-reported primary endpoints were unequally distributed for clopidogrel and ticagrelor in the PLATO trial. This pattern suggests the importance of questioning the impact of sponsor-mediated censoring on event reporting by investigators in indication-seeking trials. In PLATO, this pattern seems to have favored the experimental drug and may require further assessment.