Quality control in multicentric clinical trials. An experience of the EORTC Gynecological Cancer Cooperative Group

Clinical trial monitoring effectiveness: Remote risk-based monitoring versus on-site monitoring with 100% source data verification

BACKGROUND/AIMS

Traditional on-site monitoring of clinical trials via frequent site visits and 100% source data verification is cost-consuming, and it still cannot guarantee data quality effectively. Depending on the types and designs of clinical trials, an alternative would be combining several monitoring methods, such as risk-based monitoring and remote monitoring. However, there is insufficient evidence of its effectiveness. This research compared the effectiveness of risk-based monitoring with a remote monitoring system with that of traditional on-site monitoring.

METHODS

With a cloud-based remote monitoring system called beagle View^®, we created a remote risk-based monitoring methodology that focused only on critical data and processes. We selected a randomized controlled trial conducted at Tohoku University Hospital and randomly sampled 11 subjects whose case report forms had already been reviewed by data managers. Critical data and processes were verified retrospectively by remote risk-based monitoring; later, all data and processes were confirmed by on-site monitoring. We compared the ability of remote risk-based monitoring to detect critical data and process errors with that of on-site monitoring with 100% source data verification, including an examination of clinical trial staff workload and potential cost savings.

RESULTS

Of the total data points (n = 5617), 19.7% (n = 1105, 95% confidence interval = 18.7-20.7) were identified as critical. The error rates of critical data detected by on-site monitoring, remote risk-based monitoring, and data review by data managers were 7.6% (n = 84, 95% CI = 6.2-9.3), 7.6% (n = 84, 95% confidence interval = 6.2-9.3), and 3.9% (n = 43, 95% confidence interval = 2.9-5.2), respectively. The total number of critical process errors detected by on-site monitoring was 14. Of these 14, 92.9% (n = 13, 95% confidence interval = 68.5-98.7) and 42.9% (n = 6, 95% confidence interval = 21.4-67.4) of critical process errors were detected by remote risk-based monitoring and data review by data managers, respectively. The mean time clinical trial staff spent dealing with remote risk-based monitoring was 9.9 ± 5.3 (mean ± SD) min per visit per subject. Our calculations show that remote risk-based monitoring saved between 9 and 41 on-site monitoring visits, corresponding to a cost of between US$13,500 and US$61,500 per trial site.

CONCLUSION

Remote risk-based monitoring was able to detect critical data and process errors as well as on-site monitoring with 100% source data verification, saving travel time and monitoring costs. Remote risk-based monitoring offers an effective alternative to traditional on-site monitoring of clinical trials.

Impact of a targeted monitoring on data-quality and data-management workload of randomized controlled trials: A prospective comparative study

AIMS

Monitoring risk-based approaches in clinical trials are encouraged by regulatory guidance. However, the impact of a targeted source data verification (SDV) on data-management (DM) workload and on final data quality needs to be addressed.

METHODS

MONITORING was a prospective study aiming at comparing full SDV (100% of data verified for all patients) and targeted SDV (only key data verified for all patients) followed by the same DM program (detecting missing data and checking consistency) on final data quality, global workload and staffing costs.

RESULTS

In all, 137 008 data including 18 124 key data were collected for 126 patients from 6 clinical trials. Compared to the final database obtained using the full SDV monitoring process, the final database obtained using the targeted SDV monitoring process had a residual error rate of 1.47% (95% confidence interval, 1.41-1.53%) on overall data and 0.78% (95% confidence interval, 0.65-0.91%) on key data. There were nearly 4 times more queries per study with targeted SDV than with full SDV (mean ± standard deviation: 132 ± 101 vs 34 ± 26; P = .03). For a handling time of 15 minutes per query, the global workload of the targeted SDV monitoring strategy remained below that of the full SDV monitoring strategy. From 25 minutes per query it was above, increasing progressively to represent a 50% increase for 45 minutes per query.

CONCLUSION

Targeted SDV monitoring is accompanied by increased workload for DM, which allows to obtain a small proportion of remaining errors on key data (<1%), but may substantially increase trial costs.

Multidisciplinary quality assurance and control in oncological trials: Perspectives from European Organisation for Research and Treatment of Cancer (EORTC)

Quality assurance (QA) programmes are one of the mainstays of clinical research and constitute the pillars on which European Organisation for Research Treatment of Cancer (EORTC) delivers multidisciplinary therapeutic progress. Changing practice treatments require solid evidence-based data, which can only be achieved if integral QA is part of the infrastructure sustaining research projects. Cancer treatment is a multimodality approach, which is often applied either in sequence and/or in combination. Each modality plays a key role in cancer control. The modalities by which QA is applied varies substantially within and across the disciplines. In addition, translational and diagnostic disciplines take an increasing role in the era of precision medicine. Building on the structuring effect of clinical research with fully integrated multidisciplinary QA programmes associated with the solutions addressing the chain of custody for biological material and data integrity as well as compliance ensure at the same time validity of clinical research output but also have a training effect on health care providers, who are more likely to apply such principles as routine. The principles of QA are therefore critical to be embedded in multidisciplinary infrastructure to guarantee therapeutic progress. These principles also provide the basis for the functioning of multidisciplinary tumour board. However, technical, operational and economic challenges which go with the implementation of such programmes require optimal know-how and the coordination of the multiple expertise and such efforts are best achieved through centralised infrastructure.

Impact of on-site initiation visits on patient recruitment and data quality in a randomized trial of adjuvant chemotherapy for breast cancer

Purpose To provide empirical evidence on the impact of on-site initiation visits on the following outcomes: patient recruitment, quantity and quality of data submitted to the trial coordinating office, and patients' follow-up time.

Patients and methods This methodological study was performed as part of a randomized trial comparing two combination chemotherapies for adjuvant treatment of breast cancer. Centers participating to the trial were randomized to either receive systematic on-site visits (Visited group), or not (Non-visited group).

Results The study was terminated after two years, while the main randomized trial continued. Of the 135 centers that had expressed an interest in the trial, only 69 randomized at least one patient (35/68 in the Visited group, 34/67 in the Nonvisited group). Almost two-thirds of the patients were entered by 17 centers (10 in the Visited group, seven in the Non-visited group) that accrued more than 10 patients each. None of the prespecified outcomes favored the group of centers submitted to on-site initiation visits (ie, mean number of queries par patient: 6.1 ± 9.7 versus 5.4 ± 6.4, respectively for the Visited and Non-visited groups). Spontaneous transmittal of case report forms, although required by protocol, was low in both randomized groups (mean number of pages per patient: 1.5 ± 2.0 versus 2.1 ± 2.3, respectively), with investigators submitting about one-third of the expected forms on time (29% and 39%, respectively).

Limitations This study could not evaluate the impact of repeated on-site visits on clinical outcomes.

Conclusion Systematic on-site initiation visits did not contribute significantly to this clinical trial.

The Essentials of Locoregional Control in the Treatment of Gastric Cancer

Gastric cancer is the fourth most frequent cancer in the world. For curative treatment and local control of gastric cancer, surgery is essential. The extent of the lymph node dissection is still under debate. Only one available trial showed significantly increased overall survival, whereas in all other randomised trials no significant difference could be found.

As surgery alone often is not sufficient in the curative treatment in gastric cancer, different (neo)adjuvant treatment strategies have extensively been studied. The recently published MAGIC trial showed downstaging, downsizing and an improved overall survival for patients treated with perioperative chemotherapy, compared to surgery alone (difference 13%, p = 0.009). The INT 0116 trial on the other hand, demonstrated the benefit of postoperative chemoradiotherapy compared to surgery alone for patients with a curative resection of gastric cancer. However, the quality of resections in this trial was poor, illustrating the importance of standardisation by quality control. This could be done by the Maruyama index, which quantifies the likelihood of unresected disease.

In the Netherlands, the CRITICS trial has recently been launched, which will be a quality controlled trial comparing postoperative chemoradiotherapy and chemotherapy on survival and/or locoregional control in patients who receive neoadjuvant chemotherapy followed by a D1+ gastric resection.

The value of source data verification in a cancer clinical trial

BACKGROUND

Source data verification (SDV) is a resource intensive method of quality assurance frequently used in clinical trials. There is no empirical evidence to suggest that SDV would impact on comparative treatment effect results from a clinical trial.

METHODS

Data discrepancies and comparative treatment effects obtained following 100% SDV were compared to those based on data without SDV. Overall survival (OS) and Progression-free survival (PFS) were compared using Kaplan-Meier curves, log-rank tests and Cox models. Tumour response classifications and comparative treatment Odds Ratios (ORs) for the outcome objective response rate, and number of Serious Adverse Events (SAEs) were compared. OS estimates based on SDV data were compared against estimates obtained from centrally monitored data.

FINDINGS

Data discrepancies were identified between different monitoring procedures for the majority of variables examined, with some variation in discrepancy rates. There were no systematic patterns to discrepancies and their impact was negligible on OS, the primary outcome of the trial (HR (95% CI): 1.18(0.99 to 1.41), p = 0.064 with 100% SDV; 1.18(0.99 to 1.42), p = 0.068 without SDV; 1.18(0.99 to 1.40), p = 0.073 with central monitoring). Results were similar for PFS. More extreme discrepancies were found for the subjective outcome overall objective response (OR (95% CI): 1.67(1.04 to 2.68), p = 0.03 with 100% SDV; 2.45(1.49 to 4.04), p = 0.0003 without any SDV) which was mostly due to differing CT scans.

INTERPRETATION

Quality assurance methods used in clinical trials should be informed by empirical evidence. In this empirical comparison, SDV was expensive and identified random errors that made little impact on results and clinical conclusions of the trial. Central monitoring using an external data source was a more efficient approach for the primary outcome of OS. For the subjective outcome objective response, an independent blinded review committee and tracking system to monitor missing scan data could be more efficient than SDV.

Assessment of data quality in an international multi-centre randomised trial of coronary artery surgery

Background

ART is a multi-centre randomised trial of cardiac surgery which provided a unique opportunity to evaluate the data from a large number of centres from a variety of countries. We attempted to assess data quality, including recruitment rates, timeliness and completeness of the data obtained from the centres in different socio-economic strata.

Methods

The analysis was based on the 2-page CRF completed at the 6 week follow-up. CRF pages were categorised into "clean" (no edit query) and "dirty" (any incomplete, inconsistent or illegible data). The timelines were assessed on the basis of the time interval from the visit and receipt of complete CRF. Data quality was defined as the number of data queries (in percent) and time delay (in days) between visit and receipt of correct data. Analyses were stratified according to the World Bank definitions into: "Developing" countries (Poland, Brazil and India) and "Developed" (Italy, UK, Austria and Australia).

Results

There were 18 centres in the "Developed" and 10 centres in the "Developing" countries. The rate of enrolment did not differ significantly by economic level ("Developing":4.1 persons/month, "Developed":3.7 persons/month). The time interval for the receipt of data was longer for "Developing" countries (median:37 days) compared to "Developed" ones (median:11 days) (p < 0.001). The median number of data queries was 23% in "Developed" countries compared to 19% in "Developing" ones (p = ns).

Conclusions

In this study we showed that data quality was comparable between centres from "Developed" and "Developing" countries. Data was received in a less timely fashion from Developing countries and appropriate systems should be instigated to minimize any delays. Close attention should be paid to the training of centres and to the central management of data quality.

Trial registration

ISRCTN46552265

A Peer Review Intervention for Monitoring and Evaluating sites (PRIME) that improved randomized controlled trial conduct and performance

OBJECTIVE

Good clinical practice (GCP) guidelines emphasize trial site monitoring, although the implementation is unspecified and evidence for benefit is sparse. We aimed to develop a site monitoring process using peer reviewers to improve staff training, site performance, data collection, and GCP compliance.

STUDY DESIGN AND SETTING

The Peer Review Intervention for Monitoring and Evaluating sites (PRIME) team observed and gave feedback on trial recruitment and follow-up appointments, held staff meetings, and examined documentation during annual 2-day site visits. The intervention was evaluated in the ProtecT trial, a UK randomized controlled trial of localized prostate cancer treatments (ISRCTN20141297). The ProtecT coordinator and senior nurses conducted three monitoring rounds at eight sites (2004-2007). The process evaluation used PRIME report findings, trial databases, resource use, and a site nurse survey.

RESULTS

Adverse findings decreased across all sites from 44 in round 1 to 19 in round 3. Most findings related to protocol adherence or site organizational issues, including improvements in eligibility criteria application and data collection. Staff found site monitoring acceptable and made changes after reviews.

CONCLUSION

The PRIME process used observation by peer reviewers to improve protocol adherence and train site staff, which increased trial performance and consistency.

Measuring the Quality of Data Collection in a Large Observational Cohort of HIV and AIDS

The aim of this study was to examine the quality of data collection by studying the validity of collected data. Data were extracted from the clinic charts of two anonymous outpatients by 38 data collectors. A standard for the data to be collected was determined (168 items). The validity was measured by comparing the collected items with the standard; in this way, the percentages of the collected items that were ‘correct’ could be calculated. The percentage ‘correct’ was higher for clinic chart 1 (mean: 83% correct, SD 7%) than for clinic chart 2 (mean: 78% correct, SD 8%). All categories contained incorrectly collected data. These data were divided into missing data, incorrect start-stop dates, and surplus collected data. Almost all start-stop dates would change into ‘correct’ if ‘monthyear’ was considered correct (instead of the standard ‘daymonthyear’). Not all data collectors used specific protocols, and sources other than the written comments were not always checked. This study shows that a high proportion of data was correctly collected. However, the collection of start-stop dates was not optimal, and the collected data included surplus and missing data. Data collectors should be more knowledgeable about HIV disease and trained in the use of difficult protocols, so that they can better recognize what data to collect and how it should be collected. Among physicians, there should be more agreement about what information to record in the charts, to facilitate data extraction for data collectors.

Quantifying data quality for clinical trials using electronic data capture

Background

Historically, only partial assessments of data quality have been performed in clinical trials, for which the most common method of measuring database error rates has been to compare the case report form (CRF) to database entries and count discrepancies. Importantly, errors arising from medical record abstraction and transcription are rarely evaluated as part of such quality assessments. Electronic Data Capture (EDC) technology has had a further impact, as paper CRFs typically leveraged for quality measurement are not used in EDC processes.

Methods and Principal Findings

The National Institute on Drug Abuse Treatment Clinical Trials Network has developed, implemented, and evaluated methodology for holistically assessing data quality on EDC trials. We characterize the average source-to-database error rate (14.3 errors per 10,000 fields) for the first year of use of the new evaluation method. This error rate was significantly lower than the average of published error rates for source-to-database audits, and was similar to CRF-to-database error rates reported in the published literature. We attribute this largely to an absence of medical record abstraction on the trials we examined, and to an outpatient setting characterized by less acute patient conditions.

Conclusions

Historically, medical record abstraction is the most significant source of error by an order of magnitude, and should be measured and managed during the course of clinical trials. Source-to-database error rates are highly dependent on the amount of structured data collection in the clinical setting and on the complexity of the medical record, dependencies that should be considered when developing data quality benchmarks.

Ensuring high accuracy of data abstracted from patient charts: the use of a standardized medical record as a training tool

BACKGROUND AND OBJECTIVE

The quality and integrity of information is pivotal to the validity and reliability of inferences drawn in research. The aim of this study is to demonstrate that standardized medical records can be used as a data abstraction training tool and a quality control measure to assess the validity of medical record data abstraction.

METHODS

Sixteen hospitals participating in a large multicenter study completed standardized data abstraction forms for three representative patient charts, one in each of the clinical areas of postoperative critical care and trauma, cardiac surgery, and repair of hip fracture. The completed forms were then compared to an established gold standard.

RESULTS

The mean level of accuracy of the completed data abstraction forms in each of the above three clinical areas were 91.8, 77.5, and 91.5%, respectively. Missing data accounted for 19% of all discrepancies between the abstracted information and the gold standard. If queries and amendments were made by the study's coordinating center, the mean level of accuracy increased to 94.5, 82.5, and 92.9%, respectively.

CONCLUSION

The present study stressed the need for quality control measures in abstracting information from medical records to ensure the accuracy and completeness of the data abstracted.

The quality of chemotherapy and its quality assurance

AIMS

Assessment of the quality of chemotherapy care and its quality assurance in clinical trials and daily practice.

METHODS

Using Medline, literature was searched combining the following words: quality assurance or quality of care, combined with anti-neoplastic agents. The bibliography of each article was reviewed for additional literature. Those reports in English, French, German or Dutch focusing quality assurance or quality of care and chemotherapy were selected.

RESULTS

One hundred and five articles were selected by Medline and after review and adding of additional literature 53 articles remained. In clinical trials information on quality of chemotherapy is sparse. Different cooperative groups reported on suboptimal dosing, suboptimal registration of chemotherapy and several trials indicated that suboptimal dosing led to impaired outcome. Most quality assurance activities in clinical trials are concerned with audit and feedback and on-site visits. In daily practice the quality of chemotherapy is mostly impaired by the fact that it is not given although indicated and if it is given non-evidence based chemotherapy or administration schedules and reduced dose intensity decrease the quality of care. Especially, age, comorbidity and socio-economic status reduce the chance of receiving good quality of care regarding chemotherapy. Activities mostly used for quality assurance are generation of guidelines, specialisation and multidisciplinary care.

CONCLUSIONS

Most quality assurance activities in clinical trials and daily practice are directed to structure and process parameters. More evidence that quality of care is related to outcome should be sought. Quality assurance in daily practice should aim at guideline implementation, specialisation and multidisciplinary care and should pay attention especially to the older patients, patients with comorbidity and patients from lower socio-economic classes.

Population characteristics in cervical cancer trials: search for external validity

OBJECTIVE

The purpose of this study was to compare the characteristics of patients with cervical cancer who were enrolled in cooperative group trials with characteristics of the cervical cancer population of the United States and to determine the generalizability of the results of those trials to the disease population in the United States.

STUDY DESIGN

Phase III trials in cervical cancer that were conducted by cooperative groups in the United States from 1981 through 1997 were identified. These groups were placed into 4 categories on the basis of disease stage and primary treatment modality: Stage IB, negative pelvic nodes that were treated with radical hysterectomy (n = 277 patients); Stage IB-IIA, positive pelvic nodes that were treated with radical hysterectomy (n = 239 patients); Stage IB 2 , negative pelvic and para-aortic nodes that were treated with radiation therapy (n = 369 patients); and Stage IIB-IVA, negative para-aortic nodes that were treated with radiation therapy (n = 1190 patients). For each category, comparable patients from the Surveillance, Epidemiology, and End Results (SEER) database were identified. The age and ethnic distributions of each study population and the distributions of the SEER program were compared.

RESULTS

The age distributions were equivalent, except for patients with IB 2 disease that was treated with radiation therapy where cooperative group subjects were more likely to be younger than 50 years, (odds ratio, 0.17; 95% CI, 0.11-0.26). A statistically significant higher proportion of black and Hispanic women enrolled in cooperative group studies in comparison with surveillance, epidemiology, and end results.

CONCLUSION

Hispanic and black women were recruited proportionately to cooperative group randomized cervical cancer trials in comparison to the United States population. The age distribution of the clinical trial population is also comparable to that of the general population.

Quality assurance of surgery in gastric and rectal cancer

Multimodality and quality controlled treatment result in improved treatment outcome in patients with solid tumours. Quality assurance focuses on identifying and reducing variations in treatment strategy. Treatment outcome is subsequently improved through the introduction of programs that reduce treatment variations to an acceptable level and implement standardised treatment. In chemotherapy and radiotherapy, such programmes have been introduced successfully. In surgery however, there has been little attention for quality assurance so far. Surgery is the mainstay in the treatment of patients with gastric and rectal cancer. In gastric cancer, the extent of surgery is continuously being debated. In Japan, extended lymph node dissection is favoured whereas in the West this type of surgery is not routinely performed with two large European trials concluding that there is no survival benefit from regional lymph node clearance. Post-operative chemoradiation is part of the standard treatment in the United States, although its role in combination with adequate surgery has not been established yet. These global differences in treatment policy clearly relate to the extent and quality of surgical treatment. As for gastric cancer, surgical treatment of rectal cancer patients determines patient's prognosis to a large extent. With the introduction of total mesorectal excision, local control and survival have improved substantially. Most rectal cancer patients receive adjuvant treatment, either pre- or post-operatively. The efficacy of many adjuvant treatment regimens has been investigated in combination with conventional suboptimal surgery. Traditional indications of adjuvant treatment might have to be re-examined, considering the substantial changes in surgical practise. Quality assurance programs enable the introduction of standardised and quality controlled surgery. Promising adjuvant regimens should be investigated in combination with optimal surgery.

Quality assurance in clinical trials

From the literature that was initially searched by electronic databases using the keywords quality, quality control and quality assurance in combination with clinical trials, surgery, pathology, radiotherapy, chemotherapy and data management, a comprehensive review is given on what quality assurance means, the various methods used for quality assurance in different aspects of clinical trials and the impact of this quality assurance on outcome and every day practice.

The importance of quality assurance in surgical oncology

AIMS

The aims were to review the existing methods of quality assurance in surgical oncology and to determine a relationship between surgery-related factors and the variety in outcomes in the treatment of solid cancers.

METHODS

The literature was reviewed by searching Medline and Cancerlit databases.

RESULTS

Wide variations were found in virtually all tumour types. Clear evidence was found that an improvement in the quality of the surgical procedure could have major implications for the prognosis and quality of life of cancer patients.

CONCLUSIONS

These findings emphasize the need for strict quality control procedures in surgical oncology and might imply a considerable change in cancer treatment strategies, because the routine use of adjuvant therapies could be questioned.

Quality assurance in surgical oncology (QASO) within the European Organization for Research and Treatment of Cancer (EORTC): current status and future prospects

The European Organization for Research and Treatment of Cancer (EORTC) has a long history in the development of quality assurance, in particular in radio- and chemotherapy. Quality assurance in surgical oncology is considered to be more complicated, because it is a multistep procedure depending on the individual. Because of the growing importance of the quality of surgical intervention in the multi-modality treatment approach of most cancers, the EORTC recently decided to investigate the current status of quality assurance programmes, both outside and within, the EORTC. The review of EORTC involvement in this area has been conducted on the basis of interviews with subcommittee chairmen and Data Center teams of the EORTC clinical research groups. In addition, clinical trial protocols, case report forms (CRFs) and publications by the EORTC groups related to this field were considered as possible sources of information. Several methods have been used or are currently under investigation to ensure the quality of surgery within clinical trials. These include review of reported data, standardisation of surgery and pathology forms, training sessions and site visits. However, there has been no attempt to harmonise these initiatives across the different medical specialties. The EORTC will have to address this problem within its short-term scientific strategy.