Sea change in open science and data sharing: leadership by industry

Clinical trial data sharing: a cross-sectional study of outcomes associated with two U.S. National Institutes of Health models

The impact and effectiveness of clinical trial data sharing initiatives may differ depending on the data sharing model used. We characterized outcomes associated with models previously used by the U.S. National Institutes of Health (NIH): National Heart, Lung, and Blood Institute's (NHLBI) centralized model and National Cancer Institute's (NCI) decentralized model. We identified trials completed in 2010-2013 that met NIH data sharing criteria and matched studies based on cost and/or size, determining whether trial data were shared, and for those that were, the frequency of secondary internal publications (authored by at least one author from the original research team) and shared data publications (authored by a team external to the original research team). We matched 77 NHLBI-funded trials to 77 NCI-funded trials; among these, 20 NHLBI-sponsored trials (26%) and 4 NCI-sponsored trials (5%) shared data (OR 6.4, 95% CI: 2.1, 19.8). From the 4 NCI-sponsored trials sharing data, we identified 65 secondary internal and 2 shared data publications. From the 20 NHLBI-sponsored trials sharing data, we identified 188 secondary internal and 53 shared data publications. The NHLBI's centralized data sharing model was associated with more trials sharing data and more shared data publications when compared with the NCI's decentralized model.

Characteristics of available studies and dissemination of research using major clinical data sharing platforms

BACKGROUND/AIMS

Over the past decade, numerous data sharing platforms have been launched, providing access to de-identified individual patient-level data and supporting documentation. We evaluated the characteristics of prominent clinical data sharing platforms, including types of studies listed as available for request, data requests received, and rates of dissemination of research findings from data requests.

METHODS

We reviewed publicly available information listed on the websites of six prominent clinical data sharing platforms: Biological Specimen and Data Repository Information Coordinating Center, ClinicalStudyDataRequest.com, Project Data Sphere, Supporting Open Access to Researchers-Bristol Myers Squibb, Vivli, and the Yale Open Data Access Project. We recorded key platform characteristics, including listed studies and available supporting documentation, information on the number and status of data requests, and rates of dissemination of research findings from data requests (i.e. publications in a peer-reviewed journals, preprints, conference abstracts, or results reported on the platform's website).

RESULTS

The number of clinical studies listed as available for request varied among five data sharing platforms: Biological Specimen and Data Repository Information Coordinating Center (n = 219), ClinicalStudyDataRequest.com (n = 2,897), Project Data Sphere (n = 154), Vivli (n = 5426), and the Yale Open Data Access Project (n = 395); Supporting Open Access to Researchers did not provide a list of Bristol Myers Squibb studies available for request. Individual patient-level data were nearly always reported as being available for request, as opposed to only Clinical Study Reports (Biological Specimen and Data Repository Information Coordinating Center = 211/219 (96.3%); ClinicalStudyDataRequest.com = 2884/2897 (99.6%); Project Data Sphere = 154/154 (100.0%); and the Yale Open Data Access Project = 355/395 (89.9%)); Vivli did not provide downloadable study metadata. Of 1201 data requests listed on ClinicalStudyDataRequest.com, Supporting Open Access to Researchers-Bristol Myers Squibb, Vivli, and the Yale Open Data Access Project platforms, 586 requests (48.8%) were approved (i.e. data access granted). The majority were for secondary analyses and/or developing/validating methods (ClinicalStudyDataRequest.com = 262/313 (83.7%); Supporting Open Access to Researchers-Bristol Myers Squibb = 22/30 (73.3%); Vivli = 63/84 (75.0%); the Yale Open Data Access Project = 111/159 (69.8%)); four were for re-analyses or corroborations of previous research findings (ClinicalStudyDataRequest.com = 3/313 (1.0%) and the Yale Open Data Access Project = 1/159 (0.6%)). Ninety-five (16.1%) approved data requests had results disseminated via peer-reviewed publications (ClinicalStudyDataRequest.com = 61/313 (19.5%); Supporting Open Access to Researchers-Bristol Myers Squibb = 3/30 (10.0%); Vivli = 4/84 (4.8%); the Yale Open Data Access Project = 27/159 (17.0%)). Forty-two (6.8%) additional requests reported results through preprints, conference abstracts, or on the platform's website (ClinicalStudyDataRequest.com = 12/313 (3.8%); Supporting Open Access to Researchers-Bristol Myers Squibb = 3/30 (10.0%); Vivli = 2/84 (2.4%); Yale Open Data Access Project = 25/159 (15.7%)).

CONCLUSION

Across six prominent clinical data sharing platforms, information on studies and request metrics varied in availability and format. Most data requests focused on secondary analyses and approximately one-quarter of all approved requests publicly disseminated their results. To further promote the use of shared clinical data, platforms should increase transparency, consistently clarify the availability of the listed studies and supporting documentation, and ensure that research findings from data requests are disseminated.

Healthcare data scientist qualifications, skills, and job focus: a content analysis of job postings

OBJECTIVE

Growth in big data and its potential impact on the healthcare industry have driven the need for more data scientists. In health care, big data can be used to improve care quality, increase efficiency, lower costs, and drive innovation. Given the importance of data scientists to U.S. healthcare organizations, I examine the qualifications and skills these organizations require for data scientist positions and the specific focus of their work.

MATERIALS AND METHODS

A content analysis of U.S. healthcare data scientist job postings was conducted using an inductive approach to capture and categorize core information about each posting and a deductive approach to evaluate skills required. Profiles were generated for 4 job focus areas.

RESULTS

There is a spectrum of healthcare data scientist positions that varies based on hiring organization type, job level, and job focus area. The focus of these positions ranged from performance improvement to innovation and product development with some positions more broadly defined to address organizational-specific needs. Based on the job posting sample, the primary skills these organizations required were statistics, R, machine learning, storytelling, and Python.

CONCLUSIONS

These results may be useful to organizations as they deepen our understanding of the qualifications and skills required for data scientist positions and may aid organizations in identifying skills and knowledge areas that have been overlooked in position postings.

Open Data Revolution in Clinical Research: Opportunities and Challenges

Efforts for sharing individual clinical data are gaining momentum due to a heightened recognition that integrated data sets can catalyze biomedical discoveries and drug development. Among the benefits are the fact that data sharing can help generate and investigate new research hypothesis beyond those explored in the original study. Despite several accomplishments establishing public systems and guidance for data sharing in clinical trials, this practice is not the norm. Among the reasons are ethical challenges, such as privacy of individuals, data ownership, and control. This paper creates awareness of the potential benefits and challenges of sharing individual clinical data, how to overcome these challenges, and how as a clinical pharmacology community we can shape future directions in this field.

Submissions from the SPRINT Data Analysis Challenge on clinical risk prediction: a cross-sectional evaluation

OBJECTIVES

To collate and systematically characterise the methods, results and clinical performance of the clinical risk prediction submissions to the Systolic Blood Pressure Intervention Trial (SPRINT) Data Analysis Challenge.

DESIGN

Cross-sectional evaluation.

DATA SOURCES

SPRINT Challenge online submission website.

STUDY SELECTION

Submissions to the SPRINT Challenge for clinical prediction tools or clinical risk scores.

DATA EXTRACTION

In duplicate by three independent reviewers.

RESULTS

Of 143 submissions, 29 met our inclusion criteria. Of these, 23/29 (79%) reported prediction models for an efficacy outcome (20/23 [87%] of these used the SPRINT study primary composite outcome, 14/29 [48%] used a safety outcome, and 4/29 [14%] examined a combined safety/efficacy outcome). Age and cardiovascular disease history were the most common variables retained in 80% (12/15) of the efficacy and 60% (6/10) of the safety models. However, no two submissions included an identical list of variables intending to predict the same outcomes. Model performance measures, most commonly, the C-statistic, were reported in 57% (13/23) of efficacy and 64% (9/14) of safety model submissions. Only 2/29 (7%) models reported external validation. Nine of 29 (31%) submissions developed and provided evaluable risk prediction tools. Using two hypothetical vignettes, 67% (6/9) of the tools provided expected recommendations for a low-risk patient, while 44% (4/9) did for a high-risk patient. Only 2/29 (7%) of the clinical risk prediction submissions have been published to date.

CONCLUSIONS

Despite use of the same data source, a diversity of approaches, methods and results was produced by the 29 SPRINT Challenge competition submissions for clinical risk prediction. Of the nine evaluable risk prediction tools, clinical performance was suboptimal. By collating an overview of the range of approaches taken, researchers may further optimise the development of risk prediction tools in SPRINT-eligible populations, and our findings may inform the conduct of future similar open science projects.

Overview and experience of the YODA Project with clinical trial data sharing after 5 years

The Yale University Open Data Access (YODA) Project has facilitated access to clinical trial data since 2013. The purpose of this article is to provide an overview of the Project, describe key decisions that were made when establishing data sharing policies, and suggest how our experience and the experiences of our first two data generator partners, Medtronic, Inc. and Johnson & Johnson, can be used to enhance other ongoing or future initiatives.

Clinical Trial Participants' Views of the Risks and Benefits of Data Sharing

Background Sharing of participant-level clinical trial data has potential benefits, but concerns about potential harms to research participants have led some pharmaceutical sponsors and investigators to urge caution. Little is known about clinical trial participants' perceptions of the risks of data sharing. Methods We conducted a structured survey of 771 current and recent participants from a diverse sample of clinical trials at three academic medical centers in the United States. Surveys were distributed by mail (350 completed surveys) and in clinic waiting rooms (421 completed surveys) (overall response rate, 79%). Results Less than 8% of respondents felt that the potential negative consequences of data sharing outweighed the benefits. A total of 93% were very or somewhat likely to allow their own data to be shared with university scientists, and 82% were very or somewhat likely to share with scientists in for-profit companies. Willingness to share data did not vary appreciably with the purpose for which the data would be used, with the exception that fewer participants were willing to share their data for use in litigation. The respondents' greatest concerns were that data sharing might make others less willing to enroll in clinical trials (37% very or somewhat concerned), that data would be used for marketing purposes (34%), or that data could be stolen (30%). Less concern was expressed about discrimination (22%) and exploitation of data for profit (20%). Conclusions In our study, few clinical trial participants had strong concerns about the risks of data sharing. Provided that adequate security safeguards were in place, most participants were willing to share their data for a wide range of uses. (Funded by the Greenwall Foundation.).

Data Sharing and Cardiology: Platforms and Possibilities

Sharing deidentified patient-level research data presents immense opportunities to all stakeholders involved in cardiology research and practice. Sharing data encourages the use of existing data for knowledge generation to improve practice, while also allowing for validation of disseminated research. In this review, we discuss key initiatives and platforms that have helped to accelerate progress toward greater sharing of data. These efforts are being prompted by government, universities, philanthropic sponsors of research, major industry players, and collaborations among some of these entities. As data sharing becomes a more common expectation, policy changes will be required to encourage and assist data generators with the process of sharing the data they create. Patients also will need access to their own data and to be empowered to share those data with researchers. Although medicine still lags behind other fields in achieving data sharing's full potential, cardiology research has the potential to lead the way.

Clinical research data sharing: what an open science world means for researchers involved in evidence synthesis

The International Committee of Medical Journal Editors (ICMJE) recently announced a bold step forward to require data generated by interventional clinical trials that are published in its member journals to be responsibly shared with external investigators. The movement toward a clinical research culture that supports data sharing has important implications for the design, conduct, and reporting of systematic reviews and meta-analyses. While data sharing is likely to enhance the science of evidence synthesis, facilitating the identification and inclusion of all relevant research, it will also pose key challenges, such as requiring broader search strategies and more thorough scrutiny of identified research. Furthermore, the adoption of data sharing initiatives by the clinical research community should challenge the community of researchers involved in evidence synthesis to follow suit, including the widespread adoption of systematic review registration, results reporting, and data sharing, to promote transparency and enhance the integrity of the research process.

The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses

POLICY POINTS

Currently, there is massive production of unnecessary, misleading, and conflicted systematic reviews and meta-analyses. Instead of promoting evidence-based medicine and health care, these instruments often serve mostly as easily produced publishable units or marketing tools. Suboptimal systematic reviews and meta-analyses can be harmful given the major prestige and influence these types of studies have acquired. The publication of systematic reviews and meta-analyses should be realigned to remove biases and vested interests and to integrate them better with the primary production of evidence.

CONTEXT

Currently, most systematic reviews and meta-analyses are done retrospectively with fragmented published information. This article aims to explore the growth of published systematic reviews and meta-analyses and to estimate how often they are redundant, misleading, or serving conflicted interests.

METHODS

Data included information from PubMed surveys and from empirical evaluations of meta-analyses.

FINDINGS

Publication of systematic reviews and meta-analyses has increased rapidly. In the period January 1, 1986, to December 4, 2015, PubMed tags 266,782 items as "systematic reviews" and 58,611 as "meta-analyses." Annual publications between 1991 and 2014 increased 2,728% for systematic reviews and 2,635% for meta-analyses versus only 153% for all PubMed-indexed items. Currently, probably more systematic reviews of trials than new randomized trials are published annually. Most topics addressed by meta-analyses of randomized trials have overlapping, redundant meta-analyses; same-topic meta-analyses may exceed 20 sometimes. Some fields produce massive numbers of meta-analyses; for example, 185 meta-analyses of antidepressants for depression were published between 2007 and 2014. These meta-analyses are often produced either by industry employees or by authors with industry ties and results are aligned with sponsor interests. China has rapidly become the most prolific producer of English-language, PubMed-indexed meta-analyses. The most massive presence of Chinese meta-analyses is on genetic associations (63% of global production in 2014), where almost all results are misleading since they combine fragmented information from mostly abandoned era of candidate genes. Furthermore, many contracting companies working on evidence synthesis receive industry contracts to produce meta-analyses, many of which probably remain unpublished. Many other meta-analyses have serious flaws. Of the remaining, most have weak or insufficient evidence to inform decision making. Few systematic reviews and meta-analyses are both non-misleading and useful.

CONCLUSIONS

The production of systematic reviews and meta-analyses has reached epidemic proportions. Possibly, the large majority of produced systematic reviews and meta-analyses are unnecessary, misleading, and/or conflicted.

EFSPI/PSI working group on data sharing: accessing and working with pharmaceutical clinical trial patient level datasets - a primer for academic researchers

BACKGROUND

Access to patient level datasets from clinical trial sponsors continues to be an important topic for the Pharmaceutical Industry as well as academic institutions and researchers. How to make access to patient level data actually happen raises many questions from the perspective of the researcher.

METHODS

Patient level data access models of all major pharmaceutical companies were surveyed and recommendations made to guide academic researchers in the most efficient way through the process of requesting and accessing patient level data.

RESULTS

The key considerations for researchers covered here are finding information; writing a research proposal to request data access; the review process; how data are shared; and the expectations of the data holder. A lot of clinical trial information is available on public registries and so these are great sources of information. Depending on the research proposal the required information may be available in Clinical Study Reports and therefore patient level data may not need to be requested. Many data sharing systems have an electronic form or template but in cases where these are not available the proposal needs to be created as a stand-alone document outlining the purpose, statistical analysis plan, identifying the studies for which data are required, the research team members involved, any conflicts of interest and the funding for the research. There are three main review processes - namely having an internal review board, external review board selected by the data holder or an external review board selected by a third party. Data can be shared through Open access i.e. on a public website, direct sharing between the data holder and the researcher, controlled access or the data holder identifies a contract organization to access the data and perform the analyses on behalf of the researcher. The data that are shared will have accompanying documentation to assist the researcher in understanding the original clinical trial and data collection methods. The data holder will require a legally binding data sharing agreement to be set up with the researcher. Additionally the data holder may be available to provide some support to the researcher if questions arise.

CONCLUSION

Whilst the benefits and value of patient level data sharing have yet to be fully realised, we hope that the information outlined in this article will encourage researchers to consider accessing and re-using clinical trial data to support their research questions.

Availability of Clinical Trial Data From Industry-Sponsored Cardiovascular Trials

BACKGROUND

Industry-sponsored clinical trials produce high-quality data sets that can be used by researchers to generate new knowledge. We assessed the availability of individual participant-level data (IPD) from large cardiovascular trials conducted by major pharmaceutical companies and compiled a list of available trials.

METHODS AND RESULTS

We identified all randomized cardiovascular interventional trials registered on ClinicalTrials.gov with >5000 enrollment, sponsored by 1 of the top 20 pharmaceutical companies by 2014 global sales. Availability of IPD for each trial was ascertained by searching each company's website/data-sharing portal. If availability could not be determined, each company was contacted electronically. Of 60 included trials, IPD are available for 15 trials (25%) consisting of 204 452 patients. IPD are unavailable for 15 trials (25%). Reasons for unavailability were: cosponsor did not agree to make IPD available (4 trials) and trials were not conducted within a specific time (5 trials); for the remaining 6 trials, no specific reason was provided. For 30 trials (50%), availability of IPD could not be definitively determined either because of no response or requirements for a full proposal (23 trials).

CONCLUSIONS

IPD from 1 in 4 large cardiovascular trials conducted by major pharmaceutical companies are confirmed available to researchers for secondary research, a valuable opportunity to enhance science. However, IPD from 1 in 4 trials are not available, and data availability could not be definitively determined for half of our sample. For several of these trials, companies require a full proposal to determine availability, making use of the IPD by researchers less certain.

Supporting open access to clinical trial data for researchers: The Duke Clinical Research Institute-Bristol-Myers Squibb Supporting Open Access to Researchers Initiative

There are growing calls for sponsors to increase transparency by providing access to clinical trial data. In response, Bristol-Myers Squibb and the Duke Clinical Research Institute have collaborated on a new initiative, Supporting Open Access to Researchers. The aim is to facilitate open sharing of Bristol-Myers Squibb trial data with interested researchers. Key features of the Supporting Open Access to Researchers data sharing model include an independent review committee that ensures expert consideration of each proposal, stringent data deidentification/anonymization and protection of patient privacy, requirement of prespecified statistical analysis plans, and independent review of manuscripts before submission for publication. We believe that these approaches will promote open science by allowing investigators to verify trial results as well as to pursue interesting secondary uses of trial data without compromising scientific integrity.

How to make more published research true

In a 2005 paper that has been accessed more than a million times, John Ioannidis explained why most published research findings were false. Here he revisits the topic, this time to address how to improve matters. Please see later in the article for the Editors' Summary.