Evaluating the Quality of Research Ethics Review and Oversight: A Systematic Analysis of Quality Assessment Instruments

How Do Accredited Organizations Evaluate the Quality and Effectiveness of Their Human Research Protection Programs?

BACKGROUND

Meaningfully evaluating the quality of institutional review boards (IRBs) and human research protection programs (HRPPs) is a long-recognized challenge. To be accredited by the Association for the Accreditation of Human Research Protection Programs (AAHRPP), organizations must demonstrate that they measure and improve HRPP "quality, effectiveness, and efficiency" (QEE). We sought to learn how AAHRPP-accredited organizations interpret and satisfy this standard, in order to assess strengths, weaknesses, and gaps in current approaches and to inform recommendations for improvement.

METHODS

We conducted 3 small-group interviews with a total of 19 participant representatives of accredited organizations at the 2019 AAHRPP annual meeting. Participants were eligible if they had familiarity with their organization's approach to satisfying the relevant QEE standard.

RESULTS

Participants reported lacking clear definitions for HRPP quality or effectiveness but described various approaches to assessing QEE, typically focused on turnaround time, compliance, and researcher satisfaction. Evaluation of IRB members was described as relatively superficial and information regarding research subject experience was not reported as central to QEE assessment, although participants described several efforts to improve consideration of patient, subject, and community perspectives in IRB review. Participants also described efforts to educate and build relationships with key stakeholders as important features of a high-quality HRPP. While generally satisfied with their approaches, participants expressed concern about resource and time constraints that pushed them to be reactive and automatic about QEE, rather than proactive and critical.

CONCLUSIONS

The relevant AAHRPP accreditation standard may obscure critical gaps in defining and measuring QEE elements. We recommend that AAHRPP: (1) offer a definition of QEE or require accredited organizations to provide their own, to help clarify the rationale and goals behind assessment and improvement efforts, and (2) require accredited organizations to establish QEE objectives and measures focused on participant outcomes and deliberative quality during protocol review.

Engaging key stakeholders to overcome barriers to studying the quality of research ethics oversight

The primary purpose of Institutional Review Boards (IRBs) is to protect the rights and welfare of human research participants. Evaluation and measurement of how IRBs satisfy this purpose and other important goals are open questions that demand empirical research. Research on IRBs, and the Human Research Protection Programs (HRPPs) of which they are often a part, is necessary to inform evidence-based practices, policies, and approaches to quality improvement in human research protections. However, to date, HRPP and IRB engagement in empirical research about their own activities and performance has been limited. To promote engagement of HRPPs and IRBs in self-reflective research on HRPP and IRB quality and effectiveness, barriers to their participation need to be addressed. These include: extensive workloads, limited information technology systems, and few universally accepted or consistently measured metrics for HRPP/IRB quality and effectiveness. Additionally, institutional leaders may have concerns about confidentiality. Professional norms around the value of participating in this type of research are lacking. Lastly, obtaining external funding for research on IRBs and HRPPs is challenging. As a group of HRPP professionals and researchers actively involved in a research consortium focused on IRB quality and effectiveness, we identify potential levers for supporting and encouraging HRPP and IRB engagement in research on quality and effectiveness. We maintain that this research should be informed by the core principles of patient- and community-engaged research, in which members and key stakeholders of the community to be studied are included as key informants and members of the research team. This ensures that relevant questions are asked and that data are interpreted to produce meaningful recommendations. As such, we offer several ways to increase the participation of HRPP professionals in research as participants, as data sharers, and as investigators.

"We measure what we can measure": Struggles in defining and evaluating institutional review board quality

There has been a persistent lack of clarity regarding how to define and measure the quality of Institutional Review Boards (IRBs). To address this challenge, we interviewed 43 individuals designated as IRB Stakeholders, including leaders in research ethics oversight, policymakers, investigators, research sponsors, and patient advocates, about their views regarding key features of IRB quality and how those features could be measured. We also interviewed 20 U.S. IRB directors (or individuals in similar roles) to learn how their institutions currently define and measure IRB quality and to assess satisfaction with those approaches. We analyzed the interviews, all of which were conducted in 2018, using a modified grounded theory approach. Individuals in the Stakeholder group struggled both to define IRB quality and identify appropriate measures. Those in the Director group gave less abstract and more bounded accounts, offering definitions of quality based on what their institutions currently measure. In identifying core definitional elements of IRB quality, both groups discussed efficiency, compliance, board and staff qualifications, and research facilitation. However, in an important omission by Directors, only Stakeholders named participant protection and thoughtful review as essential elements of IRB quality, despite the centrality of these factors to the very purpose of IRBs. Directors in our sample were largely satisfied with their institutions' current approaches to quality measurement, which included audits of internal processes and regulatory compliance, efficiency tracking, and feedback from board members and researchers. In addition to fleshing out what it means for IRB discretion to be exercised reasonably, adopting proposed metrics related to participant protection outcomes could help IRBs refocus on their core mission and prevent them from falling further into the broader trend of 'audit culture.'

The Challenges of Big Data for Research Ethics Committees: A Qualitative Swiss Study

Big data trends in health research challenge the oversight mechanism of the Research Ethics Committees (RECs). The traditional standards of research quality and the mandate of RECs illuminate deficits in facing the computational complexity, methodological novelty, and limited auditability of these approaches. To better understand the challenges facing RECs, we explored the perspectives and attitudes of the members of the seven Swiss Cantonal RECs via semi-structured qualitative interviews. Our interviews reveal limited experience among REC members with the review of big data research, insufficient expertise in data science, and uncertainty about how to mitigate big data research risks. Nonetheless, RECs could strengthen their oversight by training in data science and big data ethics, complementing their role with external experts and ad hoc boards, and introducing precise shared practices.

Smart Workplaces for older adults: coping 'ethically' with technology pervasiveness

Pervasive technologies such as Artificial Intelligence, Virtual Reality and the Internet of Things, despite their great potential for improved workability and well-being of older workers, entail wide ethical concerns. Aligned with these considerations we emphasize the need to present from the viewpoint of ethics the risks of personalized ICT solutions that aim to remedy health and support the well-being of the ageing population at workplaces. The ethical boundaries of digital technologies are opaque. The main motivation is to cope with the uncertainties of workplaces' digitization and develop an ethics framework, termed SmartFrameWorK, for personalized health support through ICT tools at workplace environments. SmartFrameWorK is built upon a five-dimensional approach of ethics norms: autonomy, privacy, transparency, trustworthiness and accountability to incite trust in digital workplace technologies. A typology underpins these principles and guides the ethical decision-making process with regard to older worker particular needs, context, data type-related risks and digital tools' use throughout their lifecycle. Risk analysis of pervasive technology use and multimodal data collection, highlighted the imperative for ethically aware practices for older workers' activity and behaviour monitoring. The SmartFrameWorK methodology has been applied in a case study to provide evidence that personalized digital services could elicit trust in users through a well-defined framework. Ethics compliance is a dynamic process from participants' engagement to data management. Defining ethical determinants is pivotal towards building trust and reinforcing better workability and well-being in older workers.

Ethics in Research: Relevance for Nephrology

Research is crucial to progress in nephrology. It is important that studies are conducted rigorously from the scientific perspective, as well as in adherence to ethical standards. Traditional clinical research places a high value on individual research subject autonomy. Research questions often include the clinical effectiveness of new interventions studied under highly controlled conditions. Such research has brought the promise of new game-changers in nephrology, such as the sodium-glucose cotransporter 2 inhibitors. Implementation research takes such knowledge further and investigates how to translate it into broader-scale policy and practice, to achieve swift and global uptake, with a focus on justice and equity. Newer challenges arising globally in research ethics include those relating to oversight of innovation, biobanking and big data, human-challenge studies, and research during emergencies. This article details the history of clinical research ethics and the role of research ethics committees, describes the evolving spectrum of biomedical research in human medicine, and presents emerging clinical research ethics issues using illustrative examples and a hypothetical case study. It is imperative that researchers and research ethics committees are well versed in the ethical principles of all forms of human research such that research is conducted to the highest standards and that effective interventions can be implemented at scale as rapidly as possible.

Ethics review of big data research: What should stay and what should be reformed?

BACKGROUND

Ethics review is the process of assessing the ethics of research involving humans. The Ethics Review Committee (ERC) is the key oversight mechanism designated to ensure ethics review. Whether or not this governance mechanism is still fit for purpose in the data-driven research context remains a debated issue among research ethics experts.

MAIN TEXT

In this article, we seek to address this issue in a twofold manner. First, we review the strengths and weaknesses of ERCs in ensuring ethical oversight. Second, we map these strengths and weaknesses onto specific challenges raised by big data research. We distinguish two categories of potential weakness. The first category concerns persistent weaknesses, i.e., those which are not specific to big data research, but may be exacerbated by it. The second category concerns novel weaknesses, i.e., those which are created by and inherent to big data projects. Within this second category, we further distinguish between purview weaknesses related to the ERC's scope (e.g., how big data projects may evade ERC review) and functional weaknesses, related to the ERC's way of operating. Based on this analysis, we propose reforms aimed at improving the oversight capacity of ERCs in the era of big data science.

CONCLUSIONS

We believe the oversight mechanism could benefit from these reforms because they will help to overcome data-intensive research challenges and consequently benefit research at large.

Institutional Review Board Quality, Private Equity, and Promoting Ethical Human Subjects Research

Evaluating the quality and effectiveness of the institutional review boards (IRBs) responsible for overseeing research involving human participants is critically important but perpetually challenging. Seemingly common-sense measures, such as the number of proposals approved with and without major modifications and the number of unexpected adverse events occurring in approved protocols, can be misleading indicators of participant protection, and regulatory compliance may not correspond to achieving ethical goals. These measurement challenges make it difficult to assess the validity of concerns about different IRB models. A group of U.S. senators recently raised questions about the increasing use of for-profit IRBs to review research proposals (as opposed to boards typically housed at academic medical centers and health care institutions) and, more specifically, about the growing trend of private equity ownership and consolidation of for-profit IRBs. Although all IRBs face pressure to speed reviews and none are entirely free of conflicts of interest, the private equity model is particularly susceptible to approaches that could undercut the ethical mission of IRBs to protect and promote the rights and welfare of research participants. Ideally, the quality of board oversight could be measured directly, rather than relying on the heuristic of board type; this article describes several current efforts toward this goal. In the meantime, one improvement may be to pursue a new model of IRB oversight: independent nonprofit boards that stand apart from research institutions, take advantage of business approaches to research review, and minimize conflicts of interest.