Evidencing the impact of cancer trials: insights from the 2014 UK Research Excellence Framework

INTRODUCTION

An impactful clinical trial will have real-life benefits for patients and society beyond the academic environment. This study analyses case studies of cancer trials to understand how impact is evidenced for cancer trials and how impact evaluation can be more routinely adopted and improved.

METHODS

The United Kingdom (UK) Government allocates research funding to higher-education institutions based on an assessment of the institutions' previous research efforts, in an exercise known as the Research Excellence Framework (REF). In addition to each institution's journal publications and research environment, for the first time in 2014, allocation of funding was also dependent on an evaluation of the wider, societal impact of research conducted. In the REF2014, impact assessment was performed by evaluation of impact case studies. In this study, case studies (n = 6637) submitted by institutions for the REF2014 were accessed and those focussing on cancer trials were identified. Manual content analysis was then used to assess the characteristics of the cancer trials discussed in the case studies, the impact described and the methods used by institutions to demonstrate impact.

RESULTS

Forty-six case studies describing 106 individual cancer trials were identified. The majority were phase III randomised controlled trials and those recruiting patients with breast cancer. A list of indicators of cancer trial impact was generated using the previous literature and developed inductively using these case studies. The most common impact from a cancer trial identified in the case studies was on policy, in particular citation of trial findings in clinical guidelines. Impact on health outcomes and the economy were less frequent and health outcomes were often predicted rather than evidenced. There were few descriptions identified of trialists making efforts to maximise trial impact.

DISCUSSION

Cancer trial impact narratives for the next REF assessment exercise in 2021 can be improved by evidencing actual rather than predicted Impact, with a clearer identification of the beneficiaries of cancer trials and the processes through which trial results are used. Clarification of the individuals responsible for performing impact evaluations of cancer trials and the provision of resources to do so needs to be addressed if impact evaluation is to be sustainable.

Limited alignment of publicly competitive disease funding with disease burden in Japan

OBJECTIVE

The need to align investments in health research and development (R&D) with public health needs is one of the most important public health challenges in Japan. We examined the alignment of disease-specific publicly competitive R&D funding to the disease burden in the country.

METHODS

We analyzed publicly available data on competitive public funding for health in 2015 and 2016 and compared it to disability-adjusted life year (DALYs) in 2016, which were obtained from the Global Burden of Disease (GBD) 2017 study. Their alignment was assessed as a percentage distribution among 22 GBD disease groups. Funding was allocated to the 22 disease groups based on natural language processing, using textual information such as project title and abstract for each research project, while considering for the frequency of information.

RESULTS

Total publicly competitive funding in health R&D in 2015 and 2016 reached 344.1 billion JPY (about 3.0 billion USD) for 32,204 awarded projects. About 49.5% of the funding was classifiable for disease-specific projects. Five GDB disease groups were significantly and relatively well-funded compared to their contributions to Japan's DALY, including neglected tropical diseases and malaria (funding vs DALY = 1.7% vs 0.0%, p<0.01) and neoplasms (28.5% vs 19.2%, p<0.001). In contrast, four GDB disease groups were significantly under-funded, including cardiovascular diseases (8.0% vs 14.8%, p<0.001) and musculoskeletal disorders (1.0% vs 11.9%, p<0.001). These percentages do not include unclassifiable funding.

CONCLUSIONS

While caution is necessary as this study was not able to consider public in-house funding and the methodological uncertainties could not be ruled out, the analysis may provide a snapshot of the limited alignment between publicly competitive disease-specific funding and the disease burden in the country. The results call for greater management over the allocation of scarce resources on health R&D. DALYs will serve as a crucial, but not the only, consideration in aligning Japan's research priorities with the public health needs. In addition, the algorithms for natural language processing used in this study require continued efforts to improve accuracy.

Big Science vs. Little Science: How Scientific Impact Scales with Funding

Agencies that fund scientific research must choose: is it more effective to give large grants to a few elite researchers, or small grants to many researchers? Large grants would be more effective only if scientific impact increases as an accelerating function of grant size. Here, we examine the scientific impact of individual university-based researchers in three disciplines funded by the Natural Sciences and Engineering Research Council of Canada (NSERC). We considered four indices of scientific impact: numbers of articles published, numbers of citations to those articles, the most cited article, and the number of highly cited articles, each measured over a four-year period. We related these to the amount of NSERC funding received. Impact is positively, but only weakly, related to funding. Researchers who received additional funds from a second federal granting council, the Canadian Institutes for Health Research, were not more productive than those who received only NSERC funding. Impact was generally a decelerating function of funding. Impact per dollar was therefore lower for large grant-holders. This is inconsistent with the hypothesis that larger grants lead to larger discoveries. Further, the impact of researchers who received increases in funding did not predictably increase. We conclude that scientific impact (as reflected by publications) is only weakly limited by funding. We suggest that funding strategies that target diversity, rather than “excellence”, are likely to prove to be more productive.

Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators

BACKGROUND

Funders of medical research the world over are increasingly seeking, in research assessment, to complement traditional output measures of scientific publications with more outcome-based indicators of societal and economic impact. In the United Kingdom, the Higher Education Funding Council for England (HEFCE) developed proposals for the Research Excellence Framework (REF) to allocate public research funding to higher education institutions, inter alia, on the basis of the social and economic impact of their research. In 2010, it conducted a pilot exercise to test these proposals and refine impact indicators and criteria.

METHODS

The impact indicators proposed in the 2010 REF impact pilot exercise are critically reviewed and appraised using insights from the relevant literature and empirical data collected for the University of Oxford's REF pilot submission in clinical medicine. The empirical data were gathered from existing administrative sources and an online administrative survey carried out by the university's Medical Sciences Division among 289 clinical medicine faculty members (48.1% response rate).

RESULTS

The feasibility and scope of measuring research impact in clinical medicine in a given university are assessed. Twenty impact indicators from seven categories proposed by HEFCE are presented; their strengths and limitations are discussed using insights from the relevant biomedical and research policy literature.

CONCLUSIONS

While the 2010 pilot exercise has confirmed that the majority of the proposed indicators have some validity, there are significant challenges in operationalising and measuring these indicators reliably, as well as in comparing evidence of research impact across different cases in a standardised manner. It is suggested that the public funding agencies, medical research charities, universities, and the wider medical research community work together to develop more robust methodologies for capturing and describing impact, including more valid and reliable impact indicators.

Pediatric sports-related lower extremity fractures: hospital length of stay and charges: what is the role of the primary payer?

OBJECTIVE

The purposes of this study were (a) to evaluate the distribution by primary payer (public vs. private) of U.S. pediatric patients aged 5-18 years who were hospitalized with a sports-related lower extremity fracture and (b) to discern the adjusted mean hospital length of stay and mean charge per day by payer type.

METHODS

Children who were aged 5 to 18 years and had diagnoses of lower extremity fracture and sports-related injury in the 2006 Healthcare Cost and Utilization Project Kids' Inpatient Database were included. Lower extremity fractures are defined as International Classification of Diseases, 9th Revision, Clinical Modification codes 820-829 under Section "Injury and Poisoning (800-999)," while sports-related external cause of injury codes (E-codes) are E886.0, E917.0, and E917.5. Differences in hospital length of stay and cost per day by payer type were assessed via adjusted least square mean analysis.

RESULTS

The adjusted mean hospital length of stay was 20% higher for patients with a public payer (2.50 days) versus a private payer (2.08 days). The adjusted mean charge per day differed about 10% by payer type (public, US$7,900; private, US$8,794).

CONCLUSIONS

Further research is required to identify factors that are associated with different length of stay and mean charge per day by payer type, and explore whether observed differences in hospital length of stay are the result of private payers enhancing patient care, thereby discharging patients in a more efficient manner.

A best-fit model for concept vectors in biomedical research grants

The Research, Condition, and Disease Categorization (RCDC) project was created to standardize budget reporting by research topic. Text mining techniques have been implemented to classify NIH grant applications into proper research and disease categories. A best-fit model is shown to achieve classification performance rivaling that of concept vectors produced by human experts.

Refining estimates of public health spending as measured in national health expenditure accounts: the Canadian experience

The recent focus on public health stemming from, among other things, severe acute respiratory syndrome and avian flu has created an imperative to refine health-spending estimates in the Canadian Health Accounts. This article presents the Canadian experience in attempting to address the challenges associated with developing the needed taxonomies for systematically capturing, measuring, and analyzing the national investment in the Canadian public health system. The first phase of this process was completed in 2005, which was a 2-year project to estimate public health spending based on a more classic definition by removing the administration component of the previously combined public health and administration category. Comparing the refined public health estimate with recent data from the Organization for Economic Cooperation and Development still positions Canada with the highest share of total health expenditure devoted to public health than any other country reporting. The article also provides an analysis of the comparability of public health estimates across jurisdictions within Canada as well as a discussion of the recommendations for ongoing improvement of public health spending estimates. The Canadian Institute for Health Information is an independent, not-for-profit organization that provides Canadians with essential statistics and analysis on the performance of the Canadian health system, the delivery of healthcare, and the health status of Canadians. The Canadian Institute for Health Information administers more than 20 databases and registries, including Canada's Health Accounts, which tracks historically 40 categories of health spending by 5 sources of finance for 13 provincial and territorial jurisdictions. Until 2005, expenditure on public health services in the Canadian Health Accounts included measures to prevent the spread of communicable disease, food and drug safety, health inspections, health promotion, community mental health programs, public health nursing, as well as all the costs for the general administration of government health departments.

Creating financial transparency in public health: examining best practices of system partners

Financial transparency is based on concepts for valid, standardized information that is readily accessible and routinely disseminated to stakeholders. While Congress and others continuously ask for an accounting of public health investments, transparency remains an ignored concept. The objective of this study was to examine financial transparency practices in other industries considered as part of the public health system. Key informants, regarded as financial experts on the operations of hospitals, school systems, and higher education, were a primary source of information. Principal findings were that system partners have espoused some concepts for financial transparency beginning in the early 20th century--signifying an 80-year implementation gap for public health. Critical features that promote accountability included standardized data collection methods and infrastructures, uniform practices for quantitative analysis of financial performance, and credentialing of the financial management workforce. Recommendations are offered on the basis of these findings to aid public health to close this gap by framing a movement toward transparency.

Refining estimates of public health spending as measured in national health expenditures accounts: the United States experience

Providing for the delivery of public health services and understanding the funding mechanisms for these services are topics of great currency in the United States. In 2002, the Department of Homeland Security was created and the responsibility for providing public health services was realigned among federal agencies. State and local public health agencies are under increased financial pressures even as they shoulder more responsibilities as the vital first link in the provision of public health services. Recent events, such as hurricanes Katrina and Rita, served to highlight the need to accurately access the public health delivery system at all levels of government. The National Health Expenditure Accounts (NHEA), prepared by the National Health Statistics Group, measure expenditures on healthcare goods and services in the United States. Government public health activity constitutes an important service category in the NHEA. In the most recent set of estimates, Government Public Health Activity expenditures totaled $56.1 billion in 2004, or 3.0 percent of total US health spending. Accurately measuring expenditures for public health services in the United States presents many challenges. Among these challenges is the difficult task of defining what types of government activity constitute public health services. There is no clear-cut, universally accepted definition of government public health care services, and the definitions in the proposed International Classification for Health Accounts are difficult to apply to an individual country's unique delivery systems. Other challenges include the definitional issues associated with the boundaries of healthcare as well as the requirement that census and survey data collected from government(s) be compliant with the Classification of Functions of Government (COFOG), an internationally recognized classification system developed by the United Nations.

Financing Public Health

This article documents the instability and variation in public financing of public health functions at the federal and state levels. Trust for America's Health has charted federal funding for the Centers of Disease Control and Prevention, which in turn provides a major portion of financing for state and local public health departments, and has compiled information about state-generated revenue commitments to public health activities nationwide. The federal-level analysis shows that funding has been marked by diminished support for "core" public health functions. The state-level analysis shows tremendous variation in use of state revenues to support public health functions. The combination of these factors results in very different public health capacities across the country, potentially leaving some states more vulnerable, while simultaneously posing a general threat to the nation since public health problems do not honor state borders. On the basis of this analysis, the authors suggest changes in the financing arrangements for public health, designed to assure a more stable funding stream for core public health functions and a more consistent approach to financing public health activities across the country.

Impact of state vaccine financing policy on uptake of heptavalent pneumococcal conjugate vaccine

OBJECTIVE

We examined heptavalent pneumococcal conjugate vaccine (PCV7) uptake among children aged 19 to 35 months in the United States and determined how uptake rates differed by state vaccine financing policy.

METHODS

We analyzed data from the 2001-2003 National Immunization Survey. States that changed their vaccine financing policy between 2001 and 2003 (n=17) were excluded from analysis. Logistic regression was performed to identify the association between state vaccine financing policy and receipt of 3 or more doses of PCV7 after control for demographic characteristics.

RESULTS

The proportion of children receiving 3 or more doses increased from 6.7% in 2001 to 69.0% in 2003. After controlling for demographic characteristics, children residing in states that provided all vaccines except PCV7 to all children had lower odds of receiving 3 or more doses compared to children residing in states that provided PCV7 only to children eligible for the Vaccines for Children program (odds ratio=0.58; 95% confidence interval=0.51, 0.66).

CONCLUSION

It is essential that we continue to monitor the effect that state vaccine financing policy has on the delivery of PCV7 and future vaccines, which are likely to be increasingly expensive.

[Economic evaluation of the new national breast cancer screening programme in France: application to the Bouche-du-Rhone district]

The purpose is to measure the costs of the new national breast cancer screening programme in France and to compare these with those of the previous programme in the Bouches-du-Rhône district. Direct screening costs and costs related to diagnosis and assessment were collected. Costs are presented by screening period, by organisms involved in the screening program and by corresponding phase within the screening process. The total cost of the screening program total cost has increased from 5587487 euros to 9345469 euros between the two campaigns. The main reasons are the investment costs in the new screening program, the increase in the target population and the increased fee for programs. This study presents a first estimate of the costs related to the new national breast cancer screening program. Results of this study may help to guide future decisions on the further development of breast cancer screening in France.

Burden of health care costs: businesses, households, and governments, 1987-2000

In this article, we estimate expenditures by businesses, households, and governments in providing financing for health care for 1987-2000 and track measures of burden that these costs impose. Although burden measures for businesses and the Federal Government have stabilized or improved since 1993, measures of burden for State and local governments are deteriorating slightly--a situation that is likely to worsen in the near future. As health care spending accelerates and an economy wide recession seems imminent, businesses, households, and governments that finance health care will face renewed health cost pressures on their revenue and income.

Major U.S. disability and compensation systems graphically compared

This article offers the reader a "bullet" overview of the principal systems of disability determination. The reader is referred to other sources for more in-depth coverage and a detailed historical background and overview of these systems.