Measuring research in the big data era: The evolution of performance measurement systems in the Italian teaching hospitals

Hospital performance evaluation indicators: a scoping review

BACKGROUND

Hospitals are the biggest consumers of health system budgets and hence measuring hospital performance by quantitative or qualitative accessible and reliable indicators is crucial. This review aimed to categorize and present a set of indicators for evaluating overall hospital performance.

METHODS

We conducted a literature search across three databases, i.e., PubMed, Scopus, and Web of Science, using possible keyword combinations. We included studies that explored hospital performance evaluation indicators from different dimensions.

RESULTS

We included 91 English language studies published in the past 10 years. In total, 1161 indicators were extracted from the included studies. We classified the extracted indicators into 3 categories, 14 subcategories, 21 performance dimensions, and 110 main indicators. Finally, we presented a comprehensive set of indicators with regard to different performance dimensions and classified them based on what they indicate in the production process, i.e., input, process, output, outcome and impact.

CONCLUSION

The findings provide a comprehensive set of indicators at different levels that can be used for hospital performance evaluation. Future studies can be conducted to validate and apply these indicators in different contexts. It seems that, depending on the specific conditions of each country, an appropriate set of indicators can be selected from this comprehensive list of indicators for use in the performance evaluation of hospitals in different settings.

Retrospective Impact Evaluation Continuing to Prove Challenging Irrespective of Setting: A Study of Research Impact Enablers and Challenges Cloaked as an Impact Evaluation? Comment on "'We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service"

The original article provides a detailed and insightful presentation of enablers and detractors for research participation, translation, and impact, at a regional Australian hospital and health service. This information builds on existing knowledge, from the perspective of a non-metropolitan healthcare organisation. It stands to inform all healthcare organisations keen to embed research into their institutions. However, what the article fails to do is present the results of the research impact evaluation in a systematic and useful way for the reader to assess the benefits of research investment by a healthcare organisation including delivery of better quality care and improved patient outcomes. This commentary suggests why such information is critical to justify continued research investment by healthcare organisations and to showcase the potential benefits of the embedded research model. It also discusses the limitations of undertaking impact evaluation retrospectively and suggests that a prospective approach coupled with proper data collection systems and processes upfront could help future reporting of organisational research impact.

The Italian interregional performance evaluation system

This contribution describes the Interregional Performance Evaluation System (IRPES) designed by MeS Lab of Sant’Anna School of Advanced Studies (Pisa—Italy), by highlighting its main characteristics and its evolution.

Context: Since its creation in 2008, the IRPES has been developed by researchers from different backgrounds, with the involvement of practitioners and managers, so as to offer support to local, regional and national healthcare management. The IRPES is currently adopted by 12 Italian regions, which share a common performance evaluation system to assess the respective regional healthcare systems’ performance.

Structure: The IRPES is currently composed of about 400 indicators, which monitor different dimensions of the health systems. About half of them are benchmarked against shared standards, to assess the healthcare systems’ performance. Indicators are grouped into around thirty composite indicators.

Data visualization: MeS Lab designed innovative tools to deliver a clear representation of the performance of health systems, which in turn facilitate policymakers to gain a dynamic understanding of weaknesses and strengths of the systems they manage. In particular, the more traditional representation tools (such as maps or histograms) are accompanied by new visualisation instruments, such as “the dartboard” and “the stave”.

Covid-19: With the outburst of COVID-19 pandemic emergency, assessing the resilience of healthcare systems has become the new challenge posed to the IRPES. Thus, 63 novel resilience indicators tailored for the pandemic were introduced, starting from the second semester of 2020. Continuous monitoring of the performance of health systems was equally implemented, which has been essential to policymakers during such a difficult time.

Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers

Scholars in science and technology studies and bibliometricians are increasingly revealing the performative nature of bibliometric indicators. Far from being neutral technical measures, indicators such as the Impact Factor and the h-index are deeply transforming the social and epistemic structures of contemporary science. At the same time, scholars have highlighted how bibliometric indicators are endowed with social meanings that go beyond their purely technical definitions. These social representations of bibliometric indicators are constructed and negotiated between different groups of actors within several arenas. This study aims to investigate how bibliometric indicators are used in a context, which, so far, has not yet been covered by researchers, that of daily newspapers. By a content analysis of a corpus of 583 articles that appeared in four major Italian newspapers between 1990 and 2020, we chronicle the main functions that bibliometrics and bibliometric indicators played in the Italian press. Our material shows, among other things, that the public discourse developed in newspapers creates a favorable environment for bibliometrics-centered science policies, that bibliometric indicators contribute to the social construction of scientific facts in the press, especially in science news related to medicine, and that professional bibliometric expertise struggles to be represented in newspapers and hence reach the general public.