A comparative analysis of global and national university ranking systems

Exploring the role of ranking systems towards university performance improvement: A focus group-based study

The development of multiple university ranking systems at national and global levels has been driven by increasing interest in improving efficiency in the national educational sector without compromising the demand for international standards. Global university ranking systems play an important role by providing the foundation for competing in this global era. One approach could be developing and evaluating criteria to reduce the unnecessary use of standard, less productive indicators. This study aims to systematically exploit national and global university ranking systems in terms of their indicators and relevance to national educational needs. This study uses two online qualitative focus groups with 10 participants each. The participants were purposively sampled, and the transcribed data from the focus group were thematically analyzed. Findings indicate that university ranking should include indicators like governance and digital presence, as these are missing in global ranking indicators. These findings will guide the development of a university ranking framework that policymakers and universities can implement to improve institutional performance.

Higher education research performance parameters classified by systems theory: antecedents for the development of assessment models

Purpose

This paper aims to identify and classify the parameters that construct the input, processes, output, productivity and outcome variables that affect performance. These parameters are used in the evaluation model to measure research performance in universities so that they can be used as the basis for making leadership policies both at the national and institutional levels.

Design/methodology/approach

The design of this research is a quantitative research method using a survey questionnaire that was sent to the heads of research institutions at universities in Indonesia. To obtain these parameters, a test for determining the value of the loading factor was used.

Findings

The authors found that input variable parameters consisted of 10 parameters; process variable consisted of 22 parameters; output variable parameters consisted of 8 parameters; productivity variable consisted of 4 parameters; and outcome variable parameters consisted of 10 parameters.

Originality/value

One approach to obtain parameters is through systems theory, where every element that makes up the organization contributes to the achievement of goals. This study attempted to develop parameters in the performance appraisal model of systems theory-based research institutions that are adapted to trends in the direction of research in universities. These parameters are based on aspects of input, process, output, productivity and outcome.

A novel hybrid MCDM approach to evaluate universities based on student perspective

University rankings are an essential source of comparisons between universities according to specific combinations of criteria. International or national rankings have an increasing impact on higher education institutions, stakeholders, and their environments. Thereby, on behalf of effective decision-making, university-ranking efforts should be a process involving some conflicting criteria and uncertainties in a more sensitive manner. This study presents a detailed university evaluation procedure under certain service criteria via multi-criteria decision-making (MCDM) methodologies and provides an appropriate clustering of universities according to teaching and research factors. A hierarchical cluster-based Interval Valued Neutrosophic Analytic Hierarchy Process (IVN-AHP) integrated VIKOR methodology that includes two stages, clustering and ranking, is proposed for the university evaluation problem. The hierarchical clustering method is performed using teaching and research factors in the first stage. The second stage addresses the determination weights of service criteria through IVN-AHP and the ranking of universities by using VIKOR according to service criteria under determined clusters. This study, in which the proposed methodology is applied to Turkish universities, is the most comprehensive in terms of the number of universities evaluated and participating students. Furthermore, the integration of IVN-AHP and VIKOR to solve MCDM problems is presented for the first time. This study differs from other studies in terms of novelties both methodological-based and application based. Moreover, categorizing universities with similar characteristics into groups using cluster analysis and ranking them with the MCDM methodology provide a more realistic and effective interpretation of the results.

A practical framework for ranking universities by their competitive advantages: a mixed methods study on foundation universities in Turkey

Purpose

The purpose of this research is to determine the competitive advantages of higher education institutions (HEIs) and create a new methodology to rank universities according to the competitive advantages.

Design/methodology/approach

The research determines the competitive advantages of HEIs by analysing expert opinions through a semi-structured interview form, matches codes and themes to performance indicators using Saldana's two-cycle coding methods, evaluates content validity through Lawshe and reveals the item weights of the ranking with analytical hierarchy process (AHP). Simple additive weighting (SAW) and Technique for Order of Preference by Similarity (TOPSIS) methods were used for ranking universities.

Findings

Seven dimensions stand out in regard to what should be considered while ranking HEIs: research and publication, education, management, infrastructure, financial resources, human resources and social and economic contribution. Under the 7 dimensions, 69 indicators were determined.

Practical implications

The research provides a scientific reference point where HEIs can compare themselves with other HEIs regarding where they are in the sector, especially in terms of competitive advantages.

Originality/value

Although there are many different ranking methods that rank universities in the national and international literature, almost all these methods are largely based on the outputs of the university such as the number of publications, the number of patents, the number of projects, etc. A framework which ranks universities by considering different aspects of the institution, such as management, human resources and financial resources, has not been developed yet. In this respect, this research aims to fill this gap in the literature.

Internal Operational Metrics and Center for Medicare and Medicaid Services Hospital Compare Quality Ratings

ABSTRACT

The Center for Medicare and Medicaid Services (CMS) has made several refinements to their model for calculating hospital quality star ratings (Hospital Compare) amidst criticism and evidence of bias against some institutions. We argue that the CMS model does align with important internal quality metrics and encourage a measured approach to redesign, potentially using categorizations or tiers, rather than a complete abandonment of the ratings system. We find that institutional characteristics (available resources, average severity of illness, and academic affiliation) are associated with internal quality metrics related to patient flow. Furthermore, regression results from the original and revised CMS star rating methodologies suggest that patient flow metrics (discharges before noon [p < .01] and weekend discharges [p < .001]) have a positive relationship with the Hospital Compare rating. Hospitals with better patient flow, as measured by higher levels of discharges before noon and weekend discharges, are associated with higher CMS quality ratings. These findings suggest that CMS star ratings do reflect key aspects of operational performance, specifically efforts to improve patient flow, but the ranking system should consider hospital characteristics that influence internal operations as we move toward a system capable of quality and price transparency for consumers.

Lessons from the Criticality of the Spanish High Capacity Road Network on Direct, Representative Democracies and Technocracies

This paper aims to shed some light on an issue that has not been analyzed enough in previous studies on complex transportation networks. The financial crisis and disrupting events like the COVID-19 pandemic episode are affecting how governments make crucial decisions regarding policymaking paying more attention to experts' opinions. The impacts of disruptive events that could affect each road section will be analyzed using the criticality of the high capacity road network in Spain under different policymaking scenarios, such as direct democracy based on pure provincial decentralization (federal or cantonal vision), representative democracy in which decisions are taken by the National Parliament, and the governance based on technocracy stimulated by 'what matters is what works' (Southern Local Economy, 16(4), 264-271, 2001) using an Evidence-Based Policy Making (EBPM) case study based on a Data Envelopment Analysis applied to four accessibility indicators. We will complement our analysis with the different results obtained by the different national parties that were represented in the past election (November, 2019): PSOE, PP, VOX, Unidas Podemos, Ezquerra Republicana de Catalunya, Ciudadanos, Junts per Catalunya, Partido Nacionalista Vasco and Euskal Herria Bildu, exploring in-depth the obtained differences between the technocrat solution and the direct and representative democracy results. Important insights and lessons for the future will be obtained from the different party visions observed among the regionalist (nationalist) and the state-wide parties. The spillover effects created by the networks are so important that the room for federalist solutions might be very limited.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12061-022-09451-5.

Correlation between universities’ Altmetric Attention Scores and their performance scores in Nature Index, Leiden, Times Higher Education and Quacquarelli Symonds ranking systems

Objective:

Altmetrics are claimed to measure the scientific, societal, educational, technological and economic impacts of science. They have some of these dimensions in common with university ranking and evaluating systems. Their results are, therefore, expected to be partially convergent with the systems’. Given the importance of the scientific and non-scientific impacts of science, this study investigated the correlations of universities’ altmetrics with their total and dimensional scores in Nature Index, Leiden, Times Higher Education (THE) and Quacquarelli Symonds (QS).

Methodology:

Following a correlational design, it explored an available sample of the universities commonly ranked in the systems in 2017. The data were collected from online documents using checklists and analysed by the Spearman correlation. As Altmetric Attention Score (ASS) is efficient in that it integrates several indicators into a single one, it was used as the proxy of the universities’ social performance.

Findings:

The universities showed significant positive correlations between their ASSs and their performance scores on the total and dimensional levels, except for industry income in THE, with an insignificant correlation, and proportion of collaborative publication less than 100 km. in Leiden, with an inverse correlation. The correlations ranged from weak to marginally strong.

Conclusion:

The positive relationships between the universities’ performance and ASSs signified that there existed some similarities in what they measured. However, they were of weak-to-marginally strong powers, implying that the metrics differed in what they measured. The findings contribute to the existing knowledge by providing some evidence of convergence between university-level altmetrics and university performances in various dimensions.

Study on problems and countermeasures of ideological and political teaching in colleges and universities under the background of new media era

At present, with the continuous improvement and rapid development of the socialist market economy in China, especially the sudden rise of electronic information technology, the new communication tools are changing with each passing day. In this process, new media information technology, with its unique advantages, “broke” into the vision of modern people, participated in and influenced people’s way of life, behavior habits and social operation, and brought the development of human society into a new stage, the new media age. In today’s era, the new media has been developing rapidly in terms of technology and content, and its content is timeliness and interesting. It matches the characteristics of contemporary college students in pursuit of freshness and individuality, so that it has won the general favor of college students. However, when the new media information technology affects and permeates all aspects of the University, the ideological and political education of college students does not seem to keep pace with the times. Because of the lack of experience, many colleges and universities have not integrated the new media information technology with the ideological and political education of college students, but simply turn the original paper into a web page, ignore the readability and dissemination of the content, resulting in the poor practical effect of education, and then a series of real problems urgently to be solved. Therefore, this paper makes a detailed summary of the problems and Countermeasures of the current new media and college students’ Ideological and political teaching, aiming at arousing people’s attention to the political and ideological education of college students in the new media era, and providing suggestions for the use of new media information technology to promote the development of College Students’ Ideological and Political Education under the information age.

A retrospective overview of Journal of Enterprise Information Management using bibliometric analysis

Purpose

Journal of Enterprise Information Management (JEIM) is a leading journal that publishes studies on applied information management relevant to industry personals, academicians and researchers. This study uses bibliometric tools to present a retrospective analysis of the journal's outcomes.

Design/methodology/approach

The authors applied bibliometric tools for analysing the impact, topic coverage, renowned authors with affiliation, citation, methodology and analysis of the JEIM corpus. Additionally, they used bibliographic coupling to develop a graphical visualisation and analyse the journal's thematic evolution.

Findings

With 16 yearly articles, JEIM contributed 656 research articles on various themes. The major themes that have come to define the JEIM over this time include information and systems, supply chain management, manufacturing resource planning, communication technologies and small- to medium-sized enterprises. Empirical methodology, quantitative techniques with descriptive analysis and regression methods are the most preferred. The article's primary research purpose shows the majority of theory-verifying articles. Co-authorship analysis reveals that the single-author trend is decreasing and the journal now has articles with international collaborations.

Originality/value

This study is the retrospective analysis of the JEIM, which is useful for aspiring contributors and the journal's editors.

Improvement of Quality of Higher Education Institutions as a Basis for Improvement of Quality of Life

This paper aims to propose a quality assessment model for higher education institutions in the technical–technological field and a system for decision support and optimal management strategies for quality improvement. Obtaining research results is based on surveying stakeholders in higher education and obtaining quantitative data regarding key performance indices. Quantitative data and the genetic algorithm method are applied to determine optimal management strategies for quality improvement. Quality in the higher education sector is among the current issues in the academic community. By monitoring and researching the higher education field and analysing the literature and the current situation in the system of higher education in developing countries, it can be concluded that there is no single way to assess the quality of higher education institutions. This knowledge was a good starting point for the research presented in this paper. Accordingly, the findings include developing a system for quality assessment and the ranking of higher education institutions. Additionally, evaluating the relevance of key performance indicators of higher education institutions differs from different stakeholder perspectives. However, it is possible to develop a system for decision support and the selection of the optimal strategy for improving the performance of study programs and higher education institutions with regard to quality. The practical implications include defining a decision support system that enables the adoption of optimal decisions by the management teams of higher education institutions to improve study programs and the performance of the higher education institutions. The presented system may enable the benchmarking, simulation, and verification of different scenarios for improving the quality and performance of higher education institutions. In this paper, the authors analysed the characteristics, benefits, and drawbacks of different ranking systems to develop and introduce a novel ranking system that suggests weights for the ranking criteria and different perspectives regarding new digital age requirements. The model was tested, and the results are presented to demonstrate the advantages of the developed model. The originality of the research lies in the presented novel model that can be made available to government institutions and serve as a basis for the overall ranking and evaluation of higher education institutions, with the possibility of developing a performance-based funding system. Additionally, other stakeholders can gain an insight into the performance of an institution in relation to their needs and goals.

The rank boost by inconsistency in university rankings: Evidence from 14 rankings of Chinese universities

University ranking has become an important indicator for prospective students, job recruiters, and government administrators. The fact that a university rarely has the same position in different rankings motivates us to ask: To what extent could a university’s best rank deviate from its “true” position? Here we focus on 14 rankings of Chinese universities. We find that a university’s rank in different rankings is not consistent. However, the relative positions for a particular set of universities are more similar. The increased similarity is not distributed uniformly among all rankings. Instead, the 14 rankings demonstrate four clusters where rankings are more similar inside the cluster than outside. We find that a university’s best rank strongly correlates with its consensus rank, which is, on average, 38% higher (towards the top). Therefore, the best rank usually advertised by a university adequately reflects the collective opinion of experts. We can trust it, but with a discount. With the best rank and proportionality relationship, a university’s consensus rank can be estimated with reasonable accuracy. Our work not only reveals previously unknown patterns in university rankings but also introduces a set of tools that can be readily applied to future studies.

Investigating collaborative and mobility networks: reflections on the core missions of universities

Given the significant role of universities in economic growth and social progress as well as the increasing demand for greater transparency regarding the use of public money, a valid assessment of university performance has become crucial for various stakeholders, including government, industry, funding agencies, and society at large. Contemporary assessments still focus solely on universities’ properties, thereby failing to capture their network relations. To overcome this limitation, this paper proposes a multilayer network-based method to measure the embeddedness of universities in collaboration and mobility networks. This method has several advantages: first, it is relevant for HEIs’ core missions, introducing a new dimension complementary to the existing rankings; second, it is size invariant; and last but not least, it is fully transparent. The proposed multilayer network approach enables the integration of further networks, which creates opportunities for a more comprehensive assessment of universities’ performance in achieving their core missions.

A Comparative Analysis between Global University Rankings and Environmental Sustainability of Universities

Global University Rankings (GURs) intend to measure the performance of universities worldwide. Other rankings have recently appeared that evaluate the creation of environmental policies in universities, e.g., the Universitas Indonesia (UI) GreenMetric. This work aims to analyze the interaction between the Top 500 of such rankings by considering the geographical location of universities and their typologies. A descriptive analysis and a statistical logistical regression analysis were carried out. The former demonstrated that European and North American universities predominated the Top 500 of GURs, while Asian universities did so in the Top 500 of the UI GreenMetric ranking, followed by European universities. Older universities predominated the Top 500 of GURs, while younger ones did so in the Top 500 of the UI GreenMetric ranking. The second analysis demonstrated that although Latin American universities were barely present in the Top 500 of GURs, the probability of them appearing in the Top 500 of the UI GreenMetric ranking was 5-fold. We conclude that a low association exists between universities’ academic performance and their commitment to the natural environment in the heart of their institutions. It would be advisable for GURs to include environmental indicators to promote sustainability at universities and to contribute to climate change.

Ranking Multi-Metric Scientific Achievements Using a Concept of Pareto Optimality

The ranking of multi-metric scientific achievements is a challenging task. For example, the scientific ranking of researchers utilizes two major types of indicators; namely, number of publications and citations. In fact, they focus on how to select proper indicators, considering only one indicator or combination of them. The majority of ranking methods combine several indicators, but these methods are faced with a challenging concern—the assignment of suitable/optimal weights to the targeted indicators. Pareto optimality is defined as a measure of efficiency in the multi-objective optimization which seeks the optimal solutions by considering multiple criteria/objectives simultaneously. The performance of the basic Pareto dominance depth ranking strategy decreases by increasing the number of criteria (generally speaking, when it is more than three criteria). In this paper, a new, modified Pareto dominance depth ranking strategy is proposed which uses some dominance metrics obtained from the basic Pareto dominance depth ranking and some sorted statistical metrics to rank the scientific achievements. It attempts to find the clusters of compared data by using all of indicators simultaneously. Furthermore, we apply the proposed method to address the multi-source ranking resolution problem which is very common these days; for example, there are several world-wide institutions which rank the world’s universities every year, but their rankings are not consistent. As our case studies, the proposed method was used to rank several scientific datasets (i.e., researchers, universities, and countries) for proof of concept.

Sustainability and Indicators of Newly Formed World-Class Universities (NFWCUs) between 2010 and 2018: Empirical Analysis from the Rankings of ARWU, QSWUR and THEWUR

In the 21st century, sustainability and indicators of world-class universities have come within the scope of an academic cottage industry. The complex problem of university sustainability implies a big challenge for countries and educators to implement important strategies in an integrated and comprehensive way. This paper highlights and analyzes the sustainability indicators of universities included as newly formed world-class universities (NFWCUs) in the top 100 from 2010 and 2018. The integration of three global ranking scales—the Academic Ranking of World Universities (ARWU), the Quacquarelli–Symonds World University Ranking (QS) and the Times Higher Education World University Rankings (THEs)—allows us to minimize the impact of the methodology used. This study integrates regression analysis by using statistical grouping, case studies and normative analysis. Our principal findings are as follows: among the commonly ranked top 100 universities in 2018, the ARWU, QS and THE counted 57, compared with 47 in 2010. Thus, comparing 2010 and 2018 shows that 44 of the universities appeared simultaneously in ARWU, QS and THE rankings and maintained a sustainable position in any ranking system in the family of top 100 groups. Three lower-ranked NFWCUs in the hybrid list for 2010 lost their ranking and did not appear in the group of top 100 universities in 2018, which are covered by some catch-up and young universities. The NFWCUs were from US, Australia, China, Singapore, Germany and Belgium. By systematic comparison, the US and UK continued to dominate the stability of NFWCUs in 2010 and 2018. The key sustainability indicators include a high concentration of talent, abundant resources to offer a rich learning environment and conduct advanced research. Generally, the factors were negatively associated with ranking suggesting that a higher score result in top ranking and vice versa. Teaching, research, citation and international outlook were negatively correlated with THE ranking in 2018. Similarly, Alumni and PUB were negatively associated with ARWU ranking in 2018. All factors except international student ratio were significantly correlated in QS ranking either in 2010 or 2018, where negative association was observed. The significant contribution of our study is to highlight that for the sustainability of universities, it is necessary to have an increasing emphasis on the effectiveness and efficiency of government-supported research, stability of investments and more approaches to employ international initiatives. The results also confirm the appropriate governance, developing global students and place emphasis on science and technology as additional factors in the approaches of pathways to NFWCUs, with delivery of outstanding educational programs and comprehensive internationalization as a key indicator for performance improvement and global university ranking systems.