Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

The impact of patentees assessment based on the heterogeneous patent innovation network

As a vital factor in technological innovation, patentee plays a significant role in the process of scientific and technological innovation, researching patentee has attracted the attention of experts and scholars. Previously, scholars have mainly quantified patent indicators or constructed homogeneous information networks to analyze patentees, but these methods cannot objectively measure the impact of patentees. Therefore, this study proposes a novel approach to assessing patentee impact based on a heterogeneous information network. The proposed method distinguishes the weight of different types of nodes using a weighted mechanism and extracts three types of fine-grained characteristics of network nodes. This approach results in the construction of a heterogeneous patent innovation network and the development of a new patentee impact assessment algorithm called CWAPN. Using Chinese green patents in the field of energy conservation and environmental protection as an example, experimental results show that the CWAPN algorithm can effectively assess the impact of patentees. Thereby identifying patentees who have made outstanding contributions to sustainable development in China.

Impact of medical technologies may be predicted using constructed graph bibliometrics

Scientific research is driven by allocation of funding to different research projects based in part on the predicted scientific impact of the work. Data-driven algorithms can inform decision-making of scarce funding resources by identifying likely high-impact studies using bibliometrics. Compared to standardized citation-based metrics alone, we utilize a machine learning pipeline that analyzes high-dimensional relationships among a range of bibliometric features to improve the accuracy of predicting high-impact research. Random forest classification models were trained using 28 bibliometric features calculated from a dataset of 1,485,958 publications in medicine to retrospectively predict whether a publication would become high-impact. For each random forest model, the balanced accuracy score was above 0.95 and the area under the receiver operating characteristic curve was above 0.99. The high performance of high impact research prediction using our proposed models show that machine learning technologies are promising algorithms that can support funding decision-making for medical research.

AIRank: An algorithm on evaluating the academic influence of papers based on heterogeneous academic network

Evaluation of papers’ academic influence is a hot issue in the field of scientific research management. Academic big data provides a data treasure with the coexistence of different types of academic entities, which can be used to evaluate academic influence from a more macro and comprehensive perspective. Based on academic big data, a heterogeneous academic network composed of links within and between three types of academic entities (authors, papers and venues) is constructed. In addition, a new academic influence ranking algorithm, AIRank, is proposed to evaluate papers’ academic influence. Different from the existing academic influence ranking algorithms, AIRank has made innovations in the following two aspects. (1) AIRank distinguishes the influence transmission intensity between different node pairs. Different from the strategy of evenly distributing influence among different node pairs, AIRank quantifies the intensity of influence transmission between node pairs based on investigating the citation emotional attribute, semantic similarity and academic quality differences between node pairs. Based on the intensity characteristics, AIRank realises the distribution and transmission of influence among different node pairs. (2) AIRank incorporates the influence transmission from heterogeneous neighbours in evaluating papers’ influence. According to the academic influence of author nodes and venue nodes, AIRank fine-tunes the iteration formula of paper influence to obtain the ranking of papers under the joint influence of homogeneous and heterogeneous neighbours. Experimental results show that, compared with the ranking results based on citation frequency and PageRank algorithm, AIRank algorithm can produce more differentiated and reasonable academic influence ranking results.

Mining Algorithm of Relatively Important Nodes Based on Edge Importance Greedy Strategy

Relatively important node mining has always been an essential research topic in complex networks. Existing relatively important node mining algorithms suffer from high time complexity and poor accuracy. Therefore, this paper proposes an algorithm for mining relatively important nodes based on the edge importance greedy strategy (EG). This method considers the importance of the edge to represent the degree of association between two connected nodes. Therefore, the greater the value of the connection between a node and a known important node, the more likely it is to be an important node. If the importance of the edges in an undirected network is measured, a greedy strategy can find important nodes. Compared with other relatively important node mining methods on real network data sets, such as SARS and 9/11, the experimental results show that the EG algorithm excels in both accuracy and applicability, which makes it a competitive algorithm in the mining of important nodes in a network.

Learning on knowledge graph dynamics provides an early warning of impactful research

The scientific ecosystem relies on citation-based metrics that provide only imperfect, inconsistent and easily manipulated measures of research quality. Here we describe DELPHI (Dynamic Early-warning by Learning to Predict High Impact), a framework that provides an early-warning signal for 'impactful' research by autonomously learning high-dimensional relationships among features calculated across time from the scientific literature. We prototype this framework and deduce its performance and scaling properties on time-structured publication graphs from 1980 to 2019 drawn from 42 biotechnology-related journals, including over 7.8 million individual nodes, 201 million relationships and 3.8 billion calculated metrics. We demonstrate the framework's performance by correctly identifying 19/20 seminal biotechnologies from 1980 to 2014 via a blinded retrospective study and provide 50 research papers from 2018 that DELPHI predicts will be in the top 5% of time-rescaled node centrality in the future. We propose DELPHI as a tool to aid in the construction of diversified, impact-optimized funding portfolios.

Characterizing research leadership on geographically weighted collaboration network

Research collaborations, especially long-distance and international collaborations, have become increasingly prevalent worldwide. Recent studies highlighted the significant role of research leadership in collaborations. However, existing measures of the research leadership do not take into account the intensity of leadership in the co-authorship network. More importantly, the spatial features, which influence the collaboration patterns and research outcomes, have not been incorporated in measuring the research leadership. To fill the gap, we construct an institution-level weighted co-authorship network that integrates two types of weight on the edges: the intensity of collaborations and the spatial score (the geographical distance adjusted by the cross-linguistic-border nature). Based on this network, we propose a novel metric, namely the spatial research leadership rank, to identify the leading institutions while considering both the collaboration intensity and the spatial features. The leadership of an institution is measured by the following three criteria: (a) the institution frequently plays the corresponding rule in papers with other institutions; (b) the institution frequently plays the corresponding rule in longer distance and even cross-linguistic-border collaborations; (c) the participating institutions led by the institution have high leadership status themselves. Harnessing a dataset of 323,146 journal publications in pharmaceutical sciences during 2010-2018, we perform a comprehensive analysis of the geographical distribution and dynamic patterns of research leadership flows at the institution level. The results demonstrate that the SpatialLeaderRank outperforms baseline metrics in predicting the scholarly impact of institutions. And the result remains robust in the field of Information Science and Library Science.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s11192-021-03943-w.