Equivalence between modularity optimization and maximum likelihood methods for community detection

Generalized Markov stability of network communities

We address the problem of community detection in networks by introducing a general definition of Markov stability, based on the difference between the probability fluxes of a Markov chain on the network at different timescales. The specific implementation of the quality function and the resulting optimal community structure thus become dependent both on the type of Markov process and on the specific Markov times considered. For instance, if we use a natural Markov chain dynamics and discount its stationary distribution (that is, we take as reference process the dynamics at infinite time) we obtain the standard formulation of the Markov stability. Notably, the possibility to use finite-time transition probabilities to define the reference process naturally allows detecting communities at different resolutions, without the need to consider a continuous-time Markov chain in the small time limit. The main advantage of our general formulation of Markov stability based on dynamical flows is that we work with lumped Markov chains on network partitions, having the same stationary distribution of the original process. In this way the form of the quality function becomes invariant under partitioning, leading to a self-consistent definition of community structures at different aggregation scales.

Clustering via hypergraph modularity

Despite the fact that many important problems (including clustering) can be described using hypergraphs, theoretical foundations as well as practical algorithms using hypergraphs are not well developed yet. In this paper, we propose a hypergraph modularity function that generalizes its well established and widely used graph counterpart measure of how clustered a network is. In order to define it properly, we generalize the Chung-Lu model for graphs to hypergraphs. We then provide the theoretical foundations to search for an optimal solution with respect to our hypergraph modularity function. A simple heuristic algorithm is described and applied to a few illustrative examples. We show that using a strict version of our proposed modularity function often leads to a solution where a smaller number of hyperedges get cut as compared to optimizing modularity of 2-section graph of a hypergraph.

Computational analysis of size, shape and structure of insect wings

The size, shape and structure of insect wings are intimately linked to their ability to fly. However, there are few systematic studies of the variability of the natural patterns in wing morphology across insects. We have assembled a dataset of 789 insect wings with representatives from 25 families and performed a comprehensive computational analysis of their morphology using topological and geometric notions in terms of (i) wing size and contour shape, (ii) vein topology, and (iii) shape and distribution of wing membrane domains. These morphospaces are complementary to existing methods for quantitatively characterizing wing morphology and are likely to be useful for investigating wing function and evolution. This Methods and Techniques paper is accompanied by a set of computational tools for open use.

This article has an associated First Person interview with the first author of the paper.

Summary: We provide a set of simple quantitative measures to compare morphological variation in size, shape, and structure of insect wings across species, families and orders.

A new algorithm for detecting communities in social networks based on content and structure information

Purpose

The purpose of this paper is to present an algorithm for detecting communities in social networks.

Design/methodology/approach

The majority of existing methods of community detection in social networks are based on structural information, and they neglect the content information. In this paper, the authors propose a novel approach that combines the content and structure information to discover more meaningful communities in social networks. To integrate the content information in the process of community detection, the authors propose to exploit the texts involved in social networks to identify the users’ topics of interest. These topics are detected based on the statistical and semantic measures, which allow us to divide the users into different groups so that each group represents a distinct topic. Then, the authors perform links analysis in each group to discover the users who are highly interconnected (communities).

Findings

To validate the performance of the approach, the authors carried out a set of experiments on four real life data sets, and they compared their method with classical methods that ignore the content information.

Originality/value

The experimental results demonstrate that the quality of community structure is improved when we take into account the content and structure information during the procedure of community detection.

A Regularized Stochastic Block Model for the robust community detection in complex networks

The stochastic block model is able to generate random graphs with different types of network partitions, ranging from the traditional assortative structures to the disassortative structures. Since the stochastic block model does not specify which mixing pattern is desired, the inference algorithms discover the locally most likely nodes’ partition, regardless of its type. Here we introduce a new model constraining nodes’ internal degree ratios in the objective function to guide the inference algorithms to converge to the desired type of structure in the observed network data. We show experimentally that given the regularized model, the inference algorithms, such as Markov chain Monte Carlo, reliably and quickly find the assortative or disassortative structure as directed by the value of a single parameter. In contrast, when the sought-after assortative community structure is not strong in the observed network, the traditional inference algorithms using the degree-corrected stochastic block model tend to converge to undesired disassortative partitions.

Viral networks and detection of potential zoonotic viruses in bats and rodents: A worldwide analysis

Bats and rodents are recognized to host a great diversity of viruses and several important viral zoonoses, but how this viral diversity is structured and how viruses are connected, shared and distributed among host networks is not well understood. To address this gap in knowledge, we compared the associative capacity of the host-virus networks in rodents and bats with the identification of those viruses with zoonotic potential. A virus database, detected by molecular methods, was constructed in the two taxonomic groups. We compiled 5,484 records: 825 in rodents and 4,659 in bats. We identified a total of 173 and 166 viruses, of which 53 and 40 are zoonotic viruses, in rodents and bats, respectively. Based on a network theory, a non-directed bipartite host-virus network was built for each group. Subsequently, the networks were collapsed to represent the connections among hosts and viruses. We identified both discrete and connected communities. We observed a greater degree of connectivity in bat viruses and more discrete communities in rodents. The Coronaviridae recorded in bats have the highest values of degree, betweenness and closeness centralities. In rodents, higher degree positions were distributed homogeneously between viruses and hosts. At least in our database, a higher proportion of rodent viruses were zoonotic. Rodents should thus not be underestimated as important reservoirs of zoonotic disease. We found that viruses were more frequently shared among bats than in rodents. Network theory can reveal some macroecological patterns and identify risks that were previously unrecognized. For example, we found that parvovirus in megabats and Gbagroube virus in rodents may represent a zoonotic risk due to the proximity to humans and other zoonotic viruses. We propose that epidemiological surveillance programmes should consider the connectivity of network actors as a measure of the risks of dispersion and transmission.

Community Detection on Networks with Ricci Flow

Many complex networks in the real world have community structures - groups of well-connected nodes with important functional roles. It has been well recognized that the identification of communities bears numerous practical applications. While existing approaches mainly apply statistical or graph theoretical/combinatorial methods for community detection, in this paper, we present a novel geometric approach which  enables us to borrow powerful classical geometric methods and properties. By considering networks as geometric objects and communities in a network as a geometric decomposition, we apply curvature and discrete Ricci flow, which have been used to decompose smooth manifolds with astonishing successes in mathematics, to break down communities in networks. We  tested our method on networks with ground-truth community structures, and experimentally confirmed the effectiveness of this geometric approach.

Mapping the spectrum of 3D communities in human chromosome conformation capture data

Several experiments show that the three dimensional (3D) organization of chromosomes affects genetic processes such as transcription and gene regulation. To better understand this connection, researchers developed the Hi-C method that is able to detect the pairwise physical contacts of all chromosomal loci. The Hi-C data show that chromosomes are composed of 3D compartments that range over a variety of scales. However, it is challenging to systematically detect these cross-scale structures. Most studies have therefore designed methods for specific scales to study foremost topologically associated domains (TADs) and A/B compartments. To go beyond this limitation, we tailor a network community detection method that finds communities in compact fractal globule polymer systems. Our method allows us to continuously scan through all scales with a single resolution parameter. We found: (i) polymer segments belonging to the same 3D community do not have to be in consecutive order along the polymer chain. In other words, several TADs may belong to the same 3D community. (ii) CTCF proteins-a loop-stabilizing protein that is ascribed a big role in TAD formation-are well correlated with community borders only at one level of organization. (iii) TADs and A/B compartments are traditionally treated as two weakly related 3D structures and detected with different algorithms. With our method, we detect both by simply adjusting the resolution parameter. We therefore argue that they represent two specific levels of a continuous spectrum 3D communities, rather than seeing them as different structural entities.

The emergent integrated network structure of scientific research

Scientific research is often thought of as being conducted by individuals and small teams striving for disciplinary advances. Yet as a whole, this endeavor more closely resembles a complex and integrated system of people, papers, and ideas. Studies of co-authorship and citation networks have revealed important structural properties of researchers and articles, but currently the structure of scientific ideas themselves is not well understood. In this study, we posit that topic networks may be a useful framework for revealing the nature of conceptual relationships. Using this framework, we map the landscape of interconnected research topics covered in the multidisciplinary journal PNAS since 2000, constructing networks in which nodes represent topics of study and edges give the extent to which topics occur in the same papers. The network displays small-world architecture, characterized by regions of dense local connectivity with sparse connectivity between them. In this network, dense local connectivity additionally gives rise to distinct clusters of related topics. Yet notably, these clusters tend not to align with assigned article classifications, and instead contain topics from various disciplines. Using a temporal graph, we find that small-worldness has increased over time, suggesting growing efficiency and integration of ideas. Finally, we define two measures of interdisciplinarity, one of which is found to be positively associated with PNAS's impact factor. Broadly, this work suggests that complex and dynamic patterns of knowledge emerge from scientific research, and that structures reflecting intellectual integration may be beneficial for obtaining scientific insight.

Stochastic block models: A comparison of variants and inference methods

Finding communities in complex networks is a challenging task and one promising approach is the Stochastic Block Model (SBM). But the influences from various fields led to a diversity of variants and inference methods. Therefore, a comparison of the existing techniques and an independent analysis of their capabilities and weaknesses is needed. As a first step, we review the development of different SBM variants such as the degree-corrected SBM of Karrer and Newman or Peixoto's hierarchical SBM. Beside stating all these variants in a uniform notation, we show the reasons for their development. Knowing the variants, we discuss a variety of approaches to infer the optimal partition like the Metropolis-Hastings algorithm. We perform our analysis based on our extension of the Girvan-Newman test and the Lancichinetti-Fortunato-Radicchi benchmark as well as a selection of some real world networks. Using these results, we give some guidance to the challenging task of selecting an inference method and SBM variant. In addition, we give a simple heuristic to determine the number of steps for the Metropolis-Hastings algorithms that lack a usual stop criterion. With our comparison, we hope to guide researches in the field of SBM and highlight the problem of existing techniques to focus future research. Finally, by making our code freely available, we want to promote a faster development, integration and exchange of new ideas.

Counting the number of metastable states in the modularity landscape: Algorithmic detectability limit of greedy algorithms in community detection

Modularity maximization using greedy algorithms continues to be a popular approach toward community detection in graphs, even after various better forming algorithms have been proposed. Apart from its clear mechanism and ease of implementation, this approach is persistently popular because, presumably, its risk of algorithmic failure is not well understood. This Rapid Communication provides insight into this issue by estimating the algorithmic performance limit of the stochastic block model inference using modularity maximization. This is achieved by counting the number of metastable states under a local update rule. Our results offer a quantitative insight into the level of sparsity at which a greedy algorithm typically fails.

A Scheme to Design Community Detection Algorithms in Various Networks

Network structures, consisting of nodes and edges, have applications in almost all subjects. A set of nodes is called a community if the nodes have strong interrelations. Industries (including cell phone carriers and online social media companies) need community structures to allocate network resources and provide proper and accurate services. However, most detection algorithms are derived independently, which is arduous and even unnecessary. Although recent research shows that a general detection method that serves all purposes does not exist, we believe that there is some general procedure of deriving detection algorithms. In this paper, we represent such a general scheme. We mainly focus on two types of networks: transmission networks and similarity networks. We reduce them to a unified graph model, based on which we propose a method to define and detect community structures. Finally, we also give a demonstration to show how our design scheme works.

Disease networks identify specific conditions and pleiotropy influencing multimorbidity in the general population

Multimorbidity is an emerging topic in public health policy because of its increasing prevalence and socio-economic impact. However, the age- and gender-dependent trends of disease associations at fine resolution, and the underlying genetic factors, remain incompletely understood. Here, by analyzing disease networks from electronic medical records of primary health care, we identify key conditions and shared genetic factors influencing multimorbidity. Three types of diseases are outlined: "central", which include chronic and non-chronic conditions, have higher cumulative risks of disease associations; "community roots" have lower cumulative risks, but inform on continuing clustered disease associations with age; and "seeds of bursts", which most are chronic, reveal outbreaks of disease associations leading to multimorbidity. The diseases with a major impact on multimorbidity are caused by genes that occupy central positions in the network of human disease genes. Alteration of lipid metabolism connects breast cancer, diabetic neuropathy and nutritional anemia. Evaluation of key disease associations by a genome-wide association study identifies shared genetic factors and further supports causal commonalities between nervous system diseases and nutritional anemias. This study also reveals many shared genetic signals with other diseases. Collectively, our results depict novel population-based multimorbidity patterns, identify key diseases within them, and highlight pleiotropy influencing multimorbidity.

Characterizing the Analogy Between Hyperbolic Embedding and Community Structure of Complex Networks

We show that the community structure of a network can be used as a coarse version of its embedding in a hidden space with hyperbolic geometry. The finding emerges from a systematic analysis of several real-world and synthetic networks. We take advantage of the analogy for reinterpreting results originally obtained through network hyperbolic embedding in terms of community structure only. First, we show that the robustness of a multiplex network can be controlled by tuning the correlation between the community structures across different layers. Second, we deploy an efficient greedy protocol for network navigability that makes use of routing tables based on community structure.

Community Detection Based on Differential Evolution Using Modularity Density

Currently, many community detection methods are proposed in the network science field. However, most contemporary methods only employ modularity to detect communities, which may not be adequate to represent the real community structure of networks for its resolution limit problem. In order to resolve this problem, we put forward a new community detection approach based on a differential evolution algorithm (CDDEA), taking into account modularity density as an optimized function. In the CDDEA, a new tuning parameter is used to recognize different communities. The experimental results on synthetic and real-world networks show that the proposed algorithm provides an effective method in discovering community structure in complex networks.

Weighted Stochastic Block Models of the Human Connectome across the Life Span

The human brain can be described as a complex network of anatomical connections between distinct areas, referred to as the human connectome. Fundamental characteristics of connectome organization can be revealed using the tools of network science and graph theory. Of particular interest is the network's community structure, commonly identified by modularity maximization, where communities are conceptualized as densely intra-connected and sparsely inter-connected. Here we adopt a generative modeling approach called weighted stochastic block models (WSBM) that can describe a wider range of community structure topologies by explicitly considering patterned interactions between communities. We apply this method to the study of changes in the human connectome that occur across the life span (between 6-85 years old). We find that WSBM communities exhibit greater hemispheric symmetry and are spatially less compact than those derived from modularity maximization. We identify several network blocks that exhibit significant linear and non-linear changes across age, with the most significant changes involving subregions of prefrontal cortex. Overall, we show that the WSBM generative modeling approach can be an effective tool for describing types of community structure in brain networks that go beyond modularity.

Statistical inference approach to structural reconstruction of complex networks from binary time series

Complex networks hosting binary-state dynamics arise in a variety of contexts. In spite of previous works, to fully reconstruct the network structure from observed binary data remains challenging. We articulate a statistical inference based approach to this problem. In particular, exploiting the expectation-maximization (EM) algorithm, we develop a method to ascertain the neighbors of any node in the network based solely on binary data, thereby recovering the full topology of the network. A key ingredient of our method is the maximum-likelihood estimation of the probabilities associated with actual or nonexistent links, and we show that the EM algorithm can distinguish the two kinds of probability values without any ambiguity, insofar as the length of the available binary time series is reasonably long. Our method does not require any a priori knowledge of the detailed dynamical processes, is parameter-free, and is capable of accurate reconstruction even in the presence of noise. We demonstrate the method using combinations of distinct types of binary dynamical processes and network topologies, and provide a physical understanding of the underlying reconstruction mechanism. Our statistical inference based reconstruction method contributes an additional piece to the rapidly expanding "toolbox" of data based reverse engineering of complex networked systems.

The architecture of an empirical genotype-phenotype map

Recent advances in high-throughput technologies are bringing the study of empirical genotype-phenotype (GP) maps to the fore. Here, we use data from protein-binding microarrays to study an empirical GP map of transcription factor (TF) -binding preferences. In this map, each genotype is a DNA sequence. The phenotype of this DNA sequence is its ability to bind one or more TFs. We study this GP map using genotype networks, in which nodes represent genotypes with the same phenotype, and edges connect nodes if their genotypes differ by a single small mutation. We describe the structure and arrangement of genotype networks within the space of all possible binding sites for 525 TFs from three eukaryotic species encompassing three kingdoms of life (animal, plant, and fungi). We thus provide a high-resolution depiction of the architecture of an empirical GP map. Among a number of findings, we show that these genotype networks are "small-world" and assortative, and that they ubiquitously overlap and interface with one another. We also use polymorphism data from Arabidopsis thaliana to show how genotype network structure influences the evolution of TF-binding sites in vivo. We discuss our findings in the context of regulatory evolution.

Multiresolution Consensus Clustering in Networks

Networks often exhibit structure at disparate scales. We propose a method for identifying community structure at different scales based on multiresolution modularity and consensus clustering. Our contribution consists of two parts. First, we propose a strategy for sampling the entire range of possible resolutions for the multiresolution modularity quality function. Our approach is directly based on the properties of modularity and, in particular, provides a natural way of avoiding the need to increase the resolution parameter by several orders of magnitude to break a few remaining small communities, necessitating the introduction of ad-hoc limits to the resolution range with standard sampling approaches. Second, we propose a hierarchical consensus clustering procedure, based on a modified modularity, that allows one to construct a hierarchical consensus structure given a set of input partitions. While here we are interested in its application to partitions sampled using multiresolution modularity, this consensus clustering procedure can be applied to the output of any clustering algorithm. As such, we see many potential applications of the individual parts of our multiresolution consensus clustering procedure in addition to using the procedure itself to identify hierarchical structure in networks.

Finding multiple core-periphery pairs in networks

With a core-periphery structure of networks, core nodes are densely interconnected, peripheral nodes are connected to core nodes to different extents, and peripheral nodes are sparsely interconnected. Core-periphery structure composed of a single core and periphery has been identified for various networks. However, analogous to the observation that many empirical networks are composed of densely interconnected groups of nodes, i.e., communities, a network may be better regarded as a collection of multiple cores and peripheries. We propose a scalable algorithm to detect multiple nonoverlapping groups of core-periphery structure in a network. We illustrate our algorithm using synthesized and empirical networks. For example, we find distinct core-periphery pairs with different political leanings in a network of political blogs and separation between international and domestic subnetworks of airports in some single countries in a worldwide airport network.

Versatility of nodal affiliation to communities

Graph theoretical analysis of the community structure of networks attempts to identify the communities (or modules) to which each node affiliates. However, this is in most cases an ill-posed problem, as the affiliation of a node to a single community is often ambiguous. Previous solutions have attempted to identify all of the communities to which each node affiliates. Instead of taking this approach, we introduce versatility, V, as a novel metric of nodal affiliation: V ≈ 0 means that a node is consistently assigned to a specific community; V >> 0 means it is inconsistently assigned to different communities. Versatility works in conjunction with existing community detection algorithms, and it satisfies many theoretically desirable properties in idealised networks designed to maximise ambiguity of modular decomposition. The local minima of global mean versatility identified the resolution parameters of a hierarchical community detection algorithm that least ambiguously decomposed the community structure of a social (karate club) network and the mouse brain connectome. Our results suggest that nodal versatility is useful in quantifying the inherent ambiguity of modular decomposition.

Spectral partitioning in equitable graphs

Graph partitioning problems emerge in a wide variety of complex systems, ranging from biology to finance, but can be rigorously analyzed and solved only for a few graph ensembles. Here, an ensemble of equitable graphs, i.e., random graphs with a block-regular structure, is studied, for which analytical results can be obtained. In particular, the spectral density of this ensemble is computed exactly for a modular and bipartite structure. Kesten-McKay's law for random regular graphs is found analytically to apply also for modular and bipartite structures when blocks are homogeneous. An exact solution to graph partitioning for two equal-sized communities is proposed and verified numerically, and a conjecture on the absence of an efficient recovery detectability transition in equitable graphs is suggested. A final discussion summarizes results and outlines their relevance for the solution of graph partitioning problems in other graph ensembles, in particular for the study of detectability thresholds and resolution limits in stochastic block models.

Finite-size analysis of the detectability limit of the stochastic block model

It has been shown in recent years that the stochastic block model is sometimes undetectable in the sparse limit, i.e., that no algorithm can identify a partition correlated with the partition used to generate an instance, if the instance is sparse enough and infinitely large. In this contribution, we treat the finite case explicitly, using arguments drawn from information theory and statistics. We give a necessary condition for finite-size detectability in the general SBM. We then distinguish the concept of average detectability from the concept of instance-by-instance detectability and give explicit formulas for both definitions. Using these formulas, we prove that there exist large equivalence classes of parameters, where widely different network ensembles are equally detectable with respect to our definitions of detectability. In an extensive case study, we investigate the finite-size detectability of a simplified variant of the SBM, which encompasses a number of important models as special cases. These models include the symmetric SBM, the planted coloring model, and more exotic SBMs not previously studied. We conclude with three appendices, where we study the interplay of noise and detectability, establish a connection between our information-theoretic approach and random matrix theory, and provide proofs of some of the more technical results.

The many facets of community detection in complex networks

Community detection, the decomposition of a graph into essential building blocks, has been a core research topic in network science over the past years. Since a precise notion of what constitutes a community has remained evasive, community detection algorithms have often been compared on benchmark graphs with a particular form of assortative community structure and classified based on the mathematical techniques they employ. However, this comparison can be misleading because apparent similarities in their mathematical machinery can disguise different goals and reasons for why we want to employ community detection in the first place. Here we provide a focused review of these different motivations that underpin community detection. This problem-driven classification is useful in applied network science, where it is important to select an appropriate algorithm for the given purpose. Moreover, highlighting the different facets of community detection also delineates the many lines of research and points out open directions and avenues for future research.