We further note that the minimum cut set in the transformed network must include either all or none of the parallel edges between any adjacent pair of nodes; there is no point cutting one such edge unless you cut all the others as well. With this constraint, there is a one-to-one correspondence between cut sets on the original network and the transformed network, with corresponding cut sets necessarily having the same total weight. Hence the minimum cut set on the weighted network has the same weight as the minimum cut set on the transformed network and so the minimum cut and maximum flow are equal on the original network.
This demonstrates the theorem for the case of integer edge weights. It can be extended to the non-integer case simply by making the units in which we measure the weights smaller. In the limit where the units become arbitrarily For an alternate, first-
principles proof not using integer edge weights see, for instance, Ahujaet al.[9].
small, any weight can be represented as an integer number of units and the argument above can be applied. Hence the max-flow/min-cut theorem must be generally true for any set of weights.
There exist efficient computer algorithms for calculating maximum flows on weighted networks, so the max-flow/min-cut theorem allows us to calculate minimum cuts efficiently also, and this is now the standard way of performing such calculations.21
6.14 | T
he graph
Laplacian
thing would be
Li j kiδi j−Ai j, (6.28) whereAi jis an element of the adjacency matrix andδi j is the Kronecker delta, which is 1 ifi jand 0 otherwise. Alternatively, we can writeLin matrix form as
LD−A, (6.29)
whereDis the diagonal matrix with the node degrees along its diagonal:
D
©
«
k1 0 0 · · ·
0 k2 0 · · ·
0 0 k3 · · ·
... ... ... ...
ª
®
®
®
®
¬
. (6.30)
All of these are equivalent definitions of the graph Laplacian.
One can also write a graph Laplacian for weighted networks: one simply replaces the adjacency matrix with the weighted adjacency matrix of Section 6.3 (which has the weights in the matrix elements) and the degreekiof a node by the sumÍ
jAi jof the relevant matrix elements. One can also treat multigraphs in the same way. There is, however, no natural extension of the graph Laplacian to networks with self-edges or, more importantly, to directed networks. The Laplacian is only useful for the undirected case.
The graph Laplacian crops up in a surprisingly diverse set of situations, including in the theory of random walks on networks, dynamical systems, diffusion, resistor networks, graph visualization, and graph partitioning. In the following sections we look briefly at some of these applications.
6.14.1 Graph partitioning
Graph partitioning is the task of dividing the nodes of a network into a set of groups of given sizes so as to minimize the number of edges running between the groups. It arises, for instance, in parallel computing, where you want to divide up a calculation into smaller sub-calculations that can be assigned to several different computers or CPUs, while minimizing the amount of data that will have to be sent back and forth between the CPUs (since transmitting data is usually a relatively cumbersome process that can slow down the whole computation).
Consider the simplest version of graph partitioning, the division of the nodes of a network into just two groups, which we will call group 1 and group 2. The number of edgesRrunning between the two groups, also called
thecut size, is given by
R 12 Õ
i,jin different
groups
Ai j, (6.31)
where the factor of12compensates for the fact that every pair of nodes is counted twice in the sum. (For instance, we count nodes 1 and 2 separately from nodes 2 and 1.)
We define a set of quantitiessi, one for each nodei, which represent the division of the network thus:22
si +1
if nodeibelongs to group 1,
−1 if nodeibelongs to group 2. (6.32) Then
1
2(1−sisj)
1 ifiand jare in different groups,
0 ifiand jare in the same group, (6.33) which allows us to rewrite Eq. (6.31) as
R 14Õ
i j
Ai j(1−sisj), (6.34)
with the sum now over all values ofiandj. The first term in the sum is Õ
i j
Ai j Õ
i
ki Õ
i
kis2i Õ
i j
kiδi jsisj, (6.35)
whereki is the degree of nodeias previously,δi j is the Kronecker delta, and we have made use of the fact thatÍ
jAi j ki(see Eq. (6.12)) ands2i 1 (since si ±1). Substituting back into Eq. (6.34) we then find that
R 14Õ
i j
(kiδi j−Ai j)sisj 14Õ
i j
Li jsisj, (6.36)
whereLi j kiδi j−Ai jis thei jth element of the graph Laplacian matrix—see Eq. (6.28).
Equation (6.36) can be written in matrix form as
R 14sTLs, (6.37)
22A physicist would call the variables si “Ising spins,” and indeed the graph partitioning problem is equivalent to finding the ground state of a certain type of Ising model in which the spins live on the nodes of the network.
6.14 | T
he graph
Laplacian
where s is the vector with elements si. This expression gives us a matrix formulation of the graph partitioning problem. The matrix L specifies the structure of our network, the vectorsdefines a division of that network into groups, and our goal is to find the vectorsthat minimizes the cut size (6.37) for given L. This matrix formulation leads directly to one of the standard computational methods for solving the graph partitioning problem, spectral partitioning, which makes use of the eigenvectors of the graph Laplacian to rapidly find good divisions of the network [177, 391].
6.14.2 Network visualization
We have seen many pictures of networks in this book. Some of them, like the picture of the Internet on page 2 or the picture of a food web on page 6, depict large and complicated networks that would be difficult to make sense of if the pictures were not carefully laid out to make the network structure as clear as
possible. The generation of network visualizations like these is the domain of Software packages for network visualization and analysis are discussed in more detail in Chapter 8.
specialized software packages, whose workings are outside the scope of this book. However, it is interesting to ask, broadly, what is it that characterizes a good visualization of a network?
One answer is that a good visualization is one where the lengths of most edges in the network, as drawn on the page, are short. Consider, for instance, Fig. 6.19, which shows two different pictures of the same network. In Fig. 6.19a the nodes are placed at random on the page, which means that some edges are short but many are relatively long—there are many edges that run clear across the picture from one side to the other. The net result is that the edges are a mess, crossing over one another, getting in each other’s way, and generally making it hard to see which nodes are connected to which. In Fig. 6.19b, on the other hand, the network is laid out so that connected pairs of nodes are (by and large) placed close together and the lengths of the edges are short. This results in a much clearer picture that makes the network structure easier to see.
Suppose then that we have an undirected, unweighted network that we want to lay out on the page. Real network images are two-dimensional but for the sake of simplicity let us consider a one-dimensional case for now, so that
the position of nodeiin our layout is a simple scalarxi. Our goal is to choose The problem of creating a good visualization of a net- work is closely related to the theory of graph em- beddings and latent spaces, which we examine in Sec- tion 14.7.4.
the positions so as to minimize the lengths of edges, which we could do in various ways, but the standard approach is to minimize the sum of the squares of the lengths as follows.
The distance between nodesiand jin our simple one-dimensional model is|xi−xj|and the squared distance is(xi−xj)2. The sum∆2 of the squared
(a) (b)
Figure 6.19: Two visualizations of the same network.In (a) nodes are placed randomly on the page, while in (b) nodes are placed using a network layout algorithm that tries to put connected nodes close to one another, meaning that most edges are short.
distances for all node pairs connected by an edge is then
∆2 12Õ
i j
Ai j(xi−xj)2, (6.38) where the matrix elementAi j ensures that only connected pairs are counted, and the extra factor of 12 compensates for the fact that every pair of nodes appears twice in the sum.
Expanding this expression, we have
∆2 12Õ
i j
Ai j x2i −2xixj+x2j 12
Õ
i
kix2i −2Õ
i j
Ai jxixj+Õ
j
kjx2j
Õ
i j
kiδi j−Ai j
xixjÕ
i j
Li jxixj, (6.39)
whereLi jis an element of the graph Laplacian again, and we have made use of the fact thatÍ
jAi j kiin the second equality (see Eq. (6.12)).
Equation (6.39) can be written in matrix notation as
∆2xTLx, (6.40)
6.14 | T
he graph
Laplacian
wherexis the vector with elementsxi. This expression is similar to Eq. (6.37) and, like that equation, it can form the basis for new computer algorithms, in this case algorithms for generating clear visualizations of networks using the eigenvectors of the graph Laplacian [274]. It also tells us that not all networks can be visualized equally clearly. Starting from Eq. (6.40) we can derive a lower bound on the mean-square length of an edge and hence show that in order for a network to have a good visualization where most edges are short it must have low “algebraic connectivity,” meaning that the gap between the smallest and second smallest eigenvalues of the graph Laplacian must be small—see Section 6.14.5. Thus, merely by inspecting the properties of the Laplacian for a particular network we can say whether it will even be possible to make a good visualization. For some networks, no matter hard we try, we will never be able to make a clear picture because there is no layout in which the average length of edges is small.
6.14.3 Random walks
Another context in which the graph Laplacian arises is in the study of random walks on networks. Arandom walkis a walk across a network created by taking repeated random steps. Starting at any initial node, we choose uniformly at random among the edges attached to that node, move along the chosen edge to the node at its other end, and repeat the process. Random walks are allowed to visit the same node more than once, go along the same edge more than once, or backtrack along an edge just traversed. (Self-avoiding random walks, which do none of these things, are also studied sometimes, but we will not discuss them here.) Random walks arise, for instance, in the random-walk sampling method for social networks discussed in Section 4.7 and in the random-walk betweenness measure of Section 7.1.7.
Consider a random walk that starts at a specified node and takest steps.
Letpi(t)be the probability that the walk is at nodeiat timet. If the walk is at nodejat timet−1, the probability of taking a step along any particular one of thekjedges attached to jis 1/kj, so on an undirected network the probability of being at nodeion the next step is given by
pi(t)Õ
j
Ai j
kj
pj(t−1), (6.41)
orp(t)AD−1p(t−1)in matrix form, wherepis the vector with elementspi
and, as before,Dis the diagonal matrix with the degrees of the nodes down its diagonal, as defined in Eq. (6.30).
In the limit of long time the probability distribution over nodes is given by (6.41) withtset to infinity: pi(∞)Í
jAi jpj(∞)/kj, or in matrix form:
pAD−1p, (6.42)
wherepis shorthand forp(∞). Rearranging, this can also be written as (I−AD−1)p(D−A)D−1pLD−1p0. (6.43) Thus D−1p is (any multiple of) an eigenvector of the Laplacian with eigen- value 0.
On a connected network—one with only a single component—we will see in Section 6.14.5 that there is only one eigenvector of the Laplacian that has eigenvalue zero, the vector 1 (1,1,1, . . .) whose elements are all 1. Thus D−1p a1, where ais a constant, or equivalentlyp aD1, so thatpi aki. Thus, on a connected network the probability that a random walk will be found at nodeiin the limit of long time is simply proportional to the degree of that node. If we choose the value ofaso that the probabilitiespisum to one, we get
pi ki Í
jkj ki
2m, (6.44)
where we have made use of Eq. (6.13).
We employed this result previously in Section 4.7 in our analysis of the random-walk sampling method for social networks. The basic insight behind the result is that nodes with high degree are more likely to be visited by a random walk simply because there are more ways of reaching them.
A further corollary is that in the limit of long time the probabilityP(i→ j) of walking along an edge fromitoj on any particular step of a random walk is equal to the probability pi of being at node i in the first place times the probability 1/kiof walking along that particular edge:
P(i→ j) ki
2m× 1 ki 1
2m. (6.45)
In other words, on any given step a random walk is equally likely to traverse every edge.
6.14.4 Resistor networks
As a further example of the application of the graph Laplacian, consider a network of resistors, one of the simplest examples of an electrical network.
Suppose we have a network in which the edges are identical resistors of resis- tanceRand the nodes are junctions between resistors, as shown in Fig. 6.20,
6.14 | T
he graph
Laplacian
s t
I
Figure 6.20: A resistor network with an applied voltage. In this network the edges are resistors and the nodes are electrical junctions between them. A voltage is applied between nodessandt, generating a total currentI.
and suppose we apply a voltage between two nodes sandt such that a total currentIflows fromstotthrough the network.
One basic question we could ask about such a network is what the voltage is at any given node. The current flow in the network obeys Kirchhoff’s current law, which is essentially a statement that electricity is conserved, so that the net current flowing in or out of any node is zero. LetVibe the voltage at node i, measured relative to any convenient reference potential. Then Kirchhoff’s law says that
Õ
j
Ai j
Vi−Vj
R −Ii 0, (6.46)
where Ii represents any current injected into node i by an external current source. In our case this external current is non-zero only for the two nodes s andtconnected to the external voltage:
Ii
+I foris,
−I forit, 0 otherwise.
(6.47)
(In theory there’s no reason why one could not impose more complex cur- rent source arrangements by applying additional voltages to the network and making more elements Ii non-zero, but let us stick to our simple case in this discussion.)
Noting thatÍ
jAi j ki, Eq. (6.46) can also be written askiVi−Í
jAi jVjRIi
or Õ
j
(δi jki−Ai j)Vj RIi, (6.48) which in matrix form is
LVRI, (6.49)
whereLis once again the graph Laplacian. This equation is a kind of matrix version of the standard Ohm’s lawV RIfor a single resistor, and by solving it forVwe can calculate the voltages at every node in the network.
Calculating the behavior of a resistor network might seem like a problem of rather narrow interest, but in fact the connection between the Laplacian and resistor networks has an important and perhaps surprising practical applica- tion. It is the basis for the most widely used technique forgraph sparsification, in which one aims to remove edges from a network while keeping other properties of the network the same [49,435]. Graph sparsification forms the foundation for a range of modern numerical methods for solving large systems of simultan- eous linear equations. By representing the equations in the form of a resistor network (in effect the reverse of the operations above, where we took a resis- tor network and represented it by a set of equations), then sparsifying that network while keeping its electrical properties the same, we can enormously reduce the complexity of the problem, allowing us to solve in seconds systems of equations that previously might have taken hours. Graph sparsification and its application in equation solving is just one example of the many important technological uses of network theory.
6.14.5 Properties of the graphLaplacian
The graph Laplacian has a number of specific properties that are important in many calculations. For instance, it has the property that every row of the matrix sums to zero:
Õ
j
Li jÕ
j
(kiδi j−Ai j)ki−ki 0, (6.50)
where we have made use of the fact thatÍ
jAi j ki—see Eq. (6.12). Similarly every column of the matrix also sums to zero.
Of particular interest are the eigenvalues of the graph Laplacian. Since the Laplacian is a real symmetric matrix, it necessarily has real eigenvalues.
But we can say more than this: all the eigenvalues of the Laplacian are also non-negative.
6.14 | T
he graph
Laplacian
Letλbe any eigenvalue of the graph Laplacian and letvbe the correspond- ing eigenvector, unit normalized so thatvTv1. ThenLvλvand
vTLvλvTvλ. (6.51)
Following the same line of argument we used in Eqs. (6.38) to (6.40) we can write
Õ
i j
Ai j(vi−vj)2 Õ
i j
Ai j v2i −2vivj+v2j Õ
i
kiv2i −2Õ
i j
Ai jvivj+Õ
j
kjv2j 2Õ
i j
kiδi j−Ai j
vivj2Õ
i j
Li jvivj 2vTLv. (6.52)
And combining (6.51) and (6.52) we then get λ 12Õ
i j
Ai j(vi−vj)2 ≥0. (6.53)
Thus all eigenvalues of the Laplacian are non-negative.
While the eigenvalues cannot be negative, however, they can be zero, and in fact the Laplacian always has at least one zero eigenvalue. As we have seen, every row of the matrix sums to zero, which means that the vector1 (1,1,1, . . .) is always an eigenvector of the Laplacian with eigenvalue zero:
L10. (It is not a properly normalized eigenvector. The properly normalized vector would be(1,1,1, . . .)/√
n.) Since there are no negative eigenvalues, this is the lowest of the eigenvalues of the Laplacian.
The presence of a zero eigenvalue implies, among other things, that the Laplacian has no inverse: the determinant of a matrix is the product of its eigenvalues, and hence the determinant of the Laplacian is always zero, so the matrix is singular.
The Laplacian can have more than one zero eigenvalue. Consider, for in- stance, a network that is divided intocdifferent components of sizesn1, . . . ,nc
and let us number the nodes of the network so that the firstn1nodes are those
of the first component, the nextn2 are those of the second component, and so See the discussion of block diagonal matrices in Sec- tion 6.12.
forth. With this choice the Laplacian of the network is block diagonal, looking something like this:
L
©
«
0 · · ·
0 · · ·
... ... ...
ª
®
®
®
®
®
®
®
®
¬
. (6.54)
What is more, each individual block in the matrix is itself the Laplacian of the corresponding component: it has the degrees of the nodes in that component along its diagonal and −1 in each position corresponding to an edge. This implies that each block has its own eigenvector (1,1,1, . . .) with eigenvalue zero, which in turn tells us that there must be at leastcdifferent (linearly inde- pendent) vectors that are eigenvectors of the full LaplacianLwith eigenvalue zero: the vectors that have ones in the positions corresponding to the nodes in a single component and zeros everywhere else. For instance, the vector
v(1,1,1, . . .
| {z } n1ones
, 0,0,0, . . .
| {z } zeros
), (6.55)
is an eigenvector with eigenvalue zero. Thus, in a network withccomponents there are always at leastczero eigenvalues.
Conversely, one can also show that if the network has only one component then the graph Laplacian has only a single zero eigenvalue. To see this, consider an eigenvector v with eigenvalue zero. Equation (6.53) tells us that for this vectorÍ
i jAi j(vi−vj)2 0, which can only be true ifvi vjat opposite ends of every edge. If the network has only one component, however, so that we can get from any node to any other by walking along a suitable sequence of edges, this implies thatvi must have the same value at every node, in which casev is just a multiple of the vector1. In other words, in a network with only one component there is only one eigenvector with eigenvalue zero, the vector1(or multiples of it). All other eigenvectors must have non-zero eigenvalues.
To put this another way, if a network has only one component then the second smallest eigenvalue will be non-zero. At the same time, as we have said, a network with more than one component will have more than one zero eigenvalue, meaning that the second smallest one is zero. Thus, the second smallest eigenvalue is non-zero if and only if the network is connected—if it consists of a single component. The second smallest eigenvalue of the Laplacian is called thealgebraic connectivityof the network or thespectral gap. It plays an important role in a number of areas of network theory.