RANDOM NODE-ASYNCHRONOUS UPDATES ON GRAPHS
2.2 Asynchronous Power Iteration
presented in [184, 179, 185, 168].
2.1.5 Preliminaries and Notation
In this chapter, we always assume thatAis a normal matrix, i.e.,A AH =AHA. We will useT to denote a subset of{1,· · · , 𝑁}, and its size is denoted as𝑡 =|T |. We will use the notationÍ
T to denote the summation over all subsets of{1,· · · , 𝑁} of a fixed size 𝑡 where the value of 𝑡 should follow from the context. The index selection matrix of the setDT satisfies the following identities for a given size𝑡:
Õ
T
DT = 𝑁−1
𝑡−1
I, (2.1)
1
𝑁 𝑡
Õ
T
DT A DT = 𝑡(𝑁−𝑡) diag(A) +𝑡(𝑡−1)A
𝑁(𝑁−1) , (2.2)
which will be used in the subsequent proofs.
does not have an autonomous implementation since it requires a centralized timing mechanism (synchronization) over the underlying graph.
In this study we will consider a variation of the power iteration, in which not all but a subset of indices, denoted byT, are updated simultaneously and the remaining ones stay unchanged. More precisely, given an update set T we consider the following asynchronous(coordinate-wise) power iteration:
𝑦𝑖 =
(Ax)𝑖, 𝑖 ∈ T, 𝑥𝑖, 𝑖∉T,
(2.5) wherexis the vector before update, andyis the vector after the update. In words, this update computes the multiplication Ax, but it only updates the values of the elements indexed by the setT, and keeps the remaining elements the same. In short, (2.3) is a “synchronous” update, whereas (2.5) is asynchronous. Both (2.3) and (2.5) are also referred to asstate recursions, where the graph signalxis regarded as the state of a system. The model in (2.5) was also studied in [14, 19, 32] with slight differences. Furthermore, (2.5) is reminiscent of the Hopfield neural network [84], with the difference that there is no nonlinearity in (2.5). (Studies in [14, 19] have non-linearity.)
The asynchronous update defined in (2.5) can be written as a matrix-vector multi- plication as follows:
y=Õ
𝑖∉T
e𝑖eH𝑖 x+Õ
𝑖∈T
e𝑖e𝑖HA x=Q(T ) x, (2.6) whereQ(T ) is the matrix representing the asynchronous update on a setT, and it can be written as follows:
Q(T ) =I+DT (A−I). (2.7) In the next few sections, where we perform a convergence analysis,Acan be treated as a generic matrix without considering specific relations to graphs. The relation to graph signals will be considered in Section 2.5. When the model in (2.5) is implemented on a graph (i.e.,Ais a graph operator), only the nodes in the update set T need to be synchronized. If the update set is selected as T ={1,· · · , 𝑁}, thenQ(T )=A. That is, the asynchronous update in (2.5) reduces to the classical synchronous update (graph shift) in (2.3). On the other extreme, if a single node is updated, |T |=1, then no synchronization is required at all and the nodes are
allowed to behave autonomously. We would like to note that the relation in (2.4) appears as if a node collects states of its neighbors, update its own state, and sits still. However, as we shall describe later in Section 2.6, we will consider the updates on a polynomial of the given graph operator, which will require the nodes to follow a collect-compute-broadcast scheme. These details will be elaborated in Section 2.6.5.
2.2.1 Randomized Asynchronous Updates
In this chapter we will study the behavior of a cascade of asynchronous updates where the update setT is assumed to be selected at random in each iteration. More precisely, we assume that the𝑘𝑡 ℎ iteration has the following form:
x𝑘 =Q x𝑘-1, (2.8)
wherex𝑘 denotes the signal at the𝑘𝑡 ℎiteration, andQis a random matrix due to the fact that the underlying update set is selected at random.
It should be noted thatQandQ(T )are different from each other. The matrixQ(T ) in (2.7) is a deterministic matrix. Given an update set T, Q(T ) represents the asynchronous update of (2.5). On the other hand,Q in (2.8) is a random variable whose outcomes are in the form ofQ(T ). More precisely, we consider the following probabilistic model:
P[Q=Q(T ) ]= 𝑝𝑡 𝑁
𝑡 −1
, where 𝑡 =|T |, (2.9)
where 𝑝𝑡 denotes the probability ofT having size𝑡, that is, 𝑝𝑡 =P
|T |=𝑡
, (2.10)
andÍ𝑁
𝑡=1𝑝𝑡 =1.
According to the model in (2.9), subsets of equal size are selected with equal probabilities. Therefore, the update scheme does not have any bias toward any node(s). To put differently, all the nodes are treated equally in the network. When 𝑝𝑁 =1, the model in (2.9) reduces to the regular power iteration in (2.3). When 𝑝1=1, only one node is selected uniformly at random, which corresponds to the autonomous network model of interest.
The number of nodes to be updated,T= |T |, is a discrete random variable whose distribution will be shown to determine the behavior of the asynchronous updates.
We will see later in Section 2.3.2 that the following definition is very useful in our quantitative analysis:
𝛿T = E[T(𝑁 −T) ] E[T(𝑁−1) ] =
𝑁−𝜇T −𝜎2 𝑇/𝜇T 𝑁 −1
, (2.11)
where 𝜇T and 𝜎2
𝑇 denote the mean and the variance of the random quantity T, respectively. It can be verified that 0 ≤ 𝛿T ≤ 1 with 𝛿T =0 if and only if all the nodes are updated in each iteration (synchronous power iteration), and𝛿T =1 if and only if exactly one node is updated in each iteration. As a result of this, 𝛿T will be referred to asthe amount of asynchronicityof iterations in the rest of the chapter.
We now prove:
Lemma 2.1. Expectation of the random matrixQin(2.9)is E[Q] = 𝜇T
𝑁 A+
1− 𝜇T
𝑁
I. (2.12)
Proof. The expectation ofQcan be written as E[Q] =E
E[Q|T]
, (2.13)
where the outer expectation is with respect to T (size of the sets), and the inner expectation is with respect to the content of the subsets of size T. Using (2.9) we have that
E[Q|T] =Õ
T
P[Q=Q(T ) |T] Q(T )=Õ
T
1
𝑁 T
I+DT (A−I)
, (2.14) whereÍ
T denotes a summation over subsets of sizeT. Then, E[Q|T] = 1
𝑁 T
Õ
T
I+ 1
𝑁 T
Õ
T
DT (A−I)=I+ 1
𝑁 T
𝑁-1
T-1
I (A−I), (2.15)
=T/𝑁 A+ (1−T/𝑁) I, (2.16)
where (2.15) follows from (2.1). Due to (2.13), we have
E[Q] =E[T/𝑁 A+ (1−T/𝑁) I] =𝜇T/𝑁 A+ (1−𝜇T/𝑁) I, (2.17)
which gives the result in (2.12).
Notice thatE[Q]is a convex combination of the operatorAand the identity matrix.
The quantity 𝜇T/𝑁 is the average fraction of the nodes that are updated simultane- ously per iteration, and it appears as the weight of the operator A in E[Q]. The case of 𝜇T =𝑁 results inE[Q] =A, which corresponds to the case of synchronous power iteration.