Properties of the Coherent Protocol

Feasibility

2.2 Communication Deadlock Freedom

2.2.2 Properties of the Coherent Protocol

The coherent protocol exhibits a number of interesting and useful logical properties that we shall now investigate. These properties will be exploited later to establish our

fundamental results on packet deliveries and injection fairness.

Let N(Rf), N(R0 ) , N(V/), and N(V0 ) denote, respectively, the total number of occurred R'f, R0 , V/, and V0 events across a channel c E Ck since initialization. The following invariant conditions are satisfied by a bidirectional channel that follows the coherent protocol under the defined initial conditions:

(Il)

(12) (13)

0

<

N(R0 ) - N(V/) O

<

N(Rf) - N(V0 )

O

<

N(Rf) - N(V/) (14) 0 < N(Ro) - N(Vo)

<

1

<

1

<

1

<

1

The first two inequalities simply express causality conditions implied by our demand- driven protocol: A node transfers valid data to an output channel only after the receiving partner has indicated its readiness to accept. The last two inequalities express condi- tions that are direct results of the strict alternation of ready and data-valid events and the specified initial state. In essence, it requires the communicating nodes to send their data out, i.e., increment N(V0 ) and N(V/), respectively, only after the nodes themselves have indicated their willingness to accept new data from their respective partners. To- gether, these conditions result in a cycle-by-cycle exchange of data flits between the communicating nodes connected by the channel. Subtracting the two pairs of invari- ant conditions from one another, we obtain the following pair of invariant conditions,

\:/c Eck, of a node nk:

(I5) 0 <

(I6)

⁰

<

IN(Rf) - N(Ro)I IN(V/) - N(Vo)I

<

1

<

1

The tight matching of the data transfer rates of the opposite directions of a channel has been made explicit by these two invariant conditions. The strategy is to equalize the number of data-flit transfer operations back and forth along the channel to within a difference of at most one at all times. This is equivalent to restricting the communicating partners to within a syn chronic distance of one from each other in the Petri Net theoretic sense [45]. Similarly, according to our initial set of invariants, we have for different channels of the same node nk:

(I2)

⁰

<

N(Rf) - N(Vo)

<

^{1 \:/c E Ck}

⇒

(17)

0

<

IN(R?) - N(R?)I

<

¹ \:/c1, c2 E Ck

(Il)

0

<

N(Ro) - N(V/)

<

¹ \:/c E Ck

⇒

(18)

⁰

<

IN(V/^{1) -} N(V/²

)1 <

¹ \:/c1,c2 E Ck

In other words, we are assured that the data rates across different channels of the same node are matched to each other at all times.

Observe that according to the protocol, each node inside the network carries out the required handshaking with its nearest neighbors locally without any global knowledge of the states of nodes farther away. In fact, nodes that are of distance d apart in the network are of synchronic distance

f d/21

from each other; i.e., they may have their respective total number of cycle completions differ by as much as

f d/21.

To see this, let node no be the source, and consider a path of length d joining nodes nk, k

=

I, 2, ... , d-I, d, which are, respectively, distance i away from no. According to the invariant condition (17), we have O ~ IN(R;1) - N(R?)I ~ I, Vc1, c2 E Ck of the same node nk. Therefore, nodes nk-l and nk+l, which are both neighbors of nk, can have their number of completed cycles differ by at most 1. Applying this fact to nodes no, n2, n4, ... makes it dear that the cumulative difference between no and nd can be as much as and no more than

f d/21.

Global synchrony does not exist and there is no global synchronization signal to be distributed over the entire network. As a result, a network following the protocol is potentially infinite in size, or, realistically, arbitrarily extensible.

While there is no global synchrony of events nor any unique global clock over the entire network, the local synchrony enforced by the coherent protocol allows one to define a set of local clocks that are compatible with each other. More precisely, we define the value of a discrete local clock Tk of a network node nk E N as the total number of R₀ signaling events occurred after initialization of the node. Intuitively, each R0 event signals both the completion of the previous transfer cycle and the beginning of the next cycle, when new decisions have to be made within each node to determine if data are to be forwarded to neighboring nodes or to be buffered internally. Hence, one is justified in regarding the R0 events as defining local clock cycles with each clock cycle corresponding to a routing epoch.

Let each message packet, Pm, traveling inside the network, keep count, Am, of the total number of R0 events that Pm has observed since its injection. Two local clocks, Ti and Tk, of nodes nj, nk EN, respectively, are compatible if a packet, Pm, traveling in the network starting from node nj at local time Tj = TJ with count Am = A:n, and ending at node nk at local time Tk

=

^Tk with count Am

=

^A;,,, will find, upon its arrival

at nk, an agreement:

Tke - Tl! _J

=

Ae - As _{m m}

To see that the defined set of local clocks is indeed compatible, observe that we only have to establish its validity when our packet, Pm, is being forwarded to the next node along its trajectory during a protocol cycle; otherwise, the count recorded by the packet simply tracks the local time of its resident node. During the actual transfer of Pm from one node to another, the sender signals the beginning of the transfer with a V0 event, thus incrementing N(V0 ) . The coherency constraints dictate that N(R0 )

=

N(Rf), \/c EC at the sender side at this point; hence, the sender and all its neighbors must have identical clock values. When the receiver signals the completion of transfer with its own R0 event, Pm is already at the receiver side. Therefore, both the receiver's local clock value and the count recorded by Pm are incremented, preserving the agreement between packet count and the local clock value. Compatibility of local clocks follows by induction.

As a result of the compatibility of the defined set of local clocks, the recorded count values are not only consistent with the local clocks, but also consistent with each other.

Specifically, let AA

=

Am - An be the difference in the recorded count values of packets Pm and Pn upon a rendezvous: The value ..6.A will remain identical upon all future rendezvous of the same packets Pm and Pn• In other words, we are justified in defining the value Am as being the age of the packet Pm· The existence of a well-defined age for packets in a coherent network allows us to define meaningful dynamic priorities that are essential in establishing packet-delivery guarantees. Furthermore, the basic protocol cycle provides a natural partition of continuous physical time into a sequence of discrete epochs upon which local routing decisions are made.