Stochastic Equilibrium

Performance

3.3 Stochastic Modeling and Analysis

3.3.2 Stochastic Equilibrium

Having obtained the necessary assignment statistics, we are ready to analyze the adap- tive virtual cut-through routing formulation. In general, multi-server queueing systems are notoriously difficult, and, except for a few special cases, usually defy analysis. In principle, any stochastic system can be modeled as a Markov process by incorporating complete information in the state description. In practice, however, such is not feasible,

and the challenge becomes that of picking an acceptable state representation with suf- ficient information [5]. In the present case, a complete state description would consist of the total number of packets presently stored in the node; the set of profitable output channels for each packet; whether or not each input and output channel is busy, and for how long; and, for systems with packet delivery guarantee, the relative priority of each stored packet. In fact, just keeping track of the profitable channels for packets in the buffers will require a total of ( ³³+₆⁶- 1

)

=

906192 different states, for a 3D mesh network with six bidirectional channels, and six internal buffers. There is clearly little hope of obtaining the stochastic equilibrium solution for such an enormous system; even if we did, we would be overwhelmed by too many insignificant details. In any case, such a complete state description is simply out of the question!

Another source of difficulty in network traffic modeling arises from the inherent dependency between the packet interarrival times and the packet length [26]. These dependencies render the modeling problem intractable to analysis. Therefore, in order to proceed, we shall adopt the following additional simplifying assumptions:

9. Packet arrivals at the various nodes and channels are independent of each other.

10. The packet-arrival distributions are memoryless which, in the present case, is modeled as a Poisson process with arrival rate ,\

=

cp/ L.

11. After each period of L cycles, a new set of profitable channels is chosen for each packet from the idealized distribution described in the previous subsection.

Assumptions (9) and (10) closely resemble the well-known Kleinrock Independence As- sumption [26], and were chosen with the same purpose and motivation in mind. In the original statement of the independence assumption, in addition to memoryless arrivals, each time a packet is received at a node, a new length is chosen from an exponential distribution. Since we assume fixed-length packets, such is not necessary; instead, we replace it with assumption (11), which eliminates the history dependencies in successive rounds of routing decisions. These assumptions, together with our earlier symmetry assumptions, allow us to completely characterize the stochastic state of a node by the total number of packets that are stored internally, i.e., the queueing population. In terms of standard shorthand notation in queueing theory [27], we have something akin to the

M/ D / c queueing system, i.e., memoryless arrivals; deterministic service time

=

L, and

c

2:

1 servers. However, there is one important difference between our case and those studied in conventional queueing systems: non-identical servers. This is because each packet has its own choice of profitable channels, hence, a server, i.e., a channel, may remain idle even if the queue is nonempty.

We are now ready to model the transition behavior of the system. Based on the observation made earlier, we shall observe the state of our system, i.e., the queueing population of a node, only at the sequence of snapshots at cycles: 0, L, 2L, 3L, ··•,etc ..

State transitions occur as a net result of two continuous processes: The arrivals of new packets at the intermediate node, and the departure of packets forwarded by the node. The stochastic equilibrium equations can now be obtained by coupling together the governing statistics of these two processes: the packet-assignment statistics, and the average packet-arrival rate.¹ In particular, let Qi(t) denote the state probability of having i queued packets at cycle t. We have, Vi

2:

0:

where i

+ J -

k

2:

0, Next, in these equations, we let t-too. Under our steady-state assumptions, Qi(t) -t Qi, the equilibrium-state probability, or time average, of having i queued packets at a node, and given the ergodicity of the system, this is also equal to the ensemble average. Hence, we have:

where i

+ J -

k

2:

0, and the normalizing equation:

Given the matching probabilities mi,k, and the arrival rate, A, these equations can be solved numerically using such well-known successive relaxation techniques as the Guass- Seidel method. To continue our analysis, we observe that having the equilibrium state

1Observe that under our assumption, it is possible, if not probable, to have more than c packets arriving in ::::; L cycles. The independence assumption becomes more accurate with a larger number of channels per node. Therefore, we expect the theoretical predictions for the 3D torus to be better than those for the 2D torus.

probabilities, Qi, allows us to determine S, the expected decrease in the total packet- to-destination distance per node over a £-cycle interval:

where i

+

j - k ~ 0,

which should be =AL= cp, the average throughput of a node, as we have ignored any misrouting in our model. This identity serves as an effective accuracy check for any numerical approximation obtained using finite truncation.

Given our assumption of homogeneous traffic, the total expected decrease in packet distance of the entire network over a £-cycle interval is: A

=

^NS, where N is the number of nodes in the network. The following observation then allows us to determine the steady-state packet-injection rate per node: Since the network is in a steady-state equilibrium, any decrease in packet distance must be balanced by an equivalent average influx of newly injected packets. Therefore, we have A

=

^{N qdL;} after rearranging, we have the average throughput per node: q

= ^rz;- = ~.

We are now ready to pursue our final objective in this analysis, i.e., to derive an approximate theoretical relationship between packet latency and the applied load. To proceed, we observe that the expected queueing population per node, Q, is given by:

On the other hand, according to Little's Theorem [30], we have:

Average Queueing Population= Average Throughput X Average Queueing Delay From these, we obtain the average queueing delay, Tq

=

~ Q. Finally, we have for the total average network packet latency, TN

=

^Tq

+

Tp

+

Tt, i.e., expressed as the sum of queueing delay, propagation delay, and transmission delay:

(3.1) Equation 3.1 allows us to determine the desired theoretical relationship between latency and throughput. Figures 3.4 and 3.5 depict the latency curves obtained for the 2D torus and 3D torus. In these figures, the applied load has been normalized with respect to the upper bound imposed by the network bisection capacity. Similarly, the average packet latency has been normalized with respect to the lower bound imposed by the sum of

N 20 0 R M A 15

L I

z

E 10 D A L T N E C y

5

0

Packet Length= 32, Average Distance = 16

0.0 0.2 0.4 0.6 0.8

Normalized Applied Load

Figure 3.4: Network Latency versus Applied Load: 2D Torus

Packet Length= 32, Average Distance = 24 N 16

0 14 M R

A ¹² L I 10 E

z

8 D L ⁶ A T 4 E N 2 C y

0

0.0 0.2 0.4 0.6 0.8

Normalized Applied Load

Figure 3.5: Network Latency versus Applied Load: 3D Torus 1.0

1.0

the packet length and the average message distance to destination. It is clear from the figures that at a very low applied load, the network latency is almost completely incurred by the sum of propagation delay and transmission delay. As applied load increases, the queueing delay starts to dominate and the overall latency approaches values comparable to those obtained in store-and-forward routing [25].

In the foregoing analysis, we have deliberately ignored the traffic through the internal channel connected to the message interface of a node. This traffic represents packets that are injected by and delivered to the message interface of the node. By deliberately omitting this traffic, we obtain results that are intrinsic only to the network and to the routing strategy. To see why, recall that in homogeneous steady-state equilibrium, the additional injected traffic from the message interface is exactly canceled by the arriving traffic destined to this node, on the average. Hence, the average queueing popuiation, the average resulting queueing delay, and, TN, the average network packet latency, all remain unchanged.

However, the internal channel does introduce two additional queues into the system:

a source injection queue at the message interface of the sender and a destination arrival queue at the router side of the receiver. For a reasonable-sized network, their effects can be safely ignored. Consider a typical 2D torus of size 32

x

32. The bisection bandwidth limits the injection rate to be ::; 0.25. In other words, the two additional queues are each operated at a utilization factor of ::; 0.25. In almost all queueing disciplines, this appears to be a very small perturbation that does not affect the first-order results.

Furthermore, it is clear from the above argument that this perturbation decreases as the network size increases. For smaller networks that are shorter in each dimension, however, the effect of this internal channel on the message interface cannot be ignored.

Following our assumptions, packets having arrived at their respective destination nodes that are waiting to be sent to the message interface will form a M / D /1 queue connected in tandem at the receiver side. For an 8 x 8 x 8 3D torus network, the injection-channel utilization factor, q, is limited to ::; 1; therefore, the extra delay can no longer be neglected. The average value of this extra delay, W, is given by the Pollaczek-Khinchin Formula [27] for M / D /1 queues:

W= q L

2(1 - q)

Since, under our assumption, the individual packets are also generated according to a Poisson distribution, we can finally observe that the same M/ D /1 queue is also formed at the message-interface side for packets waiting to be injected. In other words, the average source queueing time is also given by the above formula. We shall have more to say about these extra queues in section 3.5.