Chapter 4: Interconnection Issues
4.4 A Distance-Independent Measure of Locality
in the path are released by sending them to state UNRESERVED.
This procedure is used by all the nodes in sending messages. Since none of the the data in the message is stored in the network, the brea..l<:ing of paths involves no loss of data. As an added advantage, the message is sent in order, eliminating the need for the reassembly of packets as required in queued systems. The system does not deadlock. because one path ·will always win any contention. Congestion of the network will result in many paths being broken before they are completed, but since they are able to retry until they do achieve a complete path, the system exhibits liveness.
This type of Boolean N-cube was also simulated to compare it with the queued networks. In these simulations, each nodes has a queue of 4 messages produced by its processor, but there are no other queues in the system. To change the state of a link, the simulations require enough time to trruJ.sfer 64 bits to the link, such that link reservations occur in a finit,e time. The link data rates and message traffic used are the same as those used in the queued networks.
To start, we introduce the concept of a "neighborhood". From the point of view of a single processing node in a network. its neighborhood is a set of other processing nodes with which it will cornmlli"'licate v1-ith a probability greater than some threshold T. The size of the neighborhood is a measure of message locality. If the neighborhood size was the same as the number of nodes in the system, then traffic in the system could be said to be uniform.
Every node would have an equal probability of communicating with any other node. For small neighborhood sizes, traffic can be said to be very localized where the probability that a node communicates Vii.th one of its neighborhood set is much higher than for other nodes. The size of the neighborhood is denoted by a.
For given message traffic, the probability that a given node communicates with any selected node can be determined. The nodes can then be arranged by decreasing probability. The neighborhood is ideally the first a nodes in the ordered list, where a is to be a measure of locality. The probability that each of the first a nodes will communicate with the given node will be greater than or equal to the theshold T. The list can be approximated by a geometric distribution where:
1 -.B<
f
(x) = - e aa
The dependent variable x is the position in the list and
f
(x) is the probability of communication with node at that position. The constant a can be thought of as the approximate size of the neighborhood. The geometric distribution has the desirable programming property of having a simple, closed inverse. Also, its mean and variance are conveniently given as a and a2, respectively.In the sh'Tlulations, the message locality is a parameter. The parameter a is the desired size of the neighborhood and is used, as follows, to generate appropriate message traffic. A number Xis picked at random in the interval between 0 a.n.d
l.
The inverse of the geometric distribution is used to finda
what positionp in the list of nodes is represented by this probability:
p =-a Loge (aX)
At this point, p represents a randomly chosen number with the distribution of the desired message traffic. It remains to select a corresponding processing node in the network being simulated. T'nis process converts the number p to a corresponding distance in the network being simulated. That is, a distance d is chosen which is the number of communication links that must be traveled in the network to reach any one of p processing nodes.
This transformation is different for each network topology. For each of the topologies in question, the follow relationships hold, describing how many nodes R can be accessed by traveling exactly l communication links, where N is the total number of nodes in the network:
(1) Boolean N-cube (O<l:-::;;logN)
R = (logN)!
l ! (logN -l )!
(2) N-ary Tree (where b
=
branching ratio and 2:-::;; l :-::;; 21ogb N) .LiR
=
(b-1)b 2(3) Two Dimensional Array (assuming large N and no boundaries) R =4l
(4) Chordal Ring (assuming large N) R
=
4lThe above functions are integrated with respect to l (the number of links) to produce a table. The table can be indexed by p to find the corresponding entry which is the distanced, in links, that must be traveled to access p nodes.
Of the specific nodes that are found to be exactly distance d from the node that is to send a message, one member of the set is chosen at random.
The chosen node is then sent a message in the simulated system. As all the nodes in the network exhibit the same behavior and select the destinations of their messages by the same method, the overall message trai"fic has a locality determined by the original parameter a.
In the simulation results, references to "Traffic" indicate the level of message locality as set by a. A small a, in the range of 3 to 5, is highly local traffic. An a of 12 or more may be regarded has substantially non-local traffic for systems of less than 100 processing nodes. The parameter a ha,s no effect on message length or frequency, it affects only the destinations of messages.
Tue intent of this model of traffic locality is that it be used to represent communication requirements of an object-oriented program. Since it is possible for the same program to run on machines of different topology, the parameter a has been made independent of distance. The simulation results are normalized by the use of a. The use of the parameter a in the simulated traffic of each system gives an indication of how each will react to the same class of application programs. The group size, as represented by a, describes a homogeneous program execution. Real programs may exhibit nonhomogeneous communication by having objects with widely different characteristics. The placement of objects in processors can cause
nonhomogeneous communication among the processors. Such progra..'Tis might be better characterized as a composite set of several group sizes rather than one group size. The follovving simulation results are based on homogeneous programs whose locality of communication is characterized by a single parameter a.