Chapter 3 Building SpiNNaker Machines 52
3.3 Putting Everything Together
As indicated earlier, SpiNN-5 production boards can be connected together to build SpiNNakermachines. Figure3.10shows two examples ofsmall-scalemachines.
A single-board machine, containing one SpiNN-5 board, is shown in Fig- ure3.10(a). This machine is adequate for software development and small-scale neural network simulations. Figure3.10(b) shows a 24-board machine that con- sists of a fully-populated 1900 cabinet. Theblack SATAcables used for board-to- board interconnect are clearly visible on the front of the machine. Also visible are thewhiteEthernet cables for access to theSpiNNaker chips and the BMP. This machine offers over 20,000ARM processing cores with around 1.5 KW power consumption when fully loaded.
3.3.1 SpiNNaker1MAssembly
Large-scaleSpiNNakermachines are modular systems. The basic building block is a standard 1900card frame that holds 24 SpiNN-5 boards and the required ancil- lary equipment. Figure3.11(a) shows the front of the card frame with 24 boards installed, and Figure3.11(b) shows the back of the card frame with three 650 W power supplies and the card frame backplane. Each module also includes a fan tray, located below the card frame, that pulls air from the front of the card frame, between the boards, and blows it to the back. Finally, the module contains a 26-port Ethernet switch that connects the SpiNN-5 boards.
The card frame backplane houses a serialROMthat contains information about the frame location in the machine. These data are used during the boot process to
Figure 3.10.Small-scale SpiNNaker machines. (a) A cased 48-node 864-core board.
(b) A 24-board 20,736-core machine.
Figure 3.11.A card frame holds 24 SpiNN-5 boards, power supplies and a backplane.
6 m
Figure 3.12.SpiNNaker1M: 10 cabinets and 3,600 SATA cables interconnecting them.
Figure reproduced with permission from Heathcote [94].
configure theIPandMACaddresses of each SpiNN-5 board. It also contains power supply data as well as information about fan speed control and temperature limits.
Additionally, the backplane provides access to temperature sensors and to aLiquid- Crystal Display (LCD)located on the front of the card frame. The display is used to provide information, such as operating temperature and power supply levels, to the machine operator. Finally, the backplane carries aController Area Network (CAN) bus, used by theBMPto communicate with each other. The SpiNN-5 boards are connected to the backplane through an edge connector, located at the bottom right corner in Figure3.5(a).
To build larger machines, card frames are assembled together in 1900 cabinets.
Each cabinet holds five card frames, for a total of 120 SpiNN-5 boards, containing 5,760SpiNNaker chips/103,680ARMcores. Figure 3.12shows the 10 cabinets and 3,600SATAcables required to buildSpiNNaker1M that contains 1,036,800 cores.
Heat management is a significant design concern in massively parallel sys- tems. Supercomputers usually consume large amounts of power, in the order of Megawatts, that are ultimately converted into heat. Most supercomputers require bespoke liquid or mixed air-liquid cooling systems. The focus on energy-efficient design and construction means thatSpiNNaker1Mconsumes well under 100 KW when all cores are operating at full load, simplifying heat management. A couple of chillers blow cool air into the machine room. Air enters the machine through the cabinet front doors, and the card-frame fans move it through the SpiNN-5 boards towards chimneys located at the top of each cabinet where it is steered back to the chillers through a plenum. One of the chillers, with the associated chimneys and plenum, can be seen in Figure3.17. The chillers transfer heat to a water system through coils. The chillers are controlled by thermostats measuring the tempera- ture in the chimneys. Each chiller has a nominal capacity of 70 KW and a nominal air flow capacity of 3.7 m3/s.
3.3.2 SpiNNaker1MInterconnect
As with any supercomputer, physically assembling a large SpiNNaker machine poses many practical challenges in terms of arranging, installing and maintaining the thousands of metres of network cables required. Cabling techniques for con- ventional architectures and network topologies are well understood and embodied by industry standards such as TIA-942 [250]. Unfortunately, the use of a hexagonal torus topology forSpiNNakerrenders existing approaches inadequate.
Naïve arrangements of torus topologies, hexagonal or otherwise, feature physi- cally longwrap-around cables that connect units at the peripheries of the system.
Long connections can be problematic for several reasons:
Performance: Signal quality diminishes as cables get longer, requiring the use of slower signalling speeds, increased error correction overhead or more complex hardware.
Energy: Some energy is lost in cables; longer cables lose more signal energy requir- ing higher drive strengths and/or buffering to maintain signal integrity.
Cost: Shorter cables are cheaper than long ones. Longer cables imply more cabling in a given space making the task of cable installation and system maintenance more difficult, increasing labour costs by as much as5×[40].
In some cases, long connections in supercomputers may be eliminated by creative physical organisation of the system. For example, the distinctive ‘C’-shaped design of early Cray supercomputers was chosen to reduce the lengths of physical connec- tions and thus improve system performance [255]. Unfortunately, this approach
1 2 3 4 5 6 (a)
1 2 3
4 5 6
(b)
1 6 2 5 3 4
(c)
Figure 3.13.Folding and interleaving a ring network to reduce maximum cable length.
Figures reproduced with permission from Heathcote [94].
Figure 3.14. Network folding to shorten interconnect. Figures reproduced with permis- sion from Heathcote [94].
does not scale up in the general case and requires potentially expensive bespoke physical infrastructure. Alternatively, the need for long cables is often eliminated by folding and interleaving units of the network [42]. This process is illustrated for a 1D torus topology (a ring network) in Figure3.13. A naïve arrangement of units in this topology results in a long cable connecting the units at the ends of the ring (Figure3.13(a)). To eliminate these long connections, half of the units are
‘folded’ on top of the others (Figure 3.13(b)) and then this arrangement of units is interleaved (Figure3.13(c)). This ordering of units requires no long cables while still observing the physical constraint that units must be laid out in a line.
The folding and interleaving process may be extended toN-dimensional torus topologies by folding each axis in turn, as illustrated in Figure3.14. Folding once along each axis eliminates long connections crossing from left to right, top to bot- tom and from the bottom-left corner to the top-right corner. Since all axes are orthogonal in non-hexagonal topologies, the folding process only moves units along the axis being folded. Unfortunately, this type of folding does not work for hexag- onal torus topologies due to the non-orthogonality of the three axes. To exploit the folding technique used by non-hexagonal topologies, the units in a hexagonal torus topology must be mapped into a space with orthogonal coordinates. The choice of transformation to an orthogonal coordinate system can have an impact on how physically far apart logically neighbouring units are in the final arrangement.
Figure 3.15 illustrates the two transformations proposed by Heathcote [94]
to map hexagonal arrangements of units into a 2D orthogonal coordinate space.
(a) (b) (c)
Figure 3.15.Transformations to map hexagonal arrangements of units into a 2D orthog- onal coordinate space. Figures reproduced with permission from Heathcote [94].
The first transformation,shearing (Figure3.15(b)), is general purpose but intro- duces some distortion. The second transformation,slicing(Figure3.15(c)), is less general but can introduce less distortion than shearing and therefore may lead to shorter cable lengths.
Once a regular 2D grid of units has been formed, this may be folded in the con- ventional way as illustrated in Figure3.14. Any shear-transformed network may be folded this way since its wrap-around connections always follow this pattern.
Slice-transformed networks may only be folded like this when their aspect ratio is 1:2 when the pattern of wrap-around links is the same as a shear-transformed net- work. When ‘square’ networks, that is, those with a 1:1 aspect ratio, are sliced, the network must be folded twice along the Y axis to eliminate the criss-crossing wrap- around links. It is not possible to eliminate wrap-around links from sliced networks with other aspect ratios by folding. After folding, the units are interleaved, yielding a 2D arrangement of units in which no connection spans the width or height of the system. The maximum connection distance is constant for any network thus allowing the topology to scale up.
As indicated earlier, the hexagonal torus topology also applies to SpiNNaker when the boards are considered as nodes. The folded and interleaved arrangement of units produced by these techniques may be translated into physical arrangements ofSpiNNakerboards in a machine room. Figure3.16illustrates how the SpiNN-5 boards that make upSpiNNaker1M can be folded and interleaved to keep cable length short.1
3.3.3 SpiNNaker1MCabling
Due to the high density of units in a SpiNNaker system, the detailed cabling patterns used can be complex, despite their overall regularity. Figure3.17shows SpiNNakerteam researchers and PhD students cablingSpiNNaker1M, a process that took several days even with the valuable cooperation of the machine itself.
1. See also:https://youtu.be/z1_gE_ugEgE
Figure 3.16.SpiNNaker1M: Long interconnect wires are avoided by folding and interleav- ing the board array in both dimensions.
Figure 3.17.SpiNNaker1M cooperates in its cabling (l. to r. Christian Brenninkmeijer, Robert James, Garibaldi Pineda García, Alan B. Stokes, Luca Peres and Andrew Gait).
Luca earned the right to connect the last cable by providing the closest estimate to the total length of cable used (see Table 3.8).
To cope with this complexity, we developedSpiNNer[96], a collection of soft- ware tools for generating cabling plans and guiding cable installation and main- tenance of SpiNNaker machines. SpiNNer employs diagnostic hardware built intoSpiNNakerto guide the cable installation process. Figure3.18(a) shows how SpiNNer uses diagnostic LEDs on the SpiNNaker boards to indicate where to connect a cable. The software also provides step-by-step cabling instructions via aGraphical User Interface (GUI), shown in Figure3.18(b), and audible instruc- tions delivered via headphones. These instructions explicitly specify the length of cable to use for each connection, thus avoiding the common problem of techni- cians over-estimating the cable length required [156]. Diagnostic registers in the
Figure 3.18.SpiNNer guides cable installation. Figures reproduced with permission from Heathcote [94].
Table 3.8. SATA cable usage inSpiNNaker1M.
Cable length (m) Quantity Total length (m)
0.15 304 45.60
0.30 1,504 451.20
0.50 1,014 507.00
0.70 742 519.40
0.90 36 32.40
total 3,600 1,555.60
spiNNlinkinterconnect are then used to verify the correct installation of each cable in real time, ensuring that mistakes are highlighted and fixed immediately.
The ‘rule of (three-)thumbs’ proposed by Mazaris [156] was used in SpiNNaker1M. This rule suggests that a minimum of 5 cm of cable slack should be provided. AsSpiNNakeruses off-the-shelfSATAcables, only standard lengths were available. For any given span, the shortest length of cable providing at least 5 cm of slack was used. Table3.8lists the cable lengths used and the total number of cables of each length. The table shows a total cable length of over 1.5 km.
3.4 Using the Million-Core Machine: Tear it to Pieces