Sample Topologies and Legacy fpgalink Interface to Slurm
- Tobias Kenter
Sample Topologies
A number of predefined topologies can be created in the FPGALink-GUI online editor. Click on the advanced tutorial button for examples.
On the right, you see a topology denoted as Pair, connecting each channel of one FPGA in a node to the corresponding channel of the second FPGA, and created with a single click of the node context menu in the editor (import link to FPGALink-GUI).
Legacy fpgalink Interface to Slurm
As alternative to the changeFPGALinks command line tool as described on the https://upb-pc2.atlassian.net/wiki/spaces/PC2DOK/pages/1903573 overview page and with further examples and details under https://upb-pc2.atlassian.net/wiki/spaces/PC2DOK/pages/1903821, when targeting the Bittware 520N nodes, --fpgalink arguments can be passed directly to srun or salloc. When appending the --fpgalink arguments from the editor to the Slurm command (instead of using them as input to changeFPGALinks) and adding quotation marks around the individual connection strings, the configuration gets applied at the start of the job.
srun -A pc2-mitarbeiter --constraint=bittware_520n_20.4.0_max -N 1 -t 10:00 -p fpga --fpgalink="n00:acl0:ch0-n00:acl1:ch0" --fpgalink="n00:acl0:ch1-n00:acl1:ch1" --fpgalink="n00:acl0:ch2-n00:acl1:ch2" --fpgalink="n00:acl0:ch3-n00:acl1:ch3" --pty bashWe recommend using srun and sbatch, because this information is not automatically shown when using salloc (the configuration itself still works). When using salloc, you can still recover the information and setup your environment variables by invoking the changeFPGALinks command line tool.
Predefined Topologies
The Slurm --fpgalink interface provides a set of predefined topologies as follows.
Topology type | Invocation | Min-Max number of nodes | Brief description |
|---|---|---|---|
pair |
| 1-N | Pairwise connect the 2 FPGAs within each node |
clique |
| 2 | All-to-all connection for 2 nodes, 4 FPGAs |
ring |
| 1-N | Ring with two links per direction, acl0 down, acl1 up |
| 1-N | Ring with two links per direction, acl0 down, acl1 down | |
| 1-N | Ring with two links per direction, acl0 and acl1 neighbors | |
torus |
| 1-N | Torus with 2 FPGAs per row |
| 2-N | Torus with 3 FPGAs per row | |
| 2-N | Torus with 4 FPGAs per row | |
| 3-N | Torus with 5 FPGAs per row | |
| 3-N | Torus with 6 FPGAs per row |
Pair topology
Within each node, all channels of one FPGA board are connected to the respective channel of the other FPGA board. No connections between nodes are made.
The following example uses three nodes n00-n02 and connects within each node all four channels from the first FPGA board acl0 to the four channels of the second FPGA board acl1 (see figure). The pair topology example can be directly used in the FPGA-Link GUI using this link.
srun -p fpga -A pc2-mitarbeiter --constraint=19.2.0_max -N 3 --fpgalink=pair --pty bashClique topology
Within a pair of 2 nodes, each of the 4 FPGAs is connected to all 3 other FPGAs.
channel 0: to the same FPGA in the other node
channel 1: to the other FPGA in the same node
channel 2: to the other FPGA in the other node.
The following example uses two nodes n00-n01 and connects within each node all four channels from the first FPGA board acl0 to the four channels of the second FPGA board acl1 (see figure). The clique topology example can be directly used in the FPGA-Link GUI using this link.
Ring topology
This setup puts all FPGAs in a ring topology that defines for each FPGA the neighbor FPGAs "north" and "south". It connects each FPGA's channels 0 and 2 to the "north" direction and channels 1 and 3 to the "south" direction. Thus, the local perspective for each node within the topology is
Three different variants define how the FPGAs are arranged into the ring
Full example for a ringO with 4 nodes. See this example in the FPGA-Link GUI using this link.
Torus topology
This setup puts all FPGAs in a torus topology that defines for each FPGA the neighbor FPGAs "north", "south", "west", "east". It connects each FPGA's
channel 0 to the "north" direction,
channel 1 to the "south" direction,
channel 2 to the "west" direction and
channel 3 to the "east" direction.
Thus, the local perspective for each node within the topology is
The torus topology can be instantiated with a configurable width, that is number of FPGAs that are connected in "west-east" direction. With an uneven width, FPGAs in the same node can belong to consecutive rows of the torus. The number of FPGAs gets rounded down to the biggest full torus for the given width. The following block illustrates 3 different torus topologies on nodes fpga-[0001-0005].
Full example for a torus4 with 8 nodes. See this example in the FPGA-Link GUI using this link.