...
Sometimes a previously configured topology needs to be reconfigured, either because the topology was specified incorrectly or a different topology is required. This would entail saving the state of the job, canceling the running job and submitting a new job. By using the tool changeFPGAlinks mid-job you can change the topology on-the-fly without canceling the running allocation. This tool can also be used as a quick and handy way to visualize the current configuration with the FPGA-Link-GUI.
Usage
The tool currently only works directly on the FPGA-Nodes during a running job.
When the tool returns, the topology is changed to the requested one.
After changing the topology, make sure to reset or reprogram your FPGA designs. Otherwise the network interfaces may get stuck in an undefined state.
You can run the tool as often as you need topology changes.
Preparation
First submit a job like normal. You can start without any intial links set:
...
or for the Bittware 520N partition you can also start with a specific initial topology:
| Code Block | ||
|---|---|---|
| ||
fpga-tester@n2login1:~ $ srun -A pc2-mitarbeiter --constraint=bittware_520n_20.4.0_max -N 1 -p fpga -t 2:00:00 --fpgalink=ringO --pty zsh |
Then In both cases, next load the module:
| Code Block | ||
|---|---|---|
| ||
fpga-tester@n2fpga17:~ $ module load fpga/changeFPGAlinks |
Then you can run the commandchangeFPGAlinksXilinxfor the Alveo U280 partition or the commandchangeFPGAlinks for the Bittware 520N partition. The command names might be subject to changes / unification in the near future. You can run the command to specify a new topology, or run it without topology arguments to check the current status.
Examples for running the command without a topology
...
This will only generate a overview and not change anything, but can be useful for documenting )the current status:
| Code Block | ||
|---|---|---|
| ||
fpga-tester@n2fpga03:~
$ changeFPGAlinksXilinx # Not changing anything, just checking
There are currently no links set up.
Your nodes in this Job (4743991):
n2fpga03
No configuration change requested. Goodbye.
|
| Code Block | ||
|---|---|---|
| ||
fpga-tester@n2fpga17:~ $ changeFPGAlinks # Not changing anything, just checking Your current link-setup: n2fpga17:acl0:ch1-n2fpga17:acl1:ch0 n2fpga17:acl0:ch3-n2fpga17:acl1:ch2 n2fpga17:acl1:ch1-n2fpga17:acl0:ch0 n2fpga17:acl1:ch3-n2fpga17:acl0:ch2 To visualize this configuration click here: https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga17%3Aacl0%3Ach1-n2fpga17%3Aacl1%3Ach0%20n2fpga17%3Aacl0%3Ach3-n2fpga17%3Aacl1%3Ach2%20n2fpga17%3Aacl1%3Ach1-n2fpga17%3Aacl0%3Ach0%20n2fpga17%3Aacl1%3Ach3-n2fpga17%3Aacl0%3Ach2%20 Your nodes in this Job (901745): n2fpga17 No configuration change requested. Goodbye. |
Visualize here: Click!
Examples for running the command to specify a topology
with the --fpgalink= syntax to set the links:
Connect for 3 Xilinx Alveo U280 cards within a single node the respective first port to the Ethernet switch as shown here:
| Code Block |
|---|
fpga-tester@n2fpga03:~
changeFPGAlinksXilinx --fpgalink=n00:acl0:ch0-eth --fpgalink=n00:acl1:ch0-eth --fpgalink=n00:acl2:ch0-eth
There are currently no links set up.
Your nodes in this Job (4745184):
n2fpga03
Started changing link-config with ID f7888529-5edf-4208-8090-c9cb46eb9dc8
START: Thu Sep 14 17:45:09 CEST 2023
INFO: Request from user "fpga-tester" from job "4745184"
INFO: Nodelist of job: n2fpga03
INFO: Setting SPANK_FPGALINK0=n00:acl0:ch0-eth
INFO: Setting SPANK_FPGALINK1=n00:acl1:ch0-eth
INFO: Setting SPANK_FPGALINK2=n00:acl2:ch0-eth
Host list
n2fpga03
Generated connections
fpgalink n2fpga03:acl0:ch0-ethn2fpga03:acl0:ch0
fpgalink n2fpga03:acl1:ch0-ethn2fpga03:acl1:ch0
fpgalink n2fpga03:acl2:ch0-ethn2fpga03:acl2:ch0
Topology configuration request accepted after 0.4885869026184082s
[{"in":"3.2.5","out":"4.6.5","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.7","out":"4.6.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.3.1","out":"4.7.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]
To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga03%3Aacl0%3Ach0-ethn2fpga03%3Aacl0%3Ach0%20n2fpga03%3Aacl1%3Ach0-ethn2fpga03%3Aacl1%3Ach0%20n2fpga03%3Aacl2%3Ach0-ethn2fpga03%3Aacl2%3Ach0%20 |
Create a point-to-point loopback configuration for 2 Bittware 520N cards within one node as shown here:
| Code Block | ||
|---|---|---|
| ||
fpga-tester@n2fpga17:~ $ changeFPGAlinks --fpgalink=n00:acl0:ch0-n00:acl0:ch1 --fpgalink=n00:acl1:ch2-n00:acl1:ch3 --fpgalink=n00:acl1:ch0-n00:acl1:ch1 --fpgalink=n00:acl0:ch2-n00:acl0:ch3 # Change to custom topology Your current link-setup: n2fpga17:acl0:ch1-n2fpga17:acl1:ch0 n2fpga17:acl0:ch3-n2fpga17:acl1:ch2 n2fpga17:acl1:ch1-n2fpga17:acl0:ch0 n2fpga17:acl1:ch3-n2fpga17:acl0:ch2 To visualize this configuration click here: https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga17%3Aacl0%3Ach1-n2fpga17%3Aacl1%3Ach0%20n2fpga17%3Aacl0%3Ach3-n2fpga17%3Aacl1%3Ach2%20n2fpga17%3Aacl1%3Ach1-n2fpga17%3Aacl0%3Ach0%20n2fpga17%3Aacl1%3Ach3-n2fpga17%3Aacl0%3Ach2%20 Your nodes in this Job (901745): n2fpga17 Started changing link-config with ID 0defb6f3-244f-415d-8372-5eaa324b4aa0 START: Thu Sep 16 11:44:41 CEST 2021 INFO: Request from user "fpga-tester" from job "901745" INFO: Nodelist of job: n2fpga17 INFO: Setting SPANK_FPGALINK0=n00:acl0:ch0-n00:acl0:ch1 INFO: Setting SPANK_FPGALINK1=n00:acl1:ch2-n00:acl1:ch3 INFO: Setting SPANK_FPGALINK2=n00:acl1:ch0-n00:acl1:ch1 INFO: Setting SPANK_FPGALINK3=n00:acl0:ch2-n00:acl0:ch3 Host list n2fpga17 Generated connections fpgalink n2fpga17:acl0:ch0-n2fpga17:acl0:ch1 fpgalink n2fpga17:acl1:ch2-n2fpga17:acl1:ch3 fpgalink n2fpga17:acl1:ch0-n2fpga17:acl1:ch1 fpgalink n2fpga17:acl0:ch2-n2fpga17:acl0:ch3 Topology configuration request accepted after 0.205315113068s To visualize this configuration click here: https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga17%3Aacl0%3Ach0-n2fpga17%3Aacl0%3Ach1%20n2fpga17%3Aacl1%3Ach2-n2fpga17%3Aacl1%3Ach3%20n2fpga17%3Aacl1%3Ach0-n2fpga17%3Aacl1%3Ach1%20n2fpga17%3Aacl0%3Ach2-n2fpga17%3Aacl0%3Ach3%20 |
Visualize here: Click!
The topology gets changed to the requested one. Run the tool as often as you need topology changes. The next run will reflect the new topology.Create a customized point-to-point ring for 3 Xilinx Alveo cards within one node as shown here:
| Code Block |
|---|
fpga-tester@n2fpga03:~
changeFPGAlinksXilinx --fpgalink=n00:acl1:ch1-n00:acl2:ch0 --fpgalink=n00:acl0:ch1-n00:acl1:ch0 --fpgalink=n00:acl0:ch0-n00:acl2:ch1
There are currently no links set up.
Your nodes in this Job (4745184):
n2fpga03
Started changing link-config with ID aaa3b083-ae6c-4057-8185-0228a706aa66
START: Thu Sep 14 17:50:54 CEST 2023
INFO: Request from user "fpga-tester" from job "4745184"
INFO: Nodelist of job: n2fpga03
INFO: Setting SPANK_FPGALINK0=n00:acl1:ch1-n00:acl2:ch0
INFO: Setting SPANK_FPGALINK1=n00:acl0:ch1-n00:acl1:ch0
INFO: Setting SPANK_FPGALINK2=n00:acl0:ch0-n00:acl2:ch1
Host list
n2fpga03
Generated connections
fpgalink n2fpga03:acl1:ch1-n2fpga03:acl2:ch0
fpgalink n2fpga03:acl0:ch1-n2fpga03:acl1:ch0
fpgalink n2fpga03:acl0:ch0-n2fpga03:acl2:ch1
Topology configuration request accepted after 0.3137032985687256s
[{"in":"3.2.8","out":"3.3.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.6","out":"3.2.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.5","out":"3.3.2","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]
To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga03%3Aacl1%3Ach1-n2fpga03%3Aacl2%3Ach0%20n2fpga03%3Aacl0%3Ach1-n2fpga03%3Aacl1%3Ach0%20n2fpga03%3Aacl0%3Ach0-n2fpga03%3Aacl2%3Ach1%20 |