...
The
changeFPGALinkscommand line tool works directly on the FPGA-Nodes during a running job.When the tool returns, the topology is changed to the requested one.
The tool output can also be used as a quick and handy way to visualize the current configuration with the FPGA-Link-GUI.
You can run the tool multiple times within the same job to sequentially create and use different topologies.
After changing the topology, make sure to reset or reprogram your FPGA designs. Otherwise the network interfaces may get stuck in an undefined state.
Usage in Interactive Jobs
Preparation
First submit a job with a fitting constraint either for the Alveo U280 nodes or the Bittware 520N nodes. Here interactively with srun for an Alveo U280 node:
...
Then you can run the commandchangeFPGAlinks. The command names might be subject to changes in the near future. You can run the command to specify a new topology, or run it without topology arguments to check the current status.
Examples for running the
...
tool interactively without a topology
This will only generate a overview and not change anything, but can be useful for documenting the current status:
...
| Code Block | ||
|---|---|---|
| ||
fpga-tester@n2fpga17:~ $ changeFPGAlinks # Not changing anything, just checking Your current link-setup: n2fpga17:acl0:ch1-n2fpga17:acl1:ch0 n2fpga17:acl0:ch3-n2fpga17:acl1:ch2 n2fpga17:acl1:ch1-n2fpga17:acl0:ch0 n2fpga17:acl1:ch3-n2fpga17:acl0:ch2 To visualize this configuration click here: https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga17%3Aacl0%3Ach1-n2fpga17%3Aacl1%3Ach0%20n2fpga17%3Aacl0%3Ach3-n2fpga17%3Aacl1%3Ach2%20n2fpga17%3Aacl1%3Ach1-n2fpga17%3Aacl0%3Ach0%20n2fpga17%3Aacl1%3Ach3-n2fpga17%3Aacl0%3Ach2%20 Your nodes in this Job (901745): n2fpga17 No configuration change requested. Goodbye. |
Visualize here: Click!
Examples for running the
...
tool interactively to specify a topology
with the --fpgalink= syntax to set the links:
...
| Code Block |
|---|
fpga-tester@n2fpga03:~
changeFPGAlinks --fpgalink=n00:acl1:ch1-n00:acl2:ch0 --fpgalink=n00:acl0:ch1-n00:acl1:ch0 --fpgalink=n00:acl0:ch0-n00:acl2:ch1
There are currently no links set up.
Your nodes in this Job (4745184):
n2fpga03
Started changing link-config with ID aaa3b083-ae6c-4057-8185-0228a706aa66
START: Thu Sep 14 17:50:54 CEST 2023
INFO: Request from user "fpga-tester" from job "4745184"
INFO: Nodelist of job: n2fpga03
INFO: Setting SPANK_FPGALINK0=n00:acl1:ch1-n00:acl2:ch0
INFO: Setting SPANK_FPGALINK1=n00:acl0:ch1-n00:acl1:ch0
INFO: Setting SPANK_FPGALINK2=n00:acl0:ch0-n00:acl2:ch1
Host list
n2fpga03
Generated connections
fpgalink n2fpga03:acl1:ch1-n2fpga03:acl2:ch0
fpgalink n2fpga03:acl0:ch1-n2fpga03:acl1:ch0
fpgalink n2fpga03:acl0:ch0-n2fpga03:acl2:ch1
Topology configuration request accepted after 0.3137032985687256s
[{"in":"3.2.8","out":"3.3.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.6","out":"3.2.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.5","out":"3.3.2","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]
To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga03%3Aacl1%3Ach1-n2fpga03%3Aacl2%3Ach0%20n2fpga03%3Aacl0%3Ach1-n2fpga03%3Aacl1%3Ach0%20n2fpga03%3Aacl0%3Ach0-n2fpga03%3Aacl2%3Ach1%20 |
Usage in Batch Mode
Using the batch script fpgaLinkMulti.sh
| Code Block |
|---|
#!/bin/bash
#SBATCH -t 2:00:00
#SBATCH -N 3
#SBATCH -J "fpgaLinkMulti"
#SBATCH -p fpga
#SBATCH --constraint=xilinx_u280_xrt2.15
#SBATCH -A <your_project>
## setup modules
ml reset
ml fpga/changeFPGAlinks
echo "Creating link topology (invoked only once from sbatch)"
changeFPGAlinks --fpgalink=n01:acl2:ch0-n01:acl1:ch1 --fpgalink=n00:acl2:ch0-n00:acl1:ch1 --fpgalink=n00:acl0:ch1-n00:acl1:ch0 --fpgalink=n00:acl2:ch1-n01:acl0:ch0 --fpgalink=n01:acl0:ch1-n01:acl1:ch0 --fpgalink=n01:acl2:ch1-n02:acl0:ch0 --fpgalink=n02:acl0:ch1-n02:acl1:ch0 --fpgalink=n00:acl0:ch0-n02:acl2:ch1 --fpgalink=n02:acl1:ch1-n02:acl2:ch0
echo "Creating test executable"
cat > test.sh <<'EOF'
#!/bin/bash
echo Executing job $SLURM_JOB_ID task $SLURM_PROCID on node $(uname -n)
changeFPGAlinks
EOF
echo "Parallel job that displays the topology from every involved node"
srun bash test.sh |
Sample output - note that we are invoking the test with srun on all nodes concurrently and still write to the same slurm log file, so output of the second phase is likely to be out of order and interleaved:
| Code Block |
|---|
Creating link topology (invoked only once from sbatch)
There are currently no links set up.
^[[1mYour nodes in this Job (19126695^[(B^[[m):
n2fpga05
n2fpga06
n2fpga08
Started changing link-config with ID e457e500-d6ae-41ef-ac59-dde0dbfa285e and test links after setup.
START: Wed Feb 5 16:52:54 CET 2025
INFO: Request from user "kenter" from job "19126695"
INFO: Nodelist of job: n2fpga05
n2fpga06
n2fpga08
INFO: Setting SPANK_FPGALINK0=n01:acl2:ch0-n01:acl1:ch1
INFO: Setting SPANK_FPGALINK1=n00:acl2:ch0-n00:acl1:ch1
INFO: Setting SPANK_FPGALINK2=n00:acl0:ch1-n00:acl1:ch0
INFO: Setting SPANK_FPGALINK3=n00:acl2:ch1-n01:acl0:ch0
INFO: Setting SPANK_FPGALINK4=n01:acl0:ch1-n01:acl1:ch0
INFO: Setting SPANK_FPGALINK5=n01:acl2:ch1-n02:acl0:ch0
INFO: Setting SPANK_FPGALINK6=n02:acl0:ch1-n02:acl1:ch0
INFO: Setting SPANK_FPGALINK7=n00:acl0:ch0-n02:acl2:ch1
INFO: Setting SPANK_FPGALINK8=n02:acl1:ch1-n02:acl2:ch0
Host list
n2fpga05
n2fpga06
n2fpga08
Generated connections
fpgalink n2fpga06:acl2:ch0-n2fpga06:acl1:ch1
fpgalink n2fpga05:acl2:ch0-n2fpga05:acl1:ch1
fpgalink n2fpga05:acl0:ch1-n2fpga05:acl1:ch0
fpgalink n2fpga05:acl2:ch1-n2fpga06:acl0:ch0
fpgalink n2fpga06:acl0:ch1-n2fpga06:acl1:ch0
fpgalink n2fpga06:acl2:ch1-n2fpga08:acl0:ch0
fpgalink n2fpga08:acl0:ch1-n2fpga08:acl1:ch0
fpgalink n2fpga05:acl0:ch0-n2fpga08:acl2:ch1
fpgalink n2fpga08:acl1:ch1-n2fpga08:acl2:ch0
Topology configuration request accepted after 0.2925856113433838s
[{"in":"3.5.3","out":"3.5.2","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.5","out":"3.4.4","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.2","out":"3.4.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.6","out":"3.4.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.8","out":"3.5.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.5.4","out":"3.6.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.4","out":"3.6.5","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.1","out":"3.6.8","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.6","out":"3.6.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]
Begin link tests
Iteration 1
n2fpga06:acl2:ch0:3.5.3>3.5.2 input: 5.92 output: 4.18 loss: 1.74
n2fpga06:acl1:ch1:3.5.2>3.5.3 input: 7.16 output: 4.94 loss: 2.22
n2fpga05:acl2:ch0:3.4.5>3.4.4 input: 6.09 output: 4.15 loss: 1.94
n2fpga05:acl1:ch1:3.4.4>3.4.5 input: 5.59 output: 3.65 loss: 1.94
n2fpga05:acl0:ch1:3.4.2>3.4.3 input: 6.45 output: 4.34 loss: 2.11
n2fpga05:acl1:ch0:3.4.3>3.4.2 input: 5.77 output: 3.91 loss: 1.86
n2fpga05:acl2:ch1:3.4.6>3.4.7 input: 5.75 output: 4.36 loss: 1.39
n2fpga06:acl0:ch0:3.4.7>3.4.6 input: 4.26 output: 2.68 loss: 1.58
n2fpga06:acl0:ch1:3.4.8>3.5.1 input: 3.72 output: 1.91 loss: 1.81
n2fpga06:acl1:ch0:3.5.1>3.4.8 input: 6.15 output: 4.85 loss: 1.30
n2fpga06:acl2:ch1:3.5.4>3.6.3 input: 6.19 output: 4.62 loss: 1.57
n2fpga08:acl0:ch0:3.6.3>3.5.4 input: 5.95 output: 4.37 loss: 1.59
n2fpga08:acl0:ch1:3.6.4>3.6.5 input: 3.89 output: -90.00 loss: -90.00
n2fpga08:acl1:ch0:3.6.5>3.6.4 input: 4.48 output: -90.00 loss: -90.00
n2fpga05:acl0:ch0:3.4.1>3.6.8 input: 6.15 output: 4.82 loss: 1.33
n2fpga08:acl2:ch1:3.6.8>3.4.1 input: 4.58 output: 3.07 loss: 1.51
n2fpga08:acl1:ch1:3.6.6>3.6.7 input: 5.29 output: 3.68 loss: 1.61
n2fpga08:acl2:ch0:3.6.7>3.6.6 input: 6.62 output: 4.79 loss: 1.82
Iteration 2
n2fpga06:acl2:ch0:3.5.3>3.5.2 input: 5.92 output: 4.18 loss: 1.74
n2fpga06:acl1:ch1:3.5.2>3.5.3 input: 7.16 output: 4.94 loss: 2.22
n2fpga05:acl2:ch0:3.4.5>3.4.4 input: 6.08 output: 4.15 loss: 1.92
n2fpga05:acl1:ch1:3.4.4>3.4.5 input: 5.56 output: 3.64 loss: 1.93
n2fpga05:acl0:ch1:3.4.2>3.4.3 input: 6.45 output: 4.35 loss: 2.10
n2fpga05:acl1:ch0:3.4.3>3.4.2 input: 5.77 output: 3.92 loss: 1.85
n2fpga05:acl2:ch1:3.4.6>3.4.7 input: 5.75 output: 4.36 loss: 1.39
n2fpga06:acl0:ch0:3.4.7>3.4.6 input: 4.26 output: 2.68 loss: 1.58
n2fpga06:acl0:ch1:3.4.8>3.5.1 input: 3.72 output: 1.91 loss: 1.81
n2fpga06:acl1:ch0:3.5.1>3.4.8 input: 6.15 output: 4.84 loss: 1.32
n2fpga06:acl2:ch1:3.5.4>3.6.3 input: 6.19 output: 4.56 loss: 1.63
n2fpga08:acl0:ch0:3.6.3>3.5.4 input: 5.95 output: 4.28 loss: 1.67
n2fpga08:acl0:ch1:3.6.4>3.6.5 input: 3.89 output: 1.60 loss: 2.28
n2fpga08:acl1:ch0:3.6.5>3.6.4 input: 4.48 output: 2.70 loss: 1.78
n2fpga05:acl0:ch0:3.4.1>3.6.8 input: 6.15 output: 4.83 loss: 1.32
n2fpga08:acl2:ch1:3.6.8>3.4.1 input: 4.58 output: 3.03 loss: 1.55
n2fpga08:acl1:ch1:3.6.6>3.6.7 input: 5.29 output: 3.68 loss: 1.61
n2fpga08:acl2:ch0:3.6.7>3.6.6 input: 6.63 output: 4.79 loss: 1.84
^[[1mTo visualize this configuration click here:^[(B^[[m
https://pc2.github.io/fpgalink-gui/index.html?import=--fpgalink%3Dn2fpga06%3Aacl2%3Ach0-n2fpga06%3Aacl1%3Ach1%20--fpgalink%3Dn2fpga05%3Aacl2%3Ach0-n2fpga05%3Aacl1%3Ach1%20--fpgalink%3Dn2fpga05%3Aacl0%3Ach1-n2fpga05%3Aacl1%3Ach0%20--fpgalink%3Dn2fpga05%3Aacl2%3Ach1-n2fpga06%3Aacl0%3Ach0%20--fpgalink%3Dn2fpga06%3Aacl0%3Ach1-n2fpga06%3Aacl1%3Ach0%20--fpgalink%3Dn2fpga06%3Aacl2%3Ach1-n2fpga08%3Aacl0%3Ach0%20--fpgalink%3Dn2fpga08%3Aacl0%3Ach1-n2fpga08%3Aacl1%3Ach0%20--fpgalink%3Dn2fpga05%3Aacl0%3Ach0-n2fpga08%3Aacl2%3Ach1%20--fpgalink%3Dn2fpga08%3Aacl1%3Ach1-n2fpga08%3Aacl2%3Ach0
Creating test executable
Parallel job that displays the topology from every involved node
cpu-bind=MASK - n2fpga08, task 2 0 [286359]: mask |BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB| set
cpu-bind=MASK - n2fpga05, task 0 0 [395218]: mask |BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB| set
cpu-bind=MASK - n2fpga06, task 1 0 [371503]: mask |BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB| set
Executing job 19126695 task 0 on node n2fpga05
----- FPGALINK -----
Summarizing most recent topology information and exporting FPGALINK variables:
Host list
n2fpga05
n2fpga06
n2fpga08
Generated connections
FPGALINK0=n2fpga06:acl2:ch0-n2fpga06:acl1:ch1
FPGALINK1=n2fpga05:acl2:ch0-n2fpga05:acl1:ch1
FPGALINK2=n2fpga05:acl0:ch1-n2fpga05:acl1:ch0
FPGALINK3=n2fpga05:acl2:ch1-n2fpga06:acl0:ch0
FPGALINK4=n2fpga06:acl0:ch1-n2fpga06:acl1:ch0
FPGALINK5=n2fpga06:acl2:ch1-n2fpga08:acl0:ch0
FPGALINK6=n2fpga08:acl0:ch1-n2fpga08:acl1:ch0
FPGALINK7=n2fpga05:acl0:ch0-n2fpga08:acl2:ch1
FPGALINK8=n2fpga08:acl1:ch1-n2fpga08:acl2:ch0
Topology configuration request accepted after 0.2925856113433838s
[{"in":"3.5.3","out":"3.5.2","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.5","out":"3.4.4","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.2","out":"3.4.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.6","out":"3.4.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.8","out":"3.5.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.5.4","out":"3.6.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.4","out":"3.6.5","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.1","out":"3.6.8","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.6","out":"3.6.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]
To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga06%3Aacl2%3Ach0-n2fpga06%3Aacl1%3Ach1%20n2fpga05%3Aacl2%3Ach0-n2fpga05%3Aacl1%3Ach1%20n2fpga05%3Aacl0%3Ach1-n2fpga05%3Aacl1%3Ach0%20n2fpga05%3Aacl2%3Ach1-n2fpga06%3Aacl0%3Ach0%20n2fpga06%3Aacl0%3Ach1-n2fpga06%3Aacl1%3Ach0%20n2fpga06%3Aacl2%3Ach1-n2fpga08%3Aacl0%3Ach0%20n2fpga08%3Aacl0%3Ach1-n2fpga08%3Aacl1%3Ach0%20n2fpga05%3Aacl0%3Ach0-n2fpga08%3Aacl2%3Ach1%20n2fpga08%3Aacl1%3Ach1-n2fpga08%3Aacl2%3Ach0%20
Your nodes in this Job (19126695):
n2fpga05
n2fpga06
n2fpga08
No configuration change requested. Goodbye.
Executing job 19126695 task 2 on node n2fpga08
...
Executing job 19126695 task 1 on node n2fpga06
... |