Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Change Serial Channels during a running job

For the general usage of FPGA Serial Channels visit the main documentation page.

...

Overview

  • ThechangeFPGALinkscommand line tool works directly on the FPGA-Nodes during a running job.

  • When the tool returns, the topology is changed to the requested one.

  • The tool output can also be used as a quick and handy way to visualize the current configuration with the FPGA-Link-GUI.

Usage

...

  • You can run the tool multiple times within the same job to sequentially create and use different topologies.

  • After changing the topology, make sure to reset or reprogram your FPGA designs. Otherwise the network interfaces may get stuck in an undefined state.

Usage in Interactive Jobs

First submit a job like normal. You can either start with the first topology or omit it and start without any linkswith a fitting constraint either for the Alveo U280 nodes or the Bittware 520N nodes. Here interactively with srun for an Alveo U280 node:

Code Block
breakoutModewide
fpga-tester@n2login1:~
$ srun -A pc2-mitarbeiter<your_project> --constraint=emulxilinx_u280_xrt2.15 -N 1 -p fpga -t 2:00:00  --fpgalink=ringO--pty zsh

or for an Bittware 520N node with _max Constraint:

Code Block
breakoutModewide
fpga-tester@n2login1:~
$ srun -A <your_project> --constraint=bittware_520n_20.4.0_max -N 1 -p fpga -t 2:00:00 --pty zsh

Then In both cases, next load the module:

Code Block
breakoutModewide
fpga-tester@n2fpga17:~
$ module load fpga/changeFPGAlinks

Then you can run changeFPGAlinks. Again either the commandchangeFPGAlinks. The command names might be subject to changes in the near future. You can run the command to specify a new topology, or run it without topology arguments to check the current status.

...

with the --fpgalink syntax is used to change the topology= syntax to set the links:

Connect for 3 Xilinx Alveo U280 cards within a single node the respective first port to the Ethernet switch as shown here:

Code Block
fpga-tester@n2fpga03:~
changeFPGAlinks  --fpgalink=n00:acl0:ch0-eth --fpgalink=n00:acl1:ch0-eth --fpgalink=n00:acl2:ch0-eth
There are currently no links set up.
Your nodes in this Job (4745184):
	n2fpga03
Started changing link-config with ID f7888529-5edf-4208-8090-c9cb46eb9dc8
START: Thu Sep 14 17:45:09 CEST 2023
INFO: Request from user "fpga-tester" from job "4745184"
INFO: Nodelist of job: n2fpga03
INFO: Setting SPANK_FPGALINK0=n00:acl0:ch0-eth
INFO: Setting SPANK_FPGALINK1=n00:acl1:ch0-eth
INFO: Setting SPANK_FPGALINK2=n00:acl2:ch0-eth
Host list 
n2fpga03
Generated connections
fpgalink n2fpga03:acl0:ch0-ethn2fpga03:acl0:ch0
fpgalink n2fpga03:acl1:ch0-ethn2fpga03:acl1:ch0
fpgalink n2fpga03:acl2:ch0-ethn2fpga03:acl2:ch0
Topology configuration request accepted after 0.4885869026184082s
[{"in":"3.2.5","out":"4.6.5","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.7","out":"4.6.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.3.1","out":"4.7.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]

To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga03%3Aacl0%3Ach0-ethn2fpga03%3Aacl0%3Ach0%20n2fpga03%3Aacl1%3Ach0-ethn2fpga03%3Aacl1%3Ach0%20n2fpga03%3Aacl2%3Ach0-ethn2fpga03%3Aacl2%3Ach0%20

Create a point-to-point loopback configuration for 2 Bittware 520N cards within one node as shown here:

Code Block
breakoutModewide
fpga-tester@n2fpga17:~
$ changeFPGAlinks --fpgalink=n00:acl0:ch0-n00:acl0:ch1 --fpgalink=n00:acl1:ch2-n00:acl1:ch3 --fpgalink=n00:acl1:ch0-n00:acl1:ch1 --fpgalink=n00:acl0:ch2-n00:acl0:ch3 # Change to custom topology
Your current link-setup:
       n2fpga17:acl0:ch1-n2fpga17:acl1:ch0
       n2fpga17:acl0:ch3-n2fpga17:acl1:ch2
       n2fpga17:acl1:ch1-n2fpga17:acl0:ch0
       n2fpga17:acl1:ch3-n2fpga17:acl0:ch2

To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga17%3Aacl0%3Ach1-n2fpga17%3Aacl1%3Ach0%20n2fpga17%3Aacl0%3Ach3-n2fpga17%3Aacl1%3Ach2%20n2fpga17%3Aacl1%3Ach1-n2fpga17%3Aacl0%3Ach0%20n2fpga17%3Aacl1%3Ach3-n2fpga17%3Aacl0%3Ach2%20

Your nodes in this Job (901745):
       n2fpga17
Started changing link-config with ID 0defb6f3-244f-415d-8372-5eaa324b4aa0
START: Thu Sep 16 11:44:41 CEST 2021
INFO: Request from user "fpga-tester" from job "901745"
INFO: Nodelist of job: n2fpga17
INFO: Setting SPANK_FPGALINK0=n00:acl0:ch0-n00:acl0:ch1
INFO: Setting SPANK_FPGALINK1=n00:acl1:ch2-n00:acl1:ch3
INFO: Setting SPANK_FPGALINK2=n00:acl1:ch0-n00:acl1:ch1
INFO: Setting SPANK_FPGALINK3=n00:acl0:ch2-n00:acl0:ch3
Host list 
n2fpga17
Generated connections
fpgalink n2fpga17:acl0:ch0-n2fpga17:acl0:ch1
fpgalink n2fpga17:acl1:ch2-n2fpga17:acl1:ch3
fpgalink n2fpga17:acl1:ch0-n2fpga17:acl1:ch1
fpgalink n2fpga17:acl0:ch2-n2fpga17:acl0:ch3
Topology configuration request accepted after 0.205315113068s


To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga17%3Aacl0%3Ach0-n2fpga17%3Aacl0%3Ach1%20n2fpga17%3Aacl1%3Ach2-n2fpga17%3Aacl1%3Ach3%20n2fpga17%3Aacl1%3Ach0-n2fpga17%3Aacl1%3Ach1%20n2fpga17%3Aacl0%3Ach2-n2fpga17%3Aacl0%3Ach3%20

Visualize here: Click!

The topology gets changed to the requested one. Run the tool as often as you need topology changes. The next run will reflect the new topologyCreate a customized point-to-point ring for 3 Xilinx Alveo cards within one node as shown here:

Code Block
fpga-tester@n2fpga03:~
changeFPGAlinks --fpgalink=n00:acl1:ch1-n00:acl2:ch0 --fpgalink=n00:acl0:ch1-n00:acl1:ch0 --fpgalink=n00:acl0:ch0-n00:acl2:ch1
There are currently no links set up.
Your nodes in this Job (4745184):
	n2fpga03
Started changing link-config with ID aaa3b083-ae6c-4057-8185-0228a706aa66
START: Thu Sep 14 17:50:54 CEST 2023
INFO: Request from user "fpga-tester" from job "4745184"
INFO: Nodelist of job: n2fpga03
INFO: Setting SPANK_FPGALINK0=n00:acl1:ch1-n00:acl2:ch0
INFO: Setting SPANK_FPGALINK1=n00:acl0:ch1-n00:acl1:ch0
INFO: Setting SPANK_FPGALINK2=n00:acl0:ch0-n00:acl2:ch1
Host list 
n2fpga03
Generated connections
fpgalink n2fpga03:acl1:ch1-n2fpga03:acl2:ch0
fpgalink n2fpga03:acl0:ch1-n2fpga03:acl1:ch0
fpgalink n2fpga03:acl0:ch0-n2fpga03:acl2:ch1
Topology configuration request accepted after 0.3137032985687256s
[{"in":"3.2.8","out":"3.3.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.6","out":"3.2.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.2.5","out":"3.3.2","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]

To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga03%3Aacl1%3Ach1-n2fpga03%3Aacl2%3Ach0%20n2fpga03%3Aacl0%3Ach1-n2fpga03%3Aacl1%3Ach0%20n2fpga03%3Aacl0%3Ach0-n2fpga03%3Aacl2%3Ach1%20

Usage in Batch Mode

Using the batch script fpgaLinkMulti.sh

Code Block
#!/bin/bash
#SBATCH -t 2:00:00
#SBATCH -N 3
#SBATCH -J "fpgaLinkMulti"
#SBATCH -p fpga
#SBATCH --constraint=xilinx_u280_xrt2.15
#SBATCH -A <your_project>

## setup modules
ml reset
ml fpga/changeFPGAlinks

echo "Creating link topology (invoked only once from sbatch)"
changeFPGAlinks --fpgalink=n01:acl2:ch0-n01:acl1:ch1 --fpgalink=n00:acl2:ch0-n00:acl1:ch1 --fpgalink=n00:acl0:ch1-n00:acl1:ch0 --fpgalink=n00:acl2:ch1-n01:acl0:ch0 --fpgalink=n01:acl0:ch1-n01:acl1:ch0 --fpgalink=n01:acl2:ch1-n02:acl0:ch0 --fpgalink=n02:acl0:ch1-n02:acl1:ch0 --fpgalink=n00:acl0:ch0-n02:acl2:ch1 --fpgalink=n02:acl1:ch1-n02:acl2:ch0

echo "Creating test executable"
cat > test.sh <<'EOF'
#!/bin/bash
echo Executing job $SLURM_JOB_ID task $SLURM_PROCID on node $(uname -n)
changeFPGAlinks
EOF

echo "Parallel job that displays the topology from every involved node"
srun bash test.sh

Sample output - note that we are invoking the test with srun on all nodes concurrently and still write to the same slurm log file, so output of the second phase is likely to be out of order and interleaved:

Code Block
Creating link topology (invoked only once from sbatch)
There are currently no links set up.
^[[1mYour nodes in this Job (19126695^[(B^[[m):
        n2fpga05
        n2fpga06
        n2fpga08
Started changing link-config with ID e457e500-d6ae-41ef-ac59-dde0dbfa285e and test links after setup.
START: Wed Feb  5 16:52:54 CET 2025
INFO: Request from user "kenter" from job "19126695"
INFO: Nodelist of job: n2fpga05
n2fpga06
n2fpga08
INFO: Setting SPANK_FPGALINK0=n01:acl2:ch0-n01:acl1:ch1
INFO: Setting SPANK_FPGALINK1=n00:acl2:ch0-n00:acl1:ch1
INFO: Setting SPANK_FPGALINK2=n00:acl0:ch1-n00:acl1:ch0
INFO: Setting SPANK_FPGALINK3=n00:acl2:ch1-n01:acl0:ch0
INFO: Setting SPANK_FPGALINK4=n01:acl0:ch1-n01:acl1:ch0
INFO: Setting SPANK_FPGALINK5=n01:acl2:ch1-n02:acl0:ch0
INFO: Setting SPANK_FPGALINK6=n02:acl0:ch1-n02:acl1:ch0
INFO: Setting SPANK_FPGALINK7=n00:acl0:ch0-n02:acl2:ch1
INFO: Setting SPANK_FPGALINK8=n02:acl1:ch1-n02:acl2:ch0
Host list
n2fpga05
n2fpga06
n2fpga08
Generated connections
fpgalink n2fpga06:acl2:ch0-n2fpga06:acl1:ch1
fpgalink n2fpga05:acl2:ch0-n2fpga05:acl1:ch1
fpgalink n2fpga05:acl0:ch1-n2fpga05:acl1:ch0
fpgalink n2fpga05:acl2:ch1-n2fpga06:acl0:ch0
fpgalink n2fpga06:acl0:ch1-n2fpga06:acl1:ch0
fpgalink n2fpga06:acl2:ch1-n2fpga08:acl0:ch0
fpgalink n2fpga08:acl0:ch1-n2fpga08:acl1:ch0
fpgalink n2fpga05:acl0:ch0-n2fpga08:acl2:ch1
fpgalink n2fpga08:acl1:ch1-n2fpga08:acl2:ch0
Topology configuration request accepted after 0.2925856113433838s
[{"in":"3.5.3","out":"3.5.2","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.5","out":"3.4.4","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.2","out":"3.4.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.6","out":"3.4.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.8","out":"3.5.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.5.4","out":"3.6.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.4","out":"3.6.5","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.1","out":"3.6.8","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.6","out":"3.6.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]
Begin link tests
Iteration 1
n2fpga06:acl2:ch0:3.5.3>3.5.2 input: 5.92 output: 4.18 loss: 1.74
n2fpga06:acl1:ch1:3.5.2>3.5.3 input: 7.16 output: 4.94 loss: 2.22
n2fpga05:acl2:ch0:3.4.5>3.4.4 input: 6.09 output: 4.15 loss: 1.94
n2fpga05:acl1:ch1:3.4.4>3.4.5 input: 5.59 output: 3.65 loss: 1.94
n2fpga05:acl0:ch1:3.4.2>3.4.3 input: 6.45 output: 4.34 loss: 2.11
n2fpga05:acl1:ch0:3.4.3>3.4.2 input: 5.77 output: 3.91 loss: 1.86
n2fpga05:acl2:ch1:3.4.6>3.4.7 input: 5.75 output: 4.36 loss: 1.39
n2fpga06:acl0:ch0:3.4.7>3.4.6 input: 4.26 output: 2.68 loss: 1.58
n2fpga06:acl0:ch1:3.4.8>3.5.1 input: 3.72 output: 1.91 loss: 1.81
n2fpga06:acl1:ch0:3.5.1>3.4.8 input: 6.15 output: 4.85 loss: 1.30
n2fpga06:acl2:ch1:3.5.4>3.6.3 input: 6.19 output: 4.62 loss: 1.57
n2fpga08:acl0:ch0:3.6.3>3.5.4 input: 5.95 output: 4.37 loss: 1.59
n2fpga08:acl0:ch1:3.6.4>3.6.5 input: 3.89 output: -90.00 loss: -90.00
n2fpga08:acl1:ch0:3.6.5>3.6.4 input: 4.48 output: -90.00 loss: -90.00
n2fpga05:acl0:ch0:3.4.1>3.6.8 input: 6.15 output: 4.82 loss: 1.33
n2fpga08:acl2:ch1:3.6.8>3.4.1 input: 4.58 output: 3.07 loss: 1.51
n2fpga08:acl1:ch1:3.6.6>3.6.7 input: 5.29 output: 3.68 loss: 1.61
n2fpga08:acl2:ch0:3.6.7>3.6.6 input: 6.62 output: 4.79 loss: 1.82
Iteration 2
n2fpga06:acl2:ch0:3.5.3>3.5.2 input: 5.92 output: 4.18 loss: 1.74
n2fpga06:acl1:ch1:3.5.2>3.5.3 input: 7.16 output: 4.94 loss: 2.22
n2fpga05:acl2:ch0:3.4.5>3.4.4 input: 6.08 output: 4.15 loss: 1.92
n2fpga05:acl1:ch1:3.4.4>3.4.5 input: 5.56 output: 3.64 loss: 1.93
n2fpga05:acl0:ch1:3.4.2>3.4.3 input: 6.45 output: 4.35 loss: 2.10
n2fpga05:acl1:ch0:3.4.3>3.4.2 input: 5.77 output: 3.92 loss: 1.85
n2fpga05:acl2:ch1:3.4.6>3.4.7 input: 5.75 output: 4.36 loss: 1.39
n2fpga06:acl0:ch0:3.4.7>3.4.6 input: 4.26 output: 2.68 loss: 1.58
n2fpga06:acl0:ch1:3.4.8>3.5.1 input: 3.72 output: 1.91 loss: 1.81
n2fpga06:acl1:ch0:3.5.1>3.4.8 input: 6.15 output: 4.84 loss: 1.32
n2fpga06:acl2:ch1:3.5.4>3.6.3 input: 6.19 output: 4.56 loss: 1.63
n2fpga08:acl0:ch0:3.6.3>3.5.4 input: 5.95 output: 4.28 loss: 1.67
n2fpga08:acl0:ch1:3.6.4>3.6.5 input: 3.89 output: 1.60 loss: 2.28
n2fpga08:acl1:ch0:3.6.5>3.6.4 input: 4.48 output: 2.70 loss: 1.78
n2fpga05:acl0:ch0:3.4.1>3.6.8 input: 6.15 output: 4.83 loss: 1.32
n2fpga08:acl2:ch1:3.6.8>3.4.1 input: 4.58 output: 3.03 loss: 1.55
n2fpga08:acl1:ch1:3.6.6>3.6.7 input: 5.29 output: 3.68 loss: 1.61
n2fpga08:acl2:ch0:3.6.7>3.6.6 input: 6.63 output: 4.79 loss: 1.84

^[[1mTo visualize this configuration click here:^[(B^[[m
https://pc2.github.io/fpgalink-gui/index.html?import=--fpgalink%3Dn2fpga06%3Aacl2%3Ach0-n2fpga06%3Aacl1%3Ach1%20--fpgalink%3Dn2fpga05%3Aacl2%3Ach0-n2fpga05%3Aacl1%3Ach1%20--fpgalink%3Dn2fpga05%3Aacl0%3Ach1-n2fpga05%3Aacl1%3Ach0%20--fpgalink%3Dn2fpga05%3Aacl2%3Ach1-n2fpga06%3Aacl0%3Ach0%20--fpgalink%3Dn2fpga06%3Aacl0%3Ach1-n2fpga06%3Aacl1%3Ach0%20--fpgalink%3Dn2fpga06%3Aacl2%3Ach1-n2fpga08%3Aacl0%3Ach0%20--fpgalink%3Dn2fpga08%3Aacl0%3Ach1-n2fpga08%3Aacl1%3Ach0%20--fpgalink%3Dn2fpga05%3Aacl0%3Ach0-n2fpga08%3Aacl2%3Ach1%20--fpgalink%3Dn2fpga08%3Aacl1%3Ach1-n2fpga08%3Aacl2%3Ach0

Creating test executable
Parallel job that displays the topology from every involved node
cpu-bind=MASK - n2fpga08, task  2  0 [286359]: mask |BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB|  set
cpu-bind=MASK - n2fpga05, task  0  0 [395218]: mask |BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB|  set
cpu-bind=MASK - n2fpga06, task  1  0 [371503]: mask |BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB||BBBBBBBB|BBBBBBBB|  set

Executing job 19126695 task 0 on node n2fpga05
----- FPGALINK -----
Summarizing most recent topology information and exporting FPGALINK variables:
Host list
n2fpga05
n2fpga06
n2fpga08
Generated connections
FPGALINK0=n2fpga06:acl2:ch0-n2fpga06:acl1:ch1
FPGALINK1=n2fpga05:acl2:ch0-n2fpga05:acl1:ch1
FPGALINK2=n2fpga05:acl0:ch1-n2fpga05:acl1:ch0
FPGALINK3=n2fpga05:acl2:ch1-n2fpga06:acl0:ch0
FPGALINK4=n2fpga06:acl0:ch1-n2fpga06:acl1:ch0
FPGALINK5=n2fpga06:acl2:ch1-n2fpga08:acl0:ch0
FPGALINK6=n2fpga08:acl0:ch1-n2fpga08:acl1:ch0
FPGALINK7=n2fpga05:acl0:ch0-n2fpga08:acl2:ch1
FPGALINK8=n2fpga08:acl1:ch1-n2fpga08:acl2:ch0
Topology configuration request accepted after 0.2925856113433838s
[{"in":"3.5.3","out":"3.5.2","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.5","out":"3.4.4","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.2","out":"3.4.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.6","out":"3.4.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.8","out":"3.5.1","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.5.4","out":"3.6.3","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.4","out":"3.6.5","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.4.1","out":"3.6.8","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}},{"in":"3.6.6","out":"3.6.7","response":{"status":"1","msg":"OK","description":"Cross Connection added successfully!"}}]
To visualize this configuration click here:
https://pc2.github.io/fpgalink-gui/index.html?import=n2fpga06%3Aacl2%3Ach0-n2fpga06%3Aacl1%3Ach1%20n2fpga05%3Aacl2%3Ach0-n2fpga05%3Aacl1%3Ach1%20n2fpga05%3Aacl0%3Ach1-n2fpga05%3Aacl1%3Ach0%20n2fpga05%3Aacl2%3Ach1-n2fpga06%3Aacl0%3Ach0%20n2fpga06%3Aacl0%3Ach1-n2fpga06%3Aacl1%3Ach0%20n2fpga06%3Aacl2%3Ach1-n2fpga08%3Aacl0%3Ach0%20n2fpga08%3Aacl0%3Ach1-n2fpga08%3Aacl1%3Ach0%20n2fpga05%3Aacl0%3Ach0-n2fpga08%3Aacl2%3Ach1%20n2fpga08%3Aacl1%3Ach1-n2fpga08%3Aacl2%3Ach0%20
Your nodes in this Job (19126695):
        n2fpga05
        n2fpga06
        n2fpga08
No configuration change requested. Goodbye.

Executing job 19126695 task 2 on node n2fpga08
...

Executing job 19126695 task 1 on node n2fpga06
...