Using FPGAs in the HACC Nodes

Next to our main FPGA installation in Noctua 2, the cluster also contains three nodes provided by AMD as part of the HACC initiative. Along with four AMD Instinct MI210 GPUs, they contain the following FPGA acceleration boards:

2x Xilinx Alveo U55C
2x Xilinx Versal VCK5000

In contrast to the main fpga partition, these boards are configured with fixed shells (not selectable by the user) and XRT 2.16 is installed.

Job Allocation

The three HACC nodes are contained in a separate hacc partition. To be able to submit jobs to this partition, your compute project needs to be enabled to use the HACC resources. Please send a brief mail to pc2-support@uni-paderborn.de stating your compute project and what resources you would like to use.

The HACC nodes are handed out exclusively, i.e., sharing a node between multiple jobs is not possible. Because the shell is not user-selectable, nodes can be allocated without any constraint:

[tester@n2login1 ~]$ srun -N1 -t5:0 -phacc /opt/xilinx/xrt/bin/xbutil examine
[...]
System Configuration
  OS Name              : Linux
  Release              : 4.18.0-372.32.1.el8_6.x86_64
  Version              : #1 SMP Fri Oct 7 12:35:10 EDT 2022
  Machine              : x86_64
  CPU Cores            : 128
  Memory               : 515549 MB
  Distribution         : Red Hat Enterprise Linux 8.6 (Ootpa)
  GLIBC                : 2.28
  Model                : AS -4124GS-TNR

XRT
  Version              : 2.14.354
  Branch               : 2022.2
  Hash                 : 43926231f7183688add2dccfd391b36a1f000bea
  Hash Date            : 2022-10-08 16:49:53
  XOCL                 : 2.14.354, 43926231f7183688add2dccfd391b36a1f000bea
  XCLMGMT              : 2.14.354, 43926231f7183688add2dccfd391b36a1f000bea

Devices present
BDF             :  Shell                              Platform UUID                         Device ID       Device Ready*  
---------------------------------------------------------------------------------------------------------------------------
[0000:e1:00.1]  :  xilinx_vck5000_gen4x8_qdma_base_2  05DCA096-76CB-730B-8D19-EC1192FBAE3F  user(inst=132)  Yes            
[0000:c1:00.1]  :  xilinx_u55c_gen3x16_xdma_base_3    97088961-FEAE-DA91-52A2-1D9DFD63CCEF  user(inst=133)  Yes            
[0000:a1:00.1]  :  xilinx_vck5000_gen4x8_qdma_base_2  05DCA096-76CB-730B-8D19-EC1192FBAE3F  user(inst=135)  Yes            
[0000:81:00.1]  :  xilinx_u55c_gen3x16_xdma_base_3    97088961-FEAE-DA91-52A2-1D9DFD63CCEF  user(inst=134)  Yes            


* Devices that are not ready will have reduced functionality when using XRT tools

In the example output you can also see the four FPGA boards, along with their user BDFs. Make sure to select the correct device type in your host code.

Software Modules

Depending on what kind of board you are targeting, you need to load the matching FPGA shell module. Note that by default the module for Noctua 2’s Alveo U280 boards will be loaded. On the HACC nodes, you need to swap it against the one for your target board:

[tester@n2login1 ~]$ module load fpga

[tester@n2login1 ~]$ module load xilinx/xrt/2.16

# Use one of the following commands to swap the shell module against the one you need
[tester@n2login1 ~]$ module swap xilinx/u280 xilinx/u55c
[tester@n2login1 ~]$ module swap xilinx/u280 xilinx/vck5000

Additional Notes on the Use of AI Engines

The VCK5000 boards contain AI Engines that can be programmed using Vitis. To use the AIE compiler and the AIE simulator, additional modules and licenses are required:

Vitis 23.2, which is automatically loaded with XRT 2.16, is currently not able to perform hardware synthesis for the VCK5000 board. If you target hardware synthesis, you may load the module xilinx/xrt/2.15 instead, or swap out Vitis against an older version:
ml swap xilinx/vitis/23.2 xilinx/vitis/23.1
In addition to the modules listed above, you may need to load the Graphviz module:
The AIE compiler and simulator require separate software licenses that are not included in Vitis. If you need help in acquiring these licenses from AMD, please get in touch with us.