/
Otus FPGA Pilot Phase

Otus FPGA Pilot Phase

We are currently in the pilot phase for the FPGA partition. In this stage, we have acquired a small number of FPGA boards and are evaluating their suitability for our requirements. More FPGAs of different vendors will be added in the near future.

Hardware Setup

Currently we have

  • 8x Alveo V80 FPGAs (see official product page)

in our FPGA pilot phase of Otus.

An overview of the current hardware setup is given in the following table:

 

Accelerator and PCIe slot

 

 

Accelerator and PCIe slot

 

Hostname

61:00.0

71:00.0

91:00.0

note

fpga1607

Alveo V80 FPGA

 

 

Tapasco

fpga1608

Alveo V80 FPGA

 

Alveo V80 FPGA

VRT/SLASH, based on dev 18.08.26

fpga1609

Alveo V80 FPGA

 

Alveo V80 FPGA

VRT/SLASH, based on dev 18.08.26

fpga1610

Alveo U55C

Alveo U55C

Alveo U55C

 

fpga1611

Alveo U280 FPGA

Alveo U55C

Alveo V80 FPGA

VRT/SLASH, based on dev 18.08.26

fpga1612

Alveo V80 FPGA

Alveo U55C

Nvidia A40 GPU

Currently configured with SLASH VRT version 0.1.0, based on AVED 25.1

fpga1613

Nvidia A40 GPU

Nvidia A40 GPU

Alveo V80 FPGA

Currently configured with VRT based on AVED 24.1

Please note:

  • you only need to allocate one of the nodes for hardware execution. You can compile, emulate, simulate and synthesize on any other Otus node.

  • the Nvidia A40 GPUs were used for thermal evaluations of the nodes and will be removed in future.

  • the other FPGA cards (Alveo U55C and Alveo U280) are for test purposes and will be removed in future.

Software Stacks

Currently, we test the following software stacks in our pilot phase.

SLASH VRT (V80 Runtime)

The V80 cards can be used via VRT (similar abstraction to Xilinx XRT)

The VRT tool flow can be used with modules on any nodes of Otus:

module reset module load fpga module load fpga/xilinx/vrt/2026-08-18_dev_6a4cc11

Code base from dev branch is used, syntax: {DATE}_{BRANCH}_{COMMIT_HASH}, here

Example Applications

VRT comes with example applications (see https://github.com/Xilinx/SLASH/tree/dev/examples) to test the basic functionality.

In order to run them you can do the following:

Load required VRT module

module reset module load fpga module load fpga/xilinx/vrt/2026-08-18_dev_6a4cc11

  • loads the main VRT tool flow and all required dependencies

Get example repository

# Get code. git clone https://github.com/Xilinx/SLASH # Switch to branch/commit cd SLASH export SLASHBASE=`pwd` git switch dev git checkout 6a4cc11 # Init submodules. git submodule update --init --recursive --remote

Build examples for emulation

cd $SLASHBASE/examples/01_aximm # Prepare build directory. Remove -G Ninja for othe cmake -B build -S . -G Ninja -DSLASH_USE_REPO=OFF # Build host application cmake --build build # Build FPGA artefacts cmake --build build --target hls # compile HLS kernels cmake --build build --target aximm_emu # link into a emulation vrtbin # Run ./build/01_aximm 61:00 build/aximm_emu.vbin # Expected output EMU_EXEC: [startup] bound REP socket to tcp://*:5555 (verbose=off) EMU_EXEC: [manifest] manifest loaded schema=ok kernels=2 regs=16 callable=2 autostart=0 fetch.scalar=8 Test passed EMU_EXEC: [exit] received exit; fast_exit=false INFO [HLS SIM]: The maximum depth reached by any hls::stream() instance in the design is 1024

  • emu: everything runs on the CPU. See below for simulation and hardware build.

  • 61:00: Usually the PCIe ID (BDF) parameter of the V80 card to use. Can be any reasonable value in emulation.

Hardware Execution

If you want to execute a design in hardware, you need to allocate an FPGA node that has at least one V80 card attached.

Allocate node with V80 card

You can use this command to get node fpga1612 for one hour:

srun --partition=fpga -t 01:00:00 -w fpga1609 --pty bash

Load the required modules

module reset module load fpga module load fpga/xilinx/vrt/2026-08-18_dev_6a4cc11

Check FPGA status

Get the current status of the FPGAs via ami_tool. For example on fpga1609 with two V80 cards:

$ ami_tool overview AMI ------------------------------------------------------------- Version | 2.4.0 (0) Branch Hash | 6a4cc11e58893c41fb068b69823f779e7356c342 Hash Date | 20260818 Driver Version | 2.4.0 (0) BDF | Device | UUID | AMC | State ---------------------------------------------------------------------------------------- 61:00.0 | ALVEO V80 PQ | 99f0172027f50520fcbaca4bdf109725 | 2.4.0 (0) | READY 91:00.0 | ALVEO V80 PQ | 99f0172027f50520fcbaca4bdf109725 | 2.4.0 (0) | READY

  • the most important output is the State column. Only FPGAs in READY state should be used.

  • you can use the BDF 61:00 to identify one card

  • Use ami_tool -h for all options

Get status via v80-smi tool.

v80-smi list Board 0000:91:00 OK (PF0: OK) (PF1: OK) (PF2: OK) (VRTD: OK) Shell: service Board 0000:61:00 OK (PF0: OK) (PF1: OK) (PF2: OK) (VRTD: OK) Shell: service

  • look for OK state for every component and

  • most importantly look for Shell field. SLASH/VRT comes with two shells: service and compute

    • service shell supports networking (see for 06_dcmac example), while compute shell does not support networking, but offers more logic resources for user kernels instead

    • which shell a kernel uses is defined in config.cfg, snipped from 00_axilite:

[...] [connectivity] shell=service # <-- shell nk=accumulate:1:accumulate_0 nk=increment:1:increment_0 stream_connect=increment_0.axis_out:accumulate_0.axis_in sp=increment_0.m_axi_gmem0:HBM1

    • to switch between service and compute shell use

v80-smi reset --device 0000:91:00 --shell-type compute

    • status after switching

v80-smi list Board 0000:91:00 OK (PF0: OK) (PF1: OK) (PF2: OK) (VRTD: OK) Shell: compute # <-- shell switched Board 0000:61:00 OK (PF0: OK) (PF1: OK) (PF2: OK) (VRTD: OK) Shell: service

Execute example in hardware

To speed-up the process and save resources for unnecessary synthesis we have pre-synthesized the design for example 00_axilite. Copy the vrtbin file into your build directory:

# go to build directory and copy pre-synthesized design. cd build cp /opt/software/FPGA/Xilinx/VRT/vrt_dev_2026-08-18_6a4cc11/examples/01_aximm/build/aximm_hw.vbin .

You can run the design in hardware with

./01_aximm 91:00 aximm_hw.vbin # Expected output [2026-09-02 11:39:24.369] [INFO ] void vrt::impl::Device::programDevice() : Programming PDI via vrtd design writer /pc2/users/d/deffel/.cache/SLASH/vrt/vrtbin_91_00/images/top_i_slash_slash_aximm_hw_inst_0_partial.pdi [2026-09-02 11:39:25.624] [INFO ] vrt::impl::Device::Device(const string&, const string&, bool, vrt::ProgramType) : Programmed user clock to 200000000 Hz (target 200000000 Hz) Test passed

If you want to synthesize a design yourself, see the description below.

(Optional) Repeat steps for simulation and hardware build

Hardware simulation can be performed with

# Simulation cmake --build build --target aximm_sim # execute example cd build ./01_aximm 61:00 aximm_sim.vbin # Expected output Test passed

In order to synthesize a design instead of using the pre-synthesized version, you can use these steps:

#!/bin/sh # synthesis_script.sh #SBATCH -t 24:00:00 #SBATCH --cpus-per-task=8 #SBATCH --mem=64G #SBATCH -A <your_project_acronym> #SBATCH -p normal module reset module load fpga module load fpga/xilinx/vrt/2026-08-18_dev_6a4cc11 # Hardware cmake --build build --target aximm_hw

Then, we submit the synthesis_script.sh to the slurm workload manager:

sbatch ./synthesis_script.sh

Afterwards you can use the generated aximm_hw.vbin as described above for hardware execution.

Early Access and Troubleshooting

If you are interested in getting early access to the FPGA partition in the pilot phase or have issues/questions with the current setup, please contact us via Email.