System Architecture

Systems Detail Pages

If you want to move or migrate compute resources between clusters or get access to other cluster, follow this description.

Systems Comparison Table

	Otus	Noctua 2

	Otus	Noctua 2
System	Lenovo	Atos BullSequana XH2000
Processor Cores	142,656	143,872
Total Main Memory	593,3 TiB	347.5 TiB
Floating-Point Performance	CPU: 11,44 PFLOPS DP Peak (8,42 PFlop/s Linpack) GPU: 7,45 PFLOPS DP Peak (4,65 PFlop/s Linpack)	CPU: 5.4 PFLOPS DP Peak (4.19 PFlop/s Linpack) GPU: 2.49 PFLOPS DP Tensor Core Peak (ca. 1.7 PFlop/s Linpack)
Cabinets	11 racks - all components with direct liquid cooling, 2 racks - all processors with direct liquid cooling, 6 racks - air cooling	12 racks - direct liquid cooling, 7 racks - air cooling, four of them with active backdoor cooling
Communication Network CPUs	Nvidia NDR InfiniBand fabric, blocking factor 2:1 compute/FPGA nodes connected with 200Gbps GPU nodes connected with 2x400Gbps	Mellanox InfiniBand 100/200 HDR, 1:2 blocking factor
Storage System	IBM Spectrum Scale > 5 PB capacity (Flash + Hard disks)	DDN Exascaler 7990X with NVMe accelerator Lustre File System with 6 PB capacity
Compute Nodes
Number of Nodes	636	990
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz	2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz
Cores per Node	192	128
Main Memory	768 GiB	256 GiB
Large Memory Nodes
Number of Nodes	48	66
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz	2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz
Cores per Node	192	128
Main Memory	1536 GiB	1024 GiB
Huge Memory Nodes
Number of Nodes	-	5
CPUs per Node	-	2x AMD Milan 7713, 2.0 GHz, , up to 3.675 GHz
Cores per Node	-	128
Main Memory	-	2048 GiB
Local Storage	-	34 TiB SSD-based memory 12x 3.2 TB NVMe SSDs, ~70 GB/s
GPU Nodes
Number of Nodes	27	32
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz	2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz
Cores per Node	192	128
Main Memory	768 GiB	512 GiB
Accelerators per Node	4x NVIDIA H100 94GB	4x NVIDIA A100 with NVLink and 40 GB HBM2
GPU-Development Nodes
Number of Nodes	-	1
CPUs per Node	-	2x AMD EPYC Rome 7742, 2.25 GHz, up to 3.4 GHz
Cores per Node	-	128
Main Memory	-	1024 GiB
Accelerators per Node	-	8x NVIDIA A100 with NVLink and 40GB HBM2
FPGA Nodes
Number of Nodes	32	36
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz	2x AMD Milan 7713, 2.0 GHz, up to 3.675 GHz
Cores per Node	192	128
Main Memory	768 GiB	512 GiB
Accelerators per Node	pilot installation with 1 node with 3x AMD Alveo V80, expansion TBA	16 nodes with each 3x Xilinx Alveo U280 FPGA with 8GiB HBM2 and 32GiB DDR memory 16 nodes with each 2x Intel Stratix 10 GX 2800 FPGA with 32 GiB DDR memory (Bittware 520N cards)
HACC Nodes
Number of Nodes	-	3
CPUs per Node	-	2x AMD Milan 7V13, 2.0 GHz, up to 3.675 GHz
Cores per Node	-	128
Main Memory	-	512 GiB
Accelerators per Node	-	2x Xilinx Alveo U55C 2x Xilinx VCK5000 4x AMD Instinct MI210
FPGA-to-FPGA Communication Networks
Optical Switch	-	CALIENT S320 Optical Circuit Switch (OCS), 320 ports
Ethernet Switch	-	Huawei Cloudengine CE9860: 128-Port Ethernet Switch