System Architecture

Systems Detail Pages

If you want to move or migrate compute resources between clusters or get access to other cluster, follow this description.

We also have more detailled descriptions in our system papers:

Systems Comparison Table

	Otus		Noctua 2

	Otus		Noctua 2
System	Lenovo		Atos BullSequana XH2000
Processor Cores	142,656		143,872
Total Main Memory	593,3 TiB		347.5 TiB
Floating-Point Performance	CPU: 11,44 PFLOPS DP Peak (8,42 PFlop/s Linpack) GPU: 7,45 PFLOPS DP Peak (4,65 PFlop/s Linpack)		CPU: 5.4 PFLOPS DP Peak (4.19 PFlop/s Linpack) GPU: 2.49 PFLOPS DP Tensor Core Peak (ca. 1.7 PFlop/s Linpack)
Cabinets	11 racks - all components with direct liquid cooling, 2 racks - all processors with direct liquid cooling, 6 racks - air cooling		12 racks - direct liquid cooling, 7 racks - air cooling, four of them with active backdoor cooling
Communication Network CPUs	Nvidia NDR InfiniBand fabric, blocking factor 2:1 compute/FPGA nodes connected with 200Gbps GPU nodes connected with 2x400Gbps		Mellanox InfiniBand 100/200 HDR, 1:2 blocking factor
Storage System	IBM Spectrum Scale > 5 PB capacity (Flash + Hard disks)		DDN Exascaler 7990X with NVMe accelerator Lustre File System with 6 PB capacity
Compute Nodes
Number of Nodes	636		990
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz		2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz
Cores per Node	192		128
Main Memory	768 GiB		256 GiB
Large Memory Nodes
Number of Nodes	48		66
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz		2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz
Cores per Node	192		128
Main Memory	1536 GiB		1024 GiB
Huge Memory Nodes
Number of Nodes	-		5
CPUs per Node	-		2x AMD Milan 7713, 2.0 GHz, , up to 3.675 GHz
Cores per Node	-		128
Main Memory	-		2048 GiB
Local Storage	-		34 TiB SSD-based memory 12x 3.2 TB NVMe SSDs, ~70 GB/s
GPU Nodes
Number of Nodes	27	10	32
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz		2x AMD Milan 7763, 2.45 GHz, up to 3.5 GHz
Cores per Node	192		128
Main Memory	768 GiB		512 GiB
Accelerators per Node	4x NVIDIA H100 94GB (H100 nodes)	3x NVIDIA A40 48GB (A40 nodes)	4x NVIDIA A100 with NVLink and 40 GB HBM2
GPU-Development Nodes
Number of Nodes	-		1
CPUs per Node	-		2x AMD EPYC Rome 7742, 2.25 GHz, up to 3.4 GHz
Cores per Node	-		128
Main Memory	-		1024 GiB
Accelerators per Node	-		8x NVIDIA A100 with NVLink and 40GB HBM2
FPGA Nodes
Number of Nodes	22		36
CPUs per Node	2x AMD Turin 9655, 2.6 GHz, up to 4.5 GHz		2x AMD Milan 7713, 2.0 GHz, up to 3.675 GHz
Cores per Node	192		128
Main Memory	768 GiB		512 GiB
Accelerators per Node	pilot installation with 1 node with 3x AMD Alveo V80, expansion TBA		16 nodes with each 3x Xilinx Alveo U280 FPGA with 8GiB HBM2 and 32GiB DDR memory 16 nodes with each 2x Intel Stratix 10 GX 2800 FPGA with 32 GiB DDR memory (Bittware 520N cards)
HACC Nodes
Number of Nodes	-		3
CPUs per Node	-		2x AMD Milan 7V13, 2.0 GHz, up to 3.675 GHz
Cores per Node	-		128
Main Memory	-		512 GiB
Accelerators per Node	-		2x Xilinx Alveo U55C 2x Xilinx VCK5000 4x AMD Instinct MI210
FPGA-to-FPGA Communication Networks
Optical Switch	-		CALIENT S320 Optical Circuit Switch (OCS), 320 ports
Ethernet Switch	-		Huawei Cloudengine CE9860: 128-Port Ethernet Switch