AiiDA (https://www.aiida.net/) can be used in a number of different ways with PC2 HPC systems:

1. Running AiiDA on a different system, such as a local computer, and submitting calculations to PC2 HPC systems.

2. Running calculations, as well as AiiDA itself, on PC2 HPC systems.

Because these two situations need different setup steps, we will describe them independently below. We recommend the first possibility, i.e., running AiiDA on a local system and submitting calculations to PC2 HPC system because a local setup gives you more configuration freedom and also the possibility to, at the same time, submit also to other HPC systems. However, if you donÄt have a suitable local computer we include here also a guide for the second possibility of running AiIDA completely on PC2 HPC systems.

The two setups have individual advantages and disadvantages:

Running AiiDA on a Local System and Submitting Calculations to PC2 HPC Systems

This setup assumes that you are running AiiDA on a local computer or server outside of PC2. For the setup of AiIDA on the local computer follow the detailed guide https://aiida.readthedocs.io/projects/aiida-core/en/stable/installation/index.html. In case of questions or problems or questions please don't hesitate to contact us via pc2-support@uni-paderborn.de.

Create a Computer

To be able to submit calculations to a PC2 HPC system, create a computer in AiIDA by running the following on your local system where AiiDA is running:

verdi computer setup --label Noctua2 --description "Noctua2 Slurm" --scheduler core.slurm --hostname n2login1 --transport core.ssh --work-dir "[work dir]" --mpirun-command srun --shebang '#!/bin/bash' --mpiprocs-per-machine 128 --default-memory-per-machine 240000000 --not-use-double-quotes --prepend-text "" --append-text ""

Replace:

• [work dir] with a path to a directory on the parallel file system that AiiDA is going to use as a temporary working directory for your jobs, e.g., /scratch/hpc-prf-[project abbreviation]/{username}/AiiDA/scratch. {username} is automatically replaced by AiiDA with your user name.

Configure the Computer

For access via ssh the computer needs to be configured by running:

verdi -p [AiiDA profile name] computer configure core.ssh Noctua2

with the following settings:

User name: [user name]
Port number: 22
Look for keys [Y/n]: n
SSH key file []: [path to your ssh private key]
Connection timeout in s [60]: 60
Allow ssh agent [Y/n]: Y
SSH proxy jump []: [user name]@fe.noctua2.pc2.uni-paderborn.de
SSH proxy command []:
Compress file transfers [Y/n]: Y
GSS auth [False]:
GSS kex [False]:
GSS deleg_creds [False]:
GSS host [n2login1]:
Load system host keys [Y/n]: Y
Key policy (RejectPolicy, WarningPolicy, AutoAddPolicy) [RejectPolicy]: WarningPolicy
Use login shell when executing command [Y/n]:
Connection cooldown time (s) [30.0]: 

Replace:

• [path to your ssh private key]: with the path to your SSH key that is enabled for login to PC2 cluster systems. Please note that the ssh-agent is required so that password-protected ssh-keys work with AiiDA. Password-protected ssh-keys are required for PC2 HPC systems. See also our guide on SSH keys.

• [user name]: your PC2 user name

Testing the Computer

Run verdi computer test Noctua2 to test the configuration. In case of questions or problems or questions please don't hesitate to contact us via pc2-support@uni-paderborn.de.

Create a Code

The next steps show with the example of Quantum Espresso pw.x how you can use the existing software on PC2 HPC systems with AiiDA. In this example we use Quantum Espresso from the module chem/QuantumESPRESSO/7.3-foss-2023a.

1. Run AiiDA Quantum Espresso plugins:

   pip install aiida-quantumespresso 
   verdi plugin list aiida.calculations 
   aiida-pseudo install sssp   #for the example below

2. Run verdi code create core.code.installed and enter

   Computer: Noctua2
   Filepath executable: pw.x
   Label: qe_pw_7.3
   Description []: Quantum Espresso 7.3
   Default `CalcJob` plugin: quantumespresso.pw
   Escape using double quotes [y/N]:
   prepend_text: 
     module reset
     module load chem/QuantumESPRESSO/7.3-foss-2023a
     export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
     export OMP_PLACES=cores
     export OMP_PROC_BIND=true

3. Test the code:

   verdi code test qe_pw_7.3@Noctua2

Running a Calculation

1. Import an atomic structure

   wget https://aiida-tutorials.readthedocs.io/en/tutorial-2021-abc/_downloads/1383def58ffe702e2911585fea20e33d/Si.cif
   verdi data core.structure import ase Si.cif
   verdi data core.structure list

2. List structures and their IDs:

   verdi data core.structure list

3. Run the verdi shell (verdi shell) with

   code = load_code("qe_pw_7.3@Noctua2")
   builder = code.get_builder()
   structure = load_node(105) #put here the id of the structure
   builder.structure = structure
   pseudo_family = load_group('SSSP/1.3/PBE/efficiency')
   pseudos = pseudo_family.get_pseudos(structure=structure)
   builder.pseudos = pseudos
   parameters = {
     'CONTROL': {
       'calculation': 'scf',  # self-consistent field
     },
     'SYSTEM': {
       'ecutwfc': 30.,  # wave function cutoff in Ry
       'ecutrho': 240.,  # density cutoff in Ry
     },
   }
   builder.parameters = Dict(dict=parameters)
   KpointsData = DataFactory('core.array.kpoints')
   kpoints = KpointsData()
   kpoints.set_kpoints_mesh([4,4,4])
   builder.kpoints = kpoints
   
   builder.metadata.options.max_wallclock_seconds=3600
   builder.metadata.options.queue_name="normal"
   builder.metadata.options.withmpi=True
   builder.metadata.options.resources = {'num_machines': 1,'num_mpiprocs_per_machine':32,'num_cores_per_mpiproc':4}
   
   #optional
   builder.metadata.options.qos="express"
   builder.metadata.options.account="hpc-prf-exmaple"
   builder.metadata.options.max_memory_kb=... 

The last lines specify the properties of the compute job in terms of partition, time limit, and resources. More options can be found at https://aiida.readthedocs.io/projects/aiida-core/en/stable/topics/calculations/usage.html#options. The meaning is as follows:

• builder.metadata.options.max_wallclock_second: time limit of the job in seconds

• builder.metadata.options.queue_name: name of the partition, e.g., normal, largemem,…

• builder.metadata.options.withmpi: should the job be run with MPI

• builder.metadata.options.resources

  • num_machines: sbatch option -N/--nodes=

  • num_mpiprocs_per_machine: sbatch option --ntasks-per-node=

  • num_cores_per_mpiproc: sbatch option -c/--cpus-per-task=

• builder.metadata.options.account: the name of your compute project, required if you are a member of multiple compute projects

• builder.metadata.options.qos: in case you want to use special QoS like devel or express

• builder.metadata.options.max_memory_kb: maximal memory per node in case you want to overwrite the default values of memory per node
  

4. Run the job with AiiDA

   • If you run AiIDA locally with RabbitMQ:

     from aiida.engine import submit
     calcjob_node = submit(builder)

   • If you run AiiDA locally without RabbitMQ:

     from aiida.engine import launch
     calcjob_node = launch.run(builder)

5. You can monitor your jobs with verdi process list -a

Running AiiDA on PC2 Systems

We describe here the setup of AiiDA on PC2 HPC systems with Sqlite and PostgresSQL but without RabbitMQ. RabbitMQ is problematic on HPC systems because care must be taken about access control (see https://www.rabbitmq.com/docs/access-control). Please contact pc2-support@uni-paderborn.de if you need RabbitMQ.

To run AiiDA on PC2 systems, there are three options:

1. Run AiiDA on one of the login nodes, i.e., n2login[1-6]

2. Run AiiDA on the PC2 Jupyter Hub

3. Run AiiDA in a long-running compute job (at most 21 days)

Basic Setup of AiiDA

1. Load a suitable Python module, e.g.,

   module load lang/Python/3.11.3-GCCcore-12.3.0

2. Create a Python virtual environment and activate it

   python3 -m venv /pc2/groups/hpc-prf-[your project abbreviation]/$USER/AiiDa/venv
   source /pc2/groups/hpc-prf-[your project abbreviation]/$USER/AiiDa/venv

   We recommend placing Python virtual environments in the PC2 groups directories, i.e., /pc2/groups/hpc-prf-[your project abbreviation]

3. install AiiDA

   pip install aiida-core[atomic_tools,docs,tui,notebook]

4. If you want to use AiIDA in the PC2 Jupyter Hub, you need to also register the virtual environment as a Jupyter Hub kernel:

   python -m ipykernel install --user --name AiiDA --display-name "AiiDA"

Note that you can later use

%load_ext aiida 
%aiida

to load the AiiDA Jupyter notebook extension so that you can for example use verdi as

%verdi status

Database Setup

As a database backend, you can either use SQLite or PostgreSQL

Setup for SQLite

1. Create an AiiDA profile

   verdi profile setup core.sqlite_dos --profile-name [AiiDA profile name] --set-as-default --email [your mail] --first-name [your first name] --last-name [your last name] --institution "[your Institution]" --no-use-rabbitmq 

As the directory of the backend, you can choose a directory on the parallel file system.

Setup for PostgreSQL

1. Set up a PostgreSQL database

   1. Load PostgreSQL module

      module load data/PostgreSQL/16.1-GCCcore-12.3.0

   2. Copy sample configuration

      cp /opt/software/pc2/EB-SW/software/PostgreSQL/16.1-GCCcore-12.3.0/share/postgresql/postgresql.conf.sample postgresql.conf

   3. Change the settings in the config postgresql.conf:

      • port = : choose some random port (>1024, <65536)

      • listen_addresses = 'localhost'

      • unix_socket_directories = '/tmp'

      • unix_socket_permissions = 0700

   4. Create and initialize the data directory. The data directory should be on the parallel file system:

      initdb [path to data directory]

   5. Try to start PostgreSQL:

      postgres -c config_file=postgresql.conf -D [path to data directory]

   6. (while PostgreSQL is running) Create database:

      createdb  -p [port number]

   7. (while PostgreSQL is running) Configure PostgreSQL for AiiDA by running psql -p [port number]

      ALTER ROLE [your user name] WITH PASSWORD '[some random password]';
      CREATE USER aiida WITH PASSWORD '[some random password for user aiida]';
      CREATE DATABASE aiida OWNER aiida ENCODING 'UTF8' LC_COLLATE='en_US.UTF-8' LC_CTYPE='en_US.UTF-8';
      GRANT ALL PRIVILEGES ON DATABASE aiida to aiida;

   8. (while PostgreSQL is running) Test database connection:

      psql -h localhost -d aiida -U aiida -W -p [port number]

2. Create an AiiDA profile

   verdi profile setup core.psql_dos --profile-name [AiiDA profile name] --set-as-default --email [your mail] --first-name [your first name] --last-name [your last name] --institution "[your Institution]" --no-use-rabbitmq 

   • PostgreSQL engine [postgresql_psycopg2]:

   • PostgreSQL hostname [localhost]:

   • PostgreSQL port [5432]: [port number]

   • PostgreSQL username: aiida

   • PostgreSQL password: [passwort for user aiida chosen above]

   • PostgreSQL database name: aiida

   • File repository URI: A directory on the parallel file system. it needs to start with file:// , i.e., file:///scratch/hpc-prf-[project abbreviation]/...

Note that the PostgreSQL database needs to be running for AiiDA to work:

• When running AiiDA on one of the login nodes: Run the PostgreSQL database in a tmux or screen session

• When running AiiDA on the PC2 Jupyter Hub: Run PostgreSQL database in a tmux or screen session in the console in Jupyter Lab instance

• When running AiiDA in a long-running compute job: Start PostgreSQL database in the job before AiiDA in the background, i.e.,

  postgres -c config_file=postgresql.conf -D [path to data directory] &

Create a Computer

To be able to submit calculations to a PC2 HPC system, create a computer in AiIDA by running the following on your local system where AiiDA is running:

verdi computer setup --label Noctua2 --description "Noctua2 Slurm" --scheduler core.slurm --hostname localhost --transport core.local --work-dir "[work dir]" --mpirun-command srun --shebang '#!/bin/bash' --mpiprocs-per-machine 128 --default-memory-per-machine 240000000 --not-use-double-quotes --prepend-text "" --append-text ""

Replace:

• [work dir] with a path to a directory on the parallel file system that AiiDA is going to use as a temporary working directory for your jobs, e.g., /scratch/hpc-prf-[project abbreviation]/{username}/AiiDA/scratch. {username} is automatically replaced by AiiDA with your user name.

Configure the Computer

Configure the computer for your AiIDA profile by running:

verdi -p [AiiDA profile name] computer configure core.local Noctua2

• Use login shell when executing command [Y/n]: n

• Connection cooldown time (s) [0.0]:

Testing the Computer

Run verdi computer test Noctua2 to test the configuration. In case of questions or problems or questions please don't hesitate to contact us via pc2-support@uni-paderborn.de.

Then you can follow the steps to create code and run a calculation above.