Many software projects make use of CI/CD to automate certain jobs and integrate them into the development workflow with GitLab. Therefore, a runner is required to execute these jobs on a remote machine. In the following, it is described how such a runner can be set up on our systems to utilize the compute nodes for such CI jobs.

Setup of GitLab CI Runner with Jacamar CI

Jacamar CI is an HPC-focused CI/CD driver which allows scheduling of CI/CD jobs as jobs within a workload manager like Slurm. The advantage of running the CI/CD jobs directly on the cluster is the availability of all software and also the hardware (i.e. FPGAs) of the systems for these jobs.

For the setup, we need to install a GitLab runner that will continuously run in the user space. This runner will be registered in a GitLab project and execute incoming CI/CD jobs using Jacamar CI as a custom executor.

Log in to one of the cluster frontends and create a new folder for the CI (preferably on the parallel file system). Change into the created directory. This folder will - at the end of this guide - contain all required configuration files.

A good location is a subdirectory in the project path assigned to your project in /scratch/.... For example:

1. Setup Environment with Modules

Jacamar CI and GitLab-Runner are available within our modules system. In order to load the modules you can use

module reset
module load tools
module load gitlab-runner
module load jacamar

This loads the gitlab-runner and jacamar binaries in the latest versions into your environment. In the following setup you need the absolute paths of the binaries. These are

Path to Latest GitLab Binary

/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/gitlab-runner

Path to Latest Jacamar

/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/jacamar

2. Registration of GitLab Runner

Now we need to setup CI/CD in your Gitlab project. Go to the settings of your project on Gitlab and enable the feature CI/CD in the General section under Visibility, project features, permissions.
In the now appearing CI/CD section (under Settings), go to Runners. There you will find a two step setup guide to connect a new runner to your project by clicking on New project runner.

To register the new runner and generate a new configuration file execute the Gitlab runner on Noctua inside your CI Configuration directory:

gitlab-runner register --config=jacamar-config.toml

Follow the steps. Enter the instance URL and registration token from the GitLab page. If you are asked for the executor type, choose custom.

Afterwards, the file jacamar-config.toml was created.

3. Make GitLab Runner use Jacamar CI

Now we need to configure the GitLab runner to use our custom executor jacamar we configured in step 2.

Therefore, edit the configuration file jacamar-config.toml.

First add the following lines to the top of the file:

[general]
executor = "slurm"
data_dir = "/scratch/PATH/TO/WORK/DIR/.../data"

You may adjust the path to the data directory accordingly. Note, that this data will only be temporal for the execution of a job. However, keep in mind the specified path should be accessible from all compute nodes.

Inside your runners definition, add the two following lines

pre_get_sources_script="module reset"

This will load the default modules of Noctua including Slurm, which is required for the custom executor.

Also, Jacamar CI requires at least git version +2.9. The pre-installed system version of git is sufficient.

To also allow artifact uploads to the GitLab server, gitlab-runner must be in the PATH variable. This requires setting the PATH variable in the environment settings of the configuration. Again, replace the /PATH/TO/WORK/DIR with the path to your working directory

environment = ["PATH=/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/opt/software/pc2/EB-SW/software/jacamar/latest/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"]

Now we need to configure Jacamar CI as a custom executor by editing the following fields in the configuration. /PATH/TO/WORK/DIR/ is the absolute path to the directory you have set up Jacamar CI and the GitLab runner.

[runners.custom]
config_exec_timeout = 3600 
config_exec  = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
config_args  = ["--no-auth", "config", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]
prepare_exec = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
prepare_args = ["--no-auth", "prepare"]
run_exec     = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
run_args     = ["--no-auth", "run"]
cleanup_exec = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
cleanup_args = ["--no-auth", "cleanup", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]

The config should look similar to this when you are done

[general]
executor = "slurm"
data_dir = "/scratch/PATH/TO/WORK/DIR/.../data"

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "Jacamar Test Runner"
url = "https://git.uni-paderborn.de/"
token = "TOKEN"
executor = "custom"
environment = ["PATH=/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"]
pre_get_sources_script="module reset"
[runners.cache]
  MaxUploadedArchiveSize = 0
[runners.custom]
config_exec_timeout = 3600 
config_exec  = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
config_args  = ["--no-auth", "config", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]
prepare_exec = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
prepare_args = ["--no-auth", "prepare"]
run_exec     = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
run_args     = ["--no-auth", "run"]
cleanup_exec = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
cleanup_args = ["--no-auth", "cleanup", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]

You may want to increase the value of concurrent to allow the GitLab runner to schedule multiple jobs at once. That's it.

To test the GitLab runner, execute it with the jacamar configuration:

gitlab-runner run --config=jacamar-config.toml

In the runners list of your repository (Settings -> CI/CD -> Runners) you will find the status of your runner under Assigned project runners.

As long as your GitLab runner is executed on Noctua, it will process the CI jobs from your project.

4. Add CI file to repository

To run a CI job, you only need to create a .gitlab-ci.yml file in your project and commit it to GitLab. GitLab will use your new runner for the jobs.

In the .gitlab-ci.yml you will need to specify the variable SCHEDULER_PARAMETERS to make it work with our Slurm installation. In this variable, you should specify your project account and the partition where the jobs should be executed.

Example

test:
  stage: build
  variables:
    SCHEDULER_PARAMETERS: "-A PROJECT_ACCOUNT -p normal -t 0:05:00"
  script:
    - echo "Hello from " $(hostname)

Change the PROJECT_ACCOUNT to the name of your project (The name that you usually pass to sbatch via the -A option).

5. Execute GitLab Runner

The GitLab runner needs to be executed to fetch new CI jobs from GitLab. The best way is to use a systemd service which can restart the runner after a reboot of the frondend nodes.

Create a systemd user service file in your $HOME directory at .config/systemd/user/name.service which looks like the following example.

[Unit]
Description=Jacamar User Service
After=network.target

[Service]
ExecStart=/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/gitlab-runner run --config=/PATH/TO/WORK/DIR/jacamar-config.toml
WorkingDirectory=/scratch/PATH/TO/WORK/DIR/

[Install]
WantedBy=default.target

The service can be enabled with systemctl --user enable name.service and started with systemctl --user start name.service. Use systemctl --user daemon-reload after changing the service file.

Now, schedule a pipeline in your GitLab project. The runner will fetch the created jobs and execute them on the specified partition.

Shared runners can only be created for the whole Gitlab instance. Group runner can be created for groups where the owner status is available. In every other case register a runner and create a service file for each repository.

Troubleshooting

Q: I have created a runner in my project but no pipelines are executed. What can be the problem?

A: If you have specified a tag when setting up the runner you will need to also use the tag in your CI jobs or allow the runner to also execute untagged jobs via the runner settings under Settings -> CI/CD -> Runners.

Q: Is it possible to use the runner for multiple projects?

A: Yes! In your GitLab project, go to Settings -> CI/CD -> Runners and click on the pencil next to your runner to edit its settings. Uncheck the option Lock to current projects. Now when enabling CI/CD in another project, you should be able to select the runner instead of creating a new one.

Q: My jobs are running successfully but fail on completion!

A: The runner is known to fail if a ~/.bash_logout exists in your home directory that contains a clear.