Page Comparison

...

Jacamar CI is an HPC-focused CI/CD driver which allows scheduling of CI/CD jobs as jobs within a workload manager like Slurm. The advantage of running the CI/CD jobs directly on the cluster is the availability of all software and also the hardware (i.e. FPGAs) of the systems for these jobs.

For the setup, we need to install a GitLab runner that will continuously run in the user space. This runner will be registered in a GitLab project and execute incoming CI/CD jobs using Jacamar CI as a custom executor.

Log in to one of the cluster frontends and create a new folder for the CI (preferably on the parallel file system). Change into the created directory. This folder will - at the end of this guide - contain all required data and executablesconfiguration files.

A good location is the directory assigned to your project in /scratch/....

1.

...

Setup Environment with Modules

Jacamar CI releases and download the most recent RPM package. The version - w/Capabilites is not necessary, because all following will be done in user space.

For version 0.12.1 you can use the following link

curl -L --output jacamar.rpm "https://gitlab.com/api/v4/projects/13829536/packages/generic/jacamar-ci/0.12.1/jacamar-ci-0.12.1-1.el7.x86_64.rpm"

Put the RPM file into your working directory and rename it to jacamar.rpm. Use the following command to extract the Jacamar CI binary

rpm2cpio jacamar.rpm | cpio -idmv > jacamar

The binary is now found at ./opt/jacamar/bin/jacamar. jacamar-auth is not necessary. Beware that the folder jacamar is not writeable by default.

You can delete the RPM package now.

Now we and GitLab-Runner are available within our modules system. In order to load the modules you can use

module reset
module load tools
module load gitlab-runner
module load jacamar

This loads the gitlab-runner and jacamar binaries in the latest versions into your environment. In the following setup you need the absolute paths of the binaries. These are

Path to Latest GitLab Binary

/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/gitlab-runner

Path to Latest Jacamar

/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/jacamar

2. Configuration of Jacamar CI Custom Executor

We need to create a configuration for Jacamar CI. Create a new file named jacamar-config.toml and insert the following content

[general]
executor = "slurm"
data_dir = "/scratch/PATH/TO/WORK/DIR/.../data"

You may adjust the path to the data directory accordingly. Note, that this data will only be temporal for the execution of a job. However, keep in mind the specified path should be accessible from all compute nodes!.

...

3.

...

Configuration of GitLab Runner

...

Download the most recent version of the official GitLab runner and make it executable:

...

Now we need to setup CI/CD in your Gitlab project.
Go to the settings of your project on Gitlab and enable the feature CI/CD in the General section under Visibility, project features, permissions.
In the now appearing CI/CD section (under Settings), go to Runners. There you will find a two step setup guide to connect a new runner to your project under the Specific Runner heading.

To execute these two steps run the Gitlab runner on Noctua

Code Block
./usr/bin/gitlab-runner register --config=jacamar-config.toml

Follow the steps. If you are asked for the executor type, choose custom.

...

4. Make GitLab Runner use Jacamar CI

Now we need to configure the GitLab runner to use our custom executor jacamar we installed configured in step 12.

Therefore, edit the configuration file gci-config.toml.

...

This will load the default modules of Noctua including Slurm, which is required for the custom executor.

Also, Jacamar CI requires at least git version +2.9. The pre-installed system version of git should be is sufficient.

To also allow artifact uploads to the GitLab server, gitlab-runner must be in the PATH variable. This requires setting the PATH variable in the environment settings of the configuration. Again, replace the /PATH/TO/WORK/DIR with the path to your working directory

environment = ["PATH=/PATH/TO/WORK/DIR/usr/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"]

...

[runners.custom]
config_exec = "/opt/software/PATHpc2/TOEB-SW/WORKsoftware/DIRgitlab-runner/optlatest/bin/jacamar"
config_args = ["config","--no-auth", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]
prepare_exec = "/PATH/TO/WORK/DIR/optopt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/jacamar"
prepare_args = ["prepare", "--no-auth"]
run_exec = "/PATH/TO/WORK/DIR/optopt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/jacamar"
run_args = ["run", "--no-auth"]
cleanup_exec = "/opt/software/PATHpc2/TOEB-SW/WORKsoftware/DIRgitlab-runner/optlatest/bin/jacamar"
cleanup_args = ["cleanup", "--no-auth", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]

...

[general]
executor = "slurm"
data_dir = "/scratch/PATH/TO/WORK/DIR/.../data"

concurrent = 1
check_interval = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "Jacamar Test Runner"
url = "https://git.uni-paderborn.de/"
token = "TOKEN"
executor = "custom"
environment = ["PATH=/PATH/TO/WORK/DIR/usr/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"]
pre_clone_script="module reset"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.custom]
config_exec = "/opt/software/PATHpc2/TOEB-SW/WORKsoftware/DIRgitlab-runner/usrlatest/bin/jacamar"
config_args = ["config","--no-auth", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]
prepare_exec = "/PATH/TO/WORK/DIR/usropt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/jacamar"
prepare_args = ["prepare", "--no-auth"]
run_exec = "/opt/software/PATHpc2/TOEB-SW/WORKsoftware/DIRgitlab-runner/usrlatest/bin/jacamar"
run_args = ["run", "--no-auth"]
cleanup_exec = "/opt/PATH/TO/WORK/DIR/usrsoftware/pc2/EB-SW/software/gitlab-runner/latest/bin/jacamar"
cleanup_args = ["cleanup", "--no-auth", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]

...

In the .gitlab-ci.yml you will need to specify the variable SCHEDULER_PARAMETERS to make it work with our Slurm installation. In this variable, you should specify your project account and the partition where the jobs should be executed.

Example

test:
  stage: build
  variables:
    SCHEDULER_PARAMETERS: "-A PROJECT_ACCOUNT -p normal -t 0:05:00"
  script:
    - echo "Hello from " $(cat /etc/hostname)

Change the PROJECT_ACCOUNT to the name of your project (The name that you usually pass to sbatch via the -A option).

...

5. Execute GitLab Runner

The GitLab runner needs to be executed to fetch new CI jobs from GitLab. The best way is to use a systemd service which can restart the runner after a reboot of the frondend nodes.

Create a systemd user service file in your $HOME directory at .config/system/user/name.service which looks like the following example.

[Unit]
Description=Jacamar User Service
After=network.target

[Service]
ExecStart=/PATH/TO/WORK/DIRopt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/gitlab-runner run --config=/PATH/TO/WORK/DIR/jacamar-config.toml
WorkingDirectory=/scratch/PATH/TO/WORK/DIR/

[Install]
WantedBy=default.target

...