...

/opt/software/pc2/EB-SW/software/gitlab-runnerjacamar/latest/bin/jacamar

2. Registration of GitLab Runner

...

Now we need to configure the GitLab runner to use our custom executor jacamar we configured in step 2.

Therefore, edit the configuration file jacamar-config.toml.

...

The config should look similar to this when you are done

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[general]
executor = "slurm"
data_dir = "/scratch/PATH/TO/WORK/DIR/.../data"

concurrent
= 1
check_interval = 0
shutdown_timeout = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "Jacamar Test Runner"
url = "https://git.uni-paderborn.de/"
token = "TOKEN"
executor = "custom"
limit = 0
request_concurrency = 1
environment = ["PATH=/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"]
pre_get_sources_script="module reset"
[runners.cache]
  MaxUploadedArchiveSize = 0
[runners.custom]
config_exec_timeout = 3600 
config_exec  = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
config_args  = ["--no-auth", "config", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]
prepare_exec = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
prepare_args = ["--no-auth", "prepare"]
run_exec     = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
run_args     = ["--no-auth", "run"]
cleanup_exec = "/opt/software/pc2/EB-SW/software/jacamar/latest/bin/jacamar"
cleanup_args = ["--no-auth", "cleanup", "--configuration", "/scratch/PATH/TO/WORK/DIR/jacamar-config.toml"]

You may want to increase the value of concurrent to allow the GitLab runner to schedule multiple jobs at once. That's it.

To test the GitLab runner, execute it with the jacamar configuration:

...

As long as your GitLab runner is executed on Noctua, it will process the CI jobs from your project.

Concurrent CI job execution

There are multiple layers of concurrency in the runner configuration. You can define multiple [[runners]] sections to let Jacamar create multiple runners with different configurations, but this is not required for concurrency.

The concurrent variable at the top defines the total limit for all runners combined. Further, each runner has two variables in its section: limit and request_concurrency. Both limit the number of concurrent CI jobs a runner will execute.

For example, if you have 1 runner, set concurrent to 5, set limit to 0 to deactivate the limit variable and set request_concurrency to 5 to execute at most 5 CI jobs concurrently.

You can find more information in the respective documentation of Jacamar variables and Gitlab Runner variables.

4. Add CI file to repository

...

In the .gitlab-ci.yml you will need to specify the variable SCHEDULER_PARAMETERS to make it work with our Slurm installation. In this variable, you should specify your project account and the partition where the jobs should be executed.

Also the id_tokens variable has to be specified as demonstrated in the example (context).

Example

test:
  stage: build
  id_tokens:
    CI_JOB_JWT:
      aud: https://git.uni-paderborn.de
  variables:
    SCHEDULER_PARAMETERS: "-A PROJECT_ACCOUNT -p normal -t 0:05:00"
  script:
    - echo "Hello from " $(hostname)

...

The GitLab runner needs to be executed to fetch new CI jobs from GitLab. The best way is to use a systemd service which can restart the runner after a reboot of the frondend nodes.

Create a systemd user service file in your $HOME directory at .config/systemd/user/name.service which looks like the following example.

[Unit]
Description=Jacamar User Service
After=network.target

[Service]
ExecStart=/opt/software/pc2/EB-SW/software/gitlab-runner/latest/bin/gitlab-runner run --config=/PATH/TO/WORK/DIR/jacamar-config.toml
WorkingDirectory=/scratch/PATH/TO/WORK/DIR/

[Install]
WantedBy=default.target

The service can be enabled with systemctl --user enable name.service and started with systemctl --user start name.service. Use systemctl --user daemon-reload after changing the service file.

Now, To make the gitlab-runner continue executing after closing your SSH session, you can use screen or tmux.

The login nodes may be rebooted, e.g. during maintenance or an update. Afterwards, you need to login and start the gitlab-runner anew.

Now, you can schedule a pipeline in your GitLab project. The runner will fetch the created jobs and execute them on the specified partition.

Shared runners can only be created for the whole Gitlab instance. Group runner can be created for groups where the owner status is available. In every other case register a runner and create a service file for each repository.

Troubleshooting

...