Forgejo runners in custom environments

The forgejo runner is available as a self-contained binary from https://code.forgejo.org/forgejo/runner/releases. The xz-compressed download is ~5MB, and only needs to be extracted and given execute permissions to become usable.

Runners can be used in "daemon" mode, where they keep running continuously asking for new jobs. A useful alternative is "one-job" mode, whether the runner accepts and executes one job, and exits afterwards.

Personal (host) environment

Runners can execute jobs in the host environment directly, with the permissions of the user that started the runner. This has security implications, but is also useful, for example, when jobs needs to be executed in specifically crafted environments.

This mode should only be used for access-restricted runners registered for a specific user account only.

To register a runner, visit https://cerebra.fz-juelich.de/user/settings/actions/runners/new. Provide a name and an optional description, and copy the configuration file snippet with the access credentials that is shown after runner registration. The credentials in this file enable (other) runners to accept/intercept workflow executions. Therefore they must be handled like credentials with the appropriate access control.

A working, minimal configuration file can look like the following:

runner:
  capacity: 1
  insecure: false
  labels: 
    - meiner:host

host:
  # The parent directory of a job's working directory.
  # If it's empty, $HOME/.cache/act/ will be used.
  workdir_parent:

server:
  connections:
    cerebra:
      url: https://cerebra.fz-juelich.de/
      uuid: ........-....-....-....-............
      token: ........................................

When this file is saved as runner-config.yaml locally, the runner can be started with

./forgejo-runner-linux-amd64 -c runner-config.yaml one-job -w

and it will check in at the forgejo site, and wait for a job to be assigned to it, execute that job, and then exit immediately.

Importantly, the runner will only accept jobs that declare to run on meiner -- an arbitrarily chosen label.

A minimal workflow specification to match this setup would be

on:
  workflow_dispatch:

jobs:
  demo:
    runs-on: meiner
    steps:
      - run: uname -a

Shared/cluster/HPC environments

The approach described above can also be combined with a job scheduler or batch system. Instead of submitting a specialized compute job, a (single-use) runner is submitted that fetches a job from the actions queue. Below are examples for some use cases and their implementation with a specific batch system. All examples assume the runner executable and a runner configuration to be available.

Because workflows are executed in the environment and with the permissions of the user that submits the runner jobs, this approach is also only appropriate for access restricted runners registered for personal accounts or sufficiently access restricted organizations.

One-off runner with HTCondor

A low-overhead approach to scheduling a runner for a one-job execution is using condor_run. The example assumes a shared filesystem between submit host and compute host (to provision executable and configuration file). The runner would exit immediately when there are no action workflows queued, unless -w is given as a runner parameter.

condor_run \
  -a request_memory=2048 \
  ./forgejo-runner-linux-amd64 \
    -c juseless-config.yaml \
    -w one-job

Generic runner handling with HTCondor

Here is a suggestion for a more generic setup: Create a dedicated directory with all components (here ~/forgejo-runner). Place the runner executable into this directory together with the runner configuration file. In addition, create an HTCondor submit file with the following content:

# runner resource request
request_cpus = 1
request_memory = 1G
# avoid shared file system dependency
transfer_input_files = config.yaml
executable = forgejo-runner-linux-amd64
should_transfer_files = YES
# runner parameters (remove -w to not wait for a yet-to-be-queued workflow)
arguments = -c config.yaml one-job
# start runner in a clean environment
getenv = False
environment = "PATH=/usr/sbin:/usr/bin:/sbin:/bin"
# disable transfer of runner logs, not interested, workflow logs
# go directly to forgejo
when_to_transfer_output = NEVER
# queue any number of runners
universe = vanilla
queue 1

This declares a runner job with a particular (customizable) resource request. It instructs HTCondor to transfer runner executable and configuration file to the compute node, and to start the runner in one-job mode. The submitting user's environment is not inherited by the runner job to constraint the information the runner's payloads have access to.

With this setup, it is important to have action runs to be queue at the forgejo site before the runner is submitted as a batch job.

Using many workflow runners with HTCondor

When it is known that many workflows need to be executed, HTCondor's DAGMan feature can be useful. Extend the generic approach above by placing the following script into the same directory as the submit file, and make the script executable.

#!/bin/bash

set -euo pipefail

if [[ $# -ne 2 ]]; then
    echo "Usage: $0 TOTAL_JOBS MAXJOBS" >&2
    exit 1
fi

total_jobs="$1"
max_jobs="$2"

# Resolve the directory containing this script.
script_dir="$(dirname -- "${BASH_SOURCE[0]}")"
cd "$script_dir"
# prevent accumulation of log files from previous runs
rm runner.dag.*

dag_file="runner.dag"

seq -w 1 "$total_jobs" |
while read -r job_number; do
    printf 'JOB job%s runner.submit\n' "$job_number"
done > "$dag_file"

condor_submit_dag -batch-name forgejo-runner -maxjobs "$max_jobs" "$dag_file"

This will generate a job DAG of a given number of independent runner jobs, and will configure HTCondor to run no more than a specific number of them at the same time.