Help › Guide

Reqlyn in CI

Reqlyn 0.5.1 is not released yet. This help describes it ahead of release.

A pipeline runs the reqlyn command line in the Reqlyn CI image: an operation's regression cases (reqlyn test), a data-driven run (reqlyn run --data), a mock for the job to call (reqlyn mock), and runs on a server (reqlyn remote). The image holds the program and nothing else: no secret, no token and no server's address. The repository has a GitHub Action (ci/github) and a GitLab template (ci/gitlab/reqlyn.gitlab-ci.yml) that run it by a name you set.

The image

Build it on a computer that builds Reqlyn, from the repository's root:

powershell -File deploy\ci\build.ps1 -Image registry.example/reqlyn-ci:0.1.0
docker push registry.example/reqlyn-ci:0.1.0

The build vendors every dependency first from what Cargo already holds (cargo --offline vendor), so the container needs no network and no SSH key for the shared crates; build the workspace once before. The image is Debian with the program at /usr/local/bin/reqlyn, a shell, and REQLYN_DATA_DIR=/tmp/reqlyn, so each job starts with an empty data folder.

Signing it

The image is signed in its registry, by digest, with a cosign key made once on the build computer:

powershell -File deploy\ci\signing-key.ps1
powershell -File deploy\ci\sign.ps1 -Image registry.example/reqlyn-ci:0.1.0

signing-key.ps1 makes the key pair with a password of its own; the private key and the password are kept in Windows Credential Manager (dev.reqlyn.ci-signing/key and dev.reqlyn.ci-signing/password), never on disk, and the public key is deploy/ci/cosign.pub in the repository. It refuses to replace a key that is kept. sign.ps1 signs the pushed image's digest; the signature is stored beside the image in that registry and nothing is sent to a public transparency log. Anyone with the repository checks it before running the image:

cosign verify --key deploy/ci/cosign.pub --insecure-ignore-tlog=true registry.example/reqlyn-ci:0.1.0

Both scripts need cosign on the path.

Secrets in a job

A CI container has no secret store. There, a secret variable's value is read from the job's environment: REQLYN_SECRET_ and the variable's name in capitals, each character that is not a letter or a digit _. A secret variable api-key of any environment is REQLYN_SECRET_API_KEY. It is read only when the operating system's secret store cannot be reached, as in the image, and is never written anywhere. Only secret variables are given so: a credential's secret, a client certificate's key and the rest are not. Give the value from the pipeline's own secrets (a GitHub secret, a masked GitLab variable), never on a command line. The remote commands take REQLYN_SERVER and REQLYN_TOKEN the same way.

A folder workspace in the checkout (Requests) is named with REQLYN_FOLDER; its environments' files name their secret variables with no value, and the job gives each.

GitHub Actions

Copy ci/github into the repository (or refer to it where it is kept), and give it the image's name and the commands:

jobs:
  cases:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./ci/github
        with:
          image: ${{ vars.REQLYN_IMAGE }}
          run: |
            reqlyn arazzo save workflows/device.arazzo.yaml
            reqlyn test Device --junit reqlyn-cases.xml
Input For
image The CI image, by the name it was pushed as
run The commands, as a shell script run in the image; the step fails when one ends other than 0
folder A folder workspace in the checkout, relative to it, given as REQLYN_FOLDER

The step runs the image with the checkout mounted as its working folder, as the runner's own user, so a report a command writes is in the checkout. Each REQLYN_SECRET_<NAME>, REQLYN_SERVER and REQLYN_TOKEN in the step's env is passed into the container by name. The runner needs Docker, as ubuntu-latest has.

GitLab CI

Copy ci/gitlab/reqlyn.gitlab-ci.yml into the repository, include it, set REQLYN_IMAGE, and extend .reqlyn:

include:
  - local: ci/reqlyn.gitlab-ci.yml
variables:
  REQLYN_IMAGE: registry.example/reqlyn-ci:0.1.0
cases:
  extends: .reqlyn
  script:
    - reqlyn arazzo save workflows/device.arazzo.yaml
    - reqlyn test Device --junit reqlyn-cases.xml

The job runs in the image; a report named reqlyn-*.xml is kept and shown as the job's tests.

A gate: rows against a mock, then load

A job can serve a mock from the repository's own description, run an operation once for each row of a data file against it, then load it, held to thresholds, with nothing else to reach:

      - uses: ./ci/github
        with:
          image: ${{ vars.REQLYN_IMAGE }}
          run: |
            reqlyn mock workflows/device.inventory.openapi.yaml --port 8787 > /tmp/mock.log 2>&1 &
            for i in 1 2 3 4 5 6 7 8 9 10; do grep -qs 127.0.0.1 /tmp/mock.log && break; sleep 1; done
            reqlyn arazzo save workflows/device.arazzo.yaml
            reqlyn run Device --data workflows/devices.csv --var inventory=http://127.0.0.1:8787 --junit reqlyn-data.xml
            reqlyn load Device --callers 4 --seconds 10 --input '{"deviceId": 4831}' --var inventory=http://127.0.0.1:8787 --p95-under 250 --errors-under 1 --junit reqlyn-load.xml

The mock is served inside the job's container, so the load measures Reqlyn and the mock, not an API: it shows a threshold holds and that the job fails when it does not. Against a real test API, give its address in place of the mock's.

What is checked

bash deploy/ci/check.sh <image> runs the fixture repository (fixtures/ci) as the Action and the template do: its cases hold (0); its workflow broken so an output reads the wrong field fails (1, the report saying CASE_DIFFERS); the template's job runs in the image's shell; and a folder workspace's run whose secret variable inventory_key is not given fails (1) and, given as REQLYN_SECRET_INVENTORY_KEY, succeeds (0), its mock serving in the same job; the gate above holds (0, three rows and the load within both thresholds), and fails (1) with its mock slowed past the p95 threshold (--delay 300), saying Over the threshold p95 under 250 ms. GitHub and GitLab themselves did not run it.

Help for Reqlyn 0.5.1. All help topics