Help › Guide

Testing and recovery

Reqlyn 0.5.1 is not released yet. This help describes it ahead of release.

An operation is tested on this computer before it is published, and its tests travel with it. This page covers the operations editor's Test run part, the command line's reqlyn run and reqlyn test, and what Reqlyn offers when a run stops part way.

A test run

In the operations editor, the Test run part holds the input of a test, as JSON ("Start from the declared inputs" writes a template), and the actions below. "Test locally" in the header runs the operation as it is in the editor, from this computer, and the part then says how it ended:

  • returned its output: every step did what it should and the output holds its contract;
  • returned part of its output: a step the run may go on without failed, and what it would have given is null ("The run went on without these steps");
  • failed: with the step, the reason and its code;
  • paused before a step, when a breakpoint was reached.

Under it, each step with its requests: method, address, status, size and time, its attempts and why each further one was made.

From the command line, reqlyn run "<operation>" --input '<json>' prints the output as JSON and ends with 0 (whole), 1 (failed), 2 (not run) or 3 (partial). --junit report.xml also writes a JUnit report: a test case for each step and one for the output contract, each failure with its code, secrets masked.

Breakpoints

"Pause a test here" on a step makes a local test stop before it. The Test run part then shows what each earlier step gave and the request about to be sent, filled, with Continue and Stop. A pause belongs to the editor and is not saved with the operation.

Linked to a server, "Share this test" in the Test run part lets others follow each test you run of that operation live, from Runs in a browser: its steps, each request's line as the step makes it (its variables as placeholders) and status, and its pauses, never a body. "Stop sharing" ends it, as does opening another operation.

A saved example

"Keep as example" keeps the last live test as the operation's example: what each step received, with likely secrets masked first. "Replay the example" runs the operation again from it with nothing sent, which shows what a change to mappings or the output contract does without reaching any API. An example that runs out of responses (because a step was added) says so (EXAMPLE).

Regression cases

A case is a live run kept with its likely secrets masked: the input, what each step received (each response's body, its status, and the headers its step's criteria read), and the output. Replaying a case runs the operation as it is saved now through the engine with nothing sent: each step's success criteria are tested against what it kept, as a live run tests them, and the output is held against the kept one, field by field. A case kept before cases kept statuses and headers, or a step that reads XML (of which a case keeps only what its selectors selected), has its criteria not tested, and the replay says so. A saved example replays the same way.

  • Keep one: "Keep as case" in the Test run part asks for its name; or reqlyn run "<operation>" --input '…' --keep-case <name>.
  • Generate them: "Generate cases" makes three inputs from the operation's input types and names (an id, a hostname, an email, each looking like one), runs each live where the selected environment points, a mock from a description typically, and keeps each run as a case named for its input; or reqlyn test "<operation>" --generate <n>. An operation that may write is refused unless --writes is given; --plausible asks the configured model first for plausible values of string inputs Reqlyn has no words for, sending their names and types only.
  • Evaluate an LLM step: a step that calls an LLM API can hold the text at a place in its response (such as content.0.text) to an expected answer, by the share of words they have in common (computed on this computer, at least so many percent), or to a rubric the configured model judges (sent the rubric and that text, nothing else). A test, and every replay of a case, evaluates them after the run; one that does not hold fails it. With Claude assistance off, a rubric is not judged and says so, and does not fail the case. Add them under a step's "Evaluated after the run", on the desktop or in the browser; a test's output and each case say how each came out, and reqlyn test lists them under each case. They are kept with the operation, in its Arazzo source too (x-reqlyn-policy). On a server, a run evaluates them too, a rubric judged by the workspace's assistance: one that does not hold fails the run (EVALUATION_FAILED, saying which), its output kept; and a version's cases are evaluated before it is activated, so one that does not hold stops the activation.
  • Replay them: "Run cases" lists each case with "holds" or "does not hold" and where its output differs; or reqlyn test "<operation>", which ends with 1 when a case does not hold, and writes --junit too.
  • Change them: "Edit" beside a case renames it and sets the places of the output it leaves out (a date that changes each run, say), as a list separated by commas; "Remove" stops keeping it. From the command line: --rename <case> <new name>, --ignore <case> <place,place>, --remove <case>.
  • Carry them: reqlyn arazzo export writes them beside the document as <name>.cases.yaml (reqlyn: cases/1), reqlyn arazzo save keeps them, and reqlyn test "<operation>" --pack <file> replays a pack file instead, as a CI job does from a repository, keeping nothing.

Published with a version, a server replays its cases before the version can be activated anywhere, and refuses the activation when one does not hold (CASES_DO_NOT_HOLD).

Data-driven runs

reqlyn run "<operation>" --data devices.csv --parallel 10 --junit rows.xml runs the operation once for each row of the file, ten at a time: a CSV file whose first line names the columns, or a JSON array of objects. Each column is one of the operation's inputs, and a CSV cell is read as its input's kind (a whole number, a number, true or false, text); an empty cell is left out. It prints each row's outcome and a summary, ends with 1 when a row failed, and the JUnit report has a test case for each row (row 1, row 2, …). A file is read whole before anything runs: a bad row, or a column that is not an input, runs nothing.

On the desktop, "Run with data…" in an operation's Test run part asks for the file and runs the saved operation for each row, four at a time, with a line for each row and the summary; why a row failed is shown whole.

On a server, reqlyn remote run <id> <version> --environment <name> --data devices.csv --parallel 5 --junit rows.xml asks for a run of the published version for each row, five asked and waited for at a time, the cells read as the version's inputs say; it prints and reports the same way, each row being its own run on the server (in its run list, with its own evidence).

Load

reqlyn load "<operation>" --callers <n> --seconds <n> [--ramp <seconds>] [--per-minute <n>] runs a stored operation again and again on this computer: as many callers as asked, starting one after another over the ramp, for that long, at most so many runs a minute among them. Each run is live, through the selected environment, but its requests are not kept in the history. It prints its runs, how many succeeded and failed, runs a minute, each run's latency at the 50th, 90th, 95th and 99th percentiles and the most, failures by code, and the same for each step, and ends with 1 when a run failed. An operation with a step that may write is loaded only with --writes. At most 50 callers for 10 minutes. This computer has no shared rate limits; on a server, reqlyn remote load keeps to the workspace's (Using a server).

Thresholds

reqlyn load "<operation>" --callers 10 --seconds 60 --p95-under 500 --errors-under 1 --junit load.xml judges the run when it ends: over a threshold (the p95 of the runs' times at or over 500 ms, or 1% or more of the runs failing, or no run ending at all) it says which, "Over the threshold p95 under 500 ms: p95 812 ms is not under 500 ms.", and ends with 1; within both it ends with 0, even when a few runs failed within the error rate. The JUnit report has a test case for each threshold, failing with LOAD_THRESHOLD and the breach. reqlyn remote load takes the same options for a server's load run, judged when it ends.

Negative cases

Negative cases are generated from an OpenAPI description: for each operation, requests that are valid but for one thing the description forbids, a required parameter or field left out, a string past its length, a number past its bounds, a value outside its enum, text where a number goes. Beside each bound is a boundary case exactly at it, which the description allows. Unlike a regression case, each is sent, through the selected environment. An answer of 2xx to a negative case is a finding: the API accepted what its description says it does not. So is a 400 or 422 to a boundary case: the API refused what its description allows, as one off by one does.

  • From a description: reqlyn negatives <openapi file> prints each case and what it got, and ends with 1 on any finding; --keep <file> writes the cases and answers.
  • For an operation: reqlyn negatives --operation "<operation>" sends the cases of what its steps call, from the descriptions its source keeps, and keeps what they got beside its regression cases, in place of what was sent before. reqlyn test "<operation>" lists them after its cases, as last sent, with each finding; they do not change how reqlyn test ends, as they were sent then and are not replayed.
  • What is sent: a case that writes only with --writes. A case that would carry a credential, the one its operation names (x-reqlyn-credential) or a secret variable written into the description, only with --with-credentials: the description chose the server, so first the case says where it would go.

Fuzzing

reqlyn fuzz inventory.yaml --writes generates inputs at random within each operation's schemas: every required parameter and field, optional ones sometimes, numbers within their bounds, strings within their lengths and of any characters (letters, digits, punctuation, quotes, accented and other scripts, an emoji), a value of its enum, lists within their lengths. Each is sent as a negative case is, to the description's server with the selected environment's variables; a case that writes needs --writes, one that carries a credential --with-credentials. The first input the API fails on, a 5xx or no answer at all, is made smaller a step at a time while it still fails: an optional field left out, a string or list shortened, its characters made plain, a number brought toward its least. Both are printed, the one found and the smallest, and the run ends with 1.

Fuzzing sends many requests no person wrote, so it goes only through an environment marked for it: reqlyn environment lab --create --select --allows-fuzzing yes. On the desktop, the environment's menu beside Send has "Allows fuzzing" for a created environment. No environment allows it at first, and the one that always exists never does. Each run prints its seed; --seed gives the same inputs again, --runs how many for each operation, --operation only one.

Previewing what a run would send

"Preview what it sends" fills each step's request from the test input and sends nothing. It lists the steps in order under "WHAT A RUN WOULD SEND · NOTHING WAS SENT": each method and address, its body, what earlier steps would give it (named, as a run has not happened), and "writes" beside each step whose request does not only read. From the command line: reqlyn run "<operation>" --input '…' --preview.

Before and after

A step that writes may name the request that reads what it writes to (x-reqlyn-before in the source, with x-reqlyn-at for where the record is in its answer). The step then shows "Reads first, when asked: GET …". Beneath the preview, "Send this read to see what the write would change" sends only those reads, once each, and lays each value the write would send beside what is there now, as path: now → would send, or "unchanged". An XML body's elements are laid beside the read's element ending with the most of the same names (Route 53's ResourceRecordSet.TTL beside the record set read). A delete, which sends no body, lists everything the read found as path: now, removed. No write is sent. From the command line: --preview --read-first. On a server, reqlyn remote preview <operation> <version> --read-first <environment>, and "Read first, write nothing" in the Catalog.

When a run stops part way

Reqlyn never sends a write again by itself, and never undoes anything by itself. It offers you the choice, and says what each choice sends.

Resume from a step

A test that failed at a step offers "Resume from "<step>"": the steps before it are not sent again, and answer from what they gave. When the step's request writes, the API may have acted on it already, so Reqlyn asks: "Send it again" or Cancel. On a server, a run that failed can be resumed for a day after it failed: reqlyn remote resume <run> [--from <step>] [--yes], --yes only once you have seen what it did.

Undo

A step may name the request that undoes it (x-reqlyn-undo in the source: say, the cancel of a job a step started). The editor shows "Undone by "<request>", sent only when a person chooses to after a run stops part way." After a test stops part way, each step that returned and names an undo is offered, the last first, as "Undo "<step>"". Chosen, it says "This sends the request that undoes "<step>", once. It writes." with "Send the undo" and Cancel, and then the method, address and status the undo was answered. On a server: reqlyn remote undo <run> <step> --yes, while what the run received is kept (a day).

Steps that wait

  • For a value: a step whose policy holds until is asked again until its answer shows a value, say state is done, every few seconds, at most so many times: "Asked again until state is done: every 2 s, at most 30 times". Out of tries it fails with STEP_NOT_REACHED. A request that writes is never asked again (STEP_WAITS_ON_WRITE).
  • For a callback: a step whose policy holds callback gets a one-time address as {{callback_url}}, sends its request once, and waits for the upstream to post JSON to that address; the posted body is its result. An HMAC signature of the body can be required. With signed.kind: slack the step waits instead for a Slack button its message carries (value $callback_url in Arazzo): Slack posts the press to the server's /v1/slack/interactions, signed with the app's signing secret, the credential signed.credential names, and the step's result is action, label, user and username. Set the Slack app's Interactivity request URL to <REQLYN_CALLBACK_URL>/v1/slack/interactions. Only a server whose operator set REQLYN_CALLBACK_URL gives addresses; elsewhere the step fails before sending (STEP_NO_CALLBACKS).

Testing with no upstream

A mock answers what recordings answered and what a description says, with faults if you want them, and keeps what is made when asked: see Discovery. Point the operation's base address variable at the mock's address, and test as usual.

Help for Reqlyn 0.5.1. All help topics