> ## Documentation Index
> Fetch the complete documentation index at: https://docs.routr.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Run a benchmark

> For maintainers who want to know how much SIP traffic a Routr release handles and whether it leaks memory: run the perfkit benchmarks on DigitalOcean from a laptop or from GitHub, read the results at routr.io/benchmarks, and delete a run.

Routr's benchmarks run a released version on dedicated DigitalOcean servers with [perfkit](https://github.com/fonoster/perfkit): one server for Routr, a DigitalOcean Kubernetes cluster for the Connect chart, and one load generator that drives both. Each run publishes a page at [routr.io/benchmarks](https://routr.io/benchmarks). Maintainers run them after significant code changes, not on every release.

The definitions live in [`perf/`](https://github.com/fonoster/routr/tree/main/perf) in the repository.

## The benchmarks

| Id | What it measures |
| - | - |
| `edgeport-echo` | The EdgePort's limit on one server, over UDP: SIPp sends SIP MESSAGE through the EdgePort to Echo (the Go SDK's example processor, `fonoster/routr-echo:<version>-go`), from 2,000 messages/s adding 500 every 30 seconds, up to 30,000, until p99 latency reaches 50 ms or failed requests reach 0.1%. A ceiling for the EdgePort alone. |
| `one-simple` | RoutrOne's call limit on one server, over UDP: authenticated calls, held 5 seconds, between 200 registered Agents, from 50 calls/s adding 50 every 30 seconds, up to 3,000. |
| `one-mix` | The same with a realistic mix between 1,800 registered Agents: 60% answered, 15% busy, 15% cancelled, 10% to offline Agents. |
| `k8s-simple` | The Connect chart's call limit on Kubernetes (three c-8 nodes, three EdgePorts behind a DigitalOcean Network Load Balancer), over TCP: the same calls as `one-simple`. TCP because that load balancer drops UDP datagrams over 1,424 bytes. |
| `one-soak` | Memory leaks on one server: the call mix at half of `one-mix`'s limit for 2 hours, answered calls held 2 minutes. Each process's memory may grow at most 5 MB/h, failed calls must stay under 0.1%, and nothing may restart. |
| `k8s-soak` | The same on Kubernetes, simple calls at half of `k8s-simple`'s limit, per component and pod. Runs at the same time as `one-soak`. |

Three config files define them. Only `perf/perfkit.yaml` publishes to routr.io:

* `perf/perfkit.yaml`: the full run, all six benchmarks. It takes about 3.5 hours: the capacity benchmarks one after another, then both soaks at once.
* `perf/perfkit.calibration.yaml`: the calibration config. The EdgePort ramp with a 1-hour soak at 5 calls/s, for tuning the benchmarks.
* `perf/perfkit.smoke.yaml`: the smoke config. The six benchmarks, shortened, on a small cluster (two c-4 nodes): short ramps and 3-minute soaks. It takes under an hour and can be watched live like a full run. Run it first to check the harness before a full run.

A smoke or calibration run from GitHub runs everything except the `publish` job. Its report stays in the run's `perfkit-report-<run id>` artifact.

## Before you start

* **Who can run it.** Running the workflow needs write access to `fonoster/routr`. Running from a laptop needs only the DigitalOcean token.
* **Cost.** Each run creates two dedicated-CPU droplets (`c-8` for Routr, `c-16` for the load generator) and a Kubernetes cluster of three `c-8` nodes with a load balancer, about \$6 for a full run, billed until deleted. perfkit deletes everything at the end of every run, even a failed or cancelled one. `perfkit sweep` is the fallback: it deletes droplets and clusters left behind. The scheduled sweep runs every six hours from the default branch; until that branch pins perfkit v0.2.0, it removes droplets only, so after a run that failed early, run `perfkit sweep` by hand.
* **One-time setup.** An admin adds two repository secrets, under **Settings**, then **Secrets and variables**, then **Actions**:
  * `DIGITALOCEAN_TOKEN`: a DigitalOcean API token with the custom scopes listed in [perfkit's README](https://github.com/fonoster/perfkit#readme): create, read and delete for droplet, VPC, firewall, SSH key and tag; create, read, delete and access\_cluster for kubernetes; read and delete for load\_balancer and block\_storage.
  * `PERFKIT_SSH_KEY`: the private half of an ed25519 key made for perfkit only:

    ```bash theme={"system"}
    ssh-keygen -t ed25519 -N "" -f perfkit
    ```

    Paste the contents of `perfkit` (not `perfkit.pub`) into the secret, then delete both files.

## Run from GitHub

**1. Open the workflow**

In `fonoster/routr`, open **Actions**, then **Benchmarks**, then **Run workflow**.

**2. Fill in the inputs**

| Input | What to enter |
| - | - |
| `target-version` | The Routr release to benchmark, without the `v`, for example `2.31.6`. Its tag `v2.31.6` must exist, or the run stops before any droplet is created. |
| `config` | `perf/perfkit.smoke.yaml` (the default) for the short check, `perf/perfkit.calibration.yaml` for calibration, or `perf/perfkit.yaml` for the full run. Only `perf/perfkit.yaml` publishes. |
| `benchmarks` | Ids to run, comma-separated, for example `edgeport-echo`. Leave it empty to run all of them. A benchmark left out shows as not run. |

**3. Wait for the page**

The run continues across several jobs because GitHub stops a job after six hours. When a `perf/perfkit.yaml` run ends, the `publish` job adds the report to the `gh-pages` branch. Other configs skip the `publish` job.
\[!TIP] [routr.io/benchmarks](https://routr.io/benchmarks) shows the new run. No commit lands on `main`, so no release starts.

## Run from a laptop

You need Node.js 22 and npm 11. `npx` installs perfkit from its git tag, and npm 10, the npm that Node 22 ships with, fails on it. Check with `npm --version`, and update with `npm install -g npm@11`.

From the repository root:

```bash theme={"system"}
export DIGITALOCEAN_TOKEN=<your token>
npx -y github:fonoster/perfkit#v0.2.0-rc.5 run --config perf/perfkit.yaml --target-version 2.31.7
```

With npm 10, run the same command through npm 11:

```bash theme={"system"}
npx -y npm@11 exec --yes --package=github:fonoster/perfkit#v0.2.0-rc.5 -- perfkit run --config perf/perfkit.yaml --target-version 2.31.7
```

Add `--benchmarks edgeport-echo` to run one benchmark, `--config perf/perfkit.smoke.yaml` for the short check, or `--dry-run` to print the plan without creating anything. The report is written to `.perfkit/<run id>/report.json`. A laptop run doesn't publish to routr.io.

## Where results appear

Every published run gets its own page:

* `https://routr.io/benchmarks/`: the latest run.
* `https://routr.io/benchmarks/runs/<run id>/`: one run, with its `report.json`.
* `https://routr.io/benchmarks/runs.json`: every run, newest first.

The pages live only on the `gh-pages` branch. Nothing under `site/` builds them, and the website deploy keeps them in place.

## Delete a run

To remove a run, such as a smoke run you've reviewed:

**1. Check out gh-pages**

```bash theme={"system"}
git fetch origin gh-pages
git switch gh-pages
```

**2. Remove the run**

Delete `benchmarks/runs/<run id>/`, and delete its entry in `benchmarks/runs.json`.

**3. Fix the latest page**

If it was the latest run, `benchmarks/index.html` still shows it. Re-render it from the newest remaining run's report, with the edited `runs.json` as history:

```bash theme={"system"}
npx -y github:fonoster/perfkit#v0.2.0-rc.5 render \
  --report benchmarks/runs/<newest remaining run id>/report.json \
  --history benchmarks/runs.json --out benchmarks
```

When no run remains, delete `benchmarks/` instead.

**4. Push**

```bash theme={"system"}
git add -A benchmarks
git commit -m "chore: delete benchmark run <run id>"
git push origin gh-pages
```

> [!TIP]
>
> `routr.io/benchmarks/runs.json` no longer lists the run, and `routr.io/benchmarks` shows the next run, or nothing when it was the only one.

## Related

> [card] [perfkit](https://github.com/fonoster/perfkit)
>
> The tool, its config format and its commands.

> [card] [Run Routr from source](https://docs.routr.io/guides/run-from-source.md)
>
> Build and test Routr on your machine.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.