> ## Documentation Index
> Fetch the complete documentation index at: https://docs.routr.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor Routr with OpenTelemetry

> For operators who want to see what Routr is doing: send traces, metrics, and logs over OTLP from routr-one, a split Docker deployment, or the Helm chart, and find SIP transactions in Grafana.

Every Routr component can send traces, metrics, and logs over OTLP/HTTP to an OpenTelemetry collector or any backend that accepts OTLP. Telemetry is off until you set an endpoint in the component's configuration file.

## What Routr sends

| Component | Traces | Metrics | Logs |
| - | - | - | - |
| EdgePort | One span per SIP transaction | `routr.sip.*` counters | Every log line |
| Dispatcher, Connect, Location, Data API, Registry, Requester, RTPRelay | gRPC client and server spans | None of their own | Every log line |

Each component reports a `service.name` of its own: `routr-edgeport`, `dispatcher`, `connect`, `location`, `data`, `registry`, `requester` or `rtprelay`. RoutrOne runs them in one process and reports as `routr-one`. The name isn't configurable: Routr 3 doesn't read `OTEL_SERVICE_NAME` or `OTEL_RESOURCE_ATTRIBUTES`.

### EdgePort spans

The EdgePort starts a `SERVER` span named `SIP <METHOD>` (for example `SIP INVITE` or `SIP REGISTER`) when a request arrives, and ends it when it sends the final response or the transaction fails. An ACK gets a span that ends at once.

| Attribute | Value |
| - | - |
| `sip.method` | Request method |
| `sip.call_id` | Call-ID of the request |
| `network.transport` | Transport of the request, for example `udp` or `wss` |
| `network.peer.address`, `network.peer.port` | Where the request came from |
| `sip.status_code` | Final response status, when one was sent |

A span ends with an error status when the final response is `5xx` or `408`, when the next hop never answered, or when the final response couldn't be sent. Search by `sip.call_id` to find every transaction of a call.

### EdgePort metrics

| Counter | Counts | Attributes |
| - | - | - |
| `routr.sip.requests` | SIP requests the EdgePort received | `sip.method`, `network.transport` |
| `routr.sip.responses` | Final responses the EdgePort sent | `sip.method`, `sip.status_code` |
| `routr.sip.timeouts` | Transactions that timed out | `sip.method` |
| `routr.sip.transport_errors` | Transactions a transport error ended | `network.transport` |
| `routr.sip.overflows` | Times one of the EdgePort's limits said no | `limit`: `udp_queue`, `transactions`, `call_queue`, `dialogs`, `async_sends` or `connections` |
| `routr.sip.malformed` | UDP datagrams the EdgePort dropped because they aren't SIP | `network.transport` |

Metrics are exported every 15 seconds. Prometheus-compatible backends rename them with underscores and a `_total` suffix, for example `routr_sip_requests_total`.

### Trace context across SIP hops

A SIP request that arrives with a W3C `traceparent` header makes its EdgePort span a child in that trace, and the EdgePort writes its own span's `traceparent` on the request it forwards. Both directions are limited to the EdgePort's `localnets`:

* A `traceparent` is trusted only when the request comes from an address in `localnets`. From anywhere else the span starts a new trace.
* The EdgePort writes `traceparent` only on requests whose Request-URI host is in `localnets`, and removes the header from requests to any other target, such as carriers and phones.

Between components, the gRPC instrumentation carries the trace context, so a request that the Dispatcher passes to Connect, and Connect to Location, is one trace.

## Send telemetry

Two keys, at the root of `routr.yaml` and of every [standalone component file](https://docs.routr.io/reference/configuration-files.md), turn telemetry on. See [`telemetry`](https://docs.routr.io/reference/configuration#telemetry).

| Key | Default | Description |
| - | - | - |
| `telemetry.otlpEndpoint` | Off | OTLP/HTTP base URL, for example `http://collector:4318`. Each signal goes to `/v1/traces`, `/v1/metrics` or `/v1/logs` under it. Absent or empty is off |
| `telemetry.sampleRatio` | `1` | The fraction of new traces kept, from `0` to `1`. A trace that arrives already started keeps its parent's decision, so a trace is kept or dropped whole. `0.01` keeps 1% of new traces |

`OTEL_EXPORTER_OTLP_ENDPOINT` no longer turns telemetry on by itself. To keep setting the endpoint from the environment, reference it with an empty default, so telemetry stays off when the variable is unset:

```yaml theme={"system"}
telemetry:
  otlpEndpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:-}
```

Routr 2's `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_METRIC_EXPORT_INTERVAL`, `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG`, `OTEL_SERVICE_NAME` and `OTEL_SDK_DISABLED` have no Routr 3 key. `sampleRatio` replaces the two sampler variables.

> [tab] Docker (routr-one)
>
> Add the `telemetry` key to `routr.yaml`, next to `spec`. This is the file from [Deploy with Docker](https://docs.routr.io/guides/deploy-with-docker.md), with telemetry on.
>
> ```yaml routr.yaml theme={"system"}
> apiVersion: v3
> kind: Routr
> ref: routr-one
> telemetry:
>   otlpEndpoint: http://otel-lgtm:4318
> spec:
>   edgeport:
>     externalAddrs:
>       - "${DOCKER_HOST_ADDRESS}"
>     transport:
>       - protocol: udp
>         port: 5060
>   data:
>     backend: postgres
>     url: secret://env/DATABASE_URL
>     apiKeys:
>       - secret://env/ROUTR_API_KEY
> ```
>
> Then add [`grafana/otel-lgtm`](https://github.com/grafana/docker-otel-lgtm) to the same `compose.yaml`. It is a single container with an OpenTelemetry collector, Tempo, Prometheus, Loki and Grafana. Use it to try telemetry; in production, point the endpoint at your own collector.
>
> ```yaml theme={"system"}
> services:
>   otel-lgtm:
>     image: grafana/otel-lgtm:0.38.0
>     ports:
>       - 3000:3000
> ```

> [tab] Docker (split)
>
> Set `telemetry.otlpEndpoint` in the file of every component you want to observe. The repository's files read it from the environment, and `compose.dev.yaml` sets the variable on every service:
>
> ```yaml dispatcher.yaml theme={"system"}
> kind: MessageDispatcher
> apiVersion: v2beta1
> ref: message-dispatcher
> telemetry:
>   otlpEndpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:-}
> spec:
>   processors:
>     - ref: connect-processor
>       addr: connect:51904
>       matchFunc: req => true
>       methods: [REGISTER, INVITE, ACK, BYE, CANCEL]
> ```

> [tab] Kubernetes (Helm)
>
> Set `global.otlpEndpoint`. The chart writes it as `telemetry.otlpEndpoint` in every component's file.
>
> ```yaml theme={"system"}
> global:
>   otlpEndpoint: http://otel-collector.observability.svc.cluster.local:4318
> ```
>
> ```bash theme={"system"}
> helm upgrade sipnet routr/routr-connect --namespace sipnet -f values.yaml
> ```

> [!TIP]
>
> Each component logs `telemetry on: traces, metrics and logs go over OTLP` at startup.

## Find a call in Grafana

With `otel-lgtm`, open Grafana at `http://localhost:3000` and sign in with `admin` / `admin`.

1. Register a softphone and place a call.
2. Open **Explore** and pick the **Tempo** data source. Search for service `routr-edgeport` (or `routr-one`) and span name `SIP INVITE`.
3. Open a span and copy its `sip.call_id`. Search for that value to list the call's other transactions, such as `SIP ACK` and `SIP BYE`.
4. Switch to the **Prometheus** data source and query `routr_sip_requests_total` or `routr_sip_responses_total` by `sip_method`.
5. Switch to **Loki** to read the components' logs.

## Troubleshooting

* **Nothing arrives.** The endpoint must accept OTLP over HTTP, usually port `4318`. Routr doesn't export over gRPC, so don't point it at `4317`.
* **The component stops at startup with `telemetry.otlpEndpoint: should be an http or https URL`.** Write the scheme, as in `http://collector:4318`.
* **Telemetry stays off though `OTEL_EXPORTER_OTLP_ENDPOINT` is set.** The file must reference it: `otlpEndpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:-}`.
* **EdgePort spans don't join the caller's trace.** The request came from an address outside `localnets`, or its `traceparent` isn't a valid version `00` value.

## Related

> [card] [routr.yaml](https://docs.routr.io/reference/configuration.md#telemetry)
>
> The `telemetry` key and every other setting.

> [card] [Helm values](https://docs.routr.io/reference/helm-values.md)
>
> `global.otlpEndpoint` and the other chart values.

> [card] [Record and replay calls](https://docs.routr.io/guides/record-and-replay-calls.md)
>
> Record every message of a call and replay it in the terminal.

> [card] [Run from source](https://docs.routr.io/guides/run-from-source.md)
>
> The development stack, with `otel-lgtm` included.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.