Skip to main content
Every Routr component can send traces, metrics, and logs over OTLP/HTTP to an OpenTelemetry collector or any backend that accepts OTLP. Telemetry is off until you set an endpoint in the component’s configuration file.

What Routr sends

Each component reports a service.name of its own: routr-edgeport, dispatcher, connect, location, data, registry, requester or rtprelay. RoutrOne runs them in one process and reports as routr-one. The name isn’t configurable: Routr 3 doesn’t read OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES.

EdgePort spans

The EdgePort starts a SERVER span named SIP <METHOD> (for example SIP INVITE or SIP REGISTER) when a request arrives, and ends it when it sends the final response or the transaction fails. An ACK gets a span that ends at once. A span ends with an error status when the final response is 5xx or 408, when the next hop never answered, or when the final response couldn’t be sent. Search by sip.call_id to find every transaction of a call.

EdgePort metrics

Metrics are exported every 15 seconds. Prometheus-compatible backends rename them with underscores and a _total suffix, for example routr_sip_requests_total.

Trace context across SIP hops

A SIP request that arrives with a W3C traceparent header makes its EdgePort span a child in that trace, and the EdgePort writes its own span’s traceparent on the request it forwards. Both directions are limited to the EdgePort’s localnets:
  • A traceparent is trusted only when the request comes from an address in localnets. From anywhere else the span starts a new trace.
  • The EdgePort writes traceparent only on requests whose Request-URI host is in localnets, and removes the header from requests to any other target, such as carriers and phones.
Between components, the gRPC instrumentation carries the trace context, so a request that the Dispatcher passes to Connect, and Connect to Location, is one trace.

Send telemetry

Two keys, at the root of routr.yaml and of every standalone component file, turn telemetry on. See telemetry. OTEL_EXPORTER_OTLP_ENDPOINT no longer turns telemetry on by itself. To keep setting the endpoint from the environment, reference it with an empty default, so telemetry stays off when the variable is unset:
Routr 2’s OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_METRIC_EXPORT_INTERVAL, OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG, OTEL_SERVICE_NAME and OTEL_SDK_DISABLED have no Routr 3 key. sampleRatio replaces the two sampler variables.
Add the telemetry key to routr.yaml, next to spec. This is the file from Deploy with Docker, with telemetry on.
routr.yaml
Then add grafana/otel-lgtm to the same compose.yaml. It is a single container with an OpenTelemetry collector, Tempo, Prometheus, Loki and Grafana. Use it to try telemetry; in production, point the endpoint at your own collector.
Each component logs telemetry on: traces, metrics and logs go over OTLP at startup.

Find a call in Grafana

With otel-lgtm, open Grafana at http://localhost:3000 and sign in with admin / admin.
  1. Register a softphone and place a call.
  2. Open Explore and pick the Tempo data source. Search for service routr-edgeport (or routr-one) and span name SIP INVITE.
  3. Open a span and copy its sip.call_id. Search for that value to list the call’s other transactions, such as SIP ACK and SIP BYE.
  4. Switch to the Prometheus data source and query routr_sip_requests_total or routr_sip_responses_total by sip_method.
  5. Switch to Loki to read the components’ logs.

Troubleshooting

  • Nothing arrives. The endpoint must accept OTLP over HTTP, usually port 4318. Routr doesn’t export over gRPC, so don’t point it at 4317.
  • The component stops at startup with telemetry.otlpEndpoint: should be an http or https URL. Write the scheme, as in http://collector:4318.
  • Telemetry stays off though OTEL_EXPORTER_OTLP_ENDPOINT is set. The file must reference it: otlpEndpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:-}.
  • EdgePort spans don’t join the caller’s trace. The request came from an address outside localnets, or its traceparent isn’t a valid version 00 value.

routr.yaml

The telemetry key and every other setting.

Helm values

global.otlpEndpoint and the other chart values.

Record and replay calls

Record every message of a call and replay it in the terminal.

Run from source

The development stack, with otel-lgtm included.