Traces and metrics

The API records traces and metrics with OpenTelemetry and sends them over OTLP to whatever backend you use: Honeycomb, Grafana Cloud, Datadog, New Relic, an OpenTelemetry Collector of your own. It is off until you set OTEL_EXPORTER_OTLP_ENDPOINT; until then nothing is recorded or sent.

Turn it on

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=your-key

The endpoint is the base URL: the API adds /v1/traces and /v1/metrics. The headers are whatever your backend asks for, usually its key, as name=value pairs separated by commas. Traces leave in batches every few seconds, metrics once a minute, and what is left is sent when the API stops. On Railway, put both in .env.railway.

To look at them on your machine, run Grafana’s all-in-one image and open http://localhost:3001:

docker run --rm -p 3001:3000 -p 4318:4318 grafana/otel-lgtm
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 bun dev

The API is filed as the service api, and bun run worker as worker. OTEL_SERVICE_NAME changes that.

What is recorded

A trace for each request, named for its route: GET /api/workspaces/:workspaceId/usage. Inside it, a span for each query, with its SQL, whichever ORM sent it: Drizzle, Prisma, TypeORM, and Better Auth through any of them. A request that sends a traceparent header continues the caller’s trace.

Jobs continue the trace they were queued in. The span job email.send sits in the trace of the sign-up that queued it, however much later and in whichever process it runs. The trace context travels in the job’s outbox row, in trace_context, then with the job in BullMQ.

Metric	What
http.server.request.duration	Each request, in seconds, by method, route and status: request rates and latency
jobs.duration	Each job run, in seconds, by type and outcome: done, retry or failed
jobs.wait	How long each job waited for a worker after it was due
jobs.queue.depth	Jobs that are due in BullMQ and not yet taken by a worker

Log lines written during a trace carry its trace_id and span_id in production, so a backend that takes logs too can show them beside the trace. See Logs.

What is left out

• The values of queries: their SQL has placeholders, $1, never what was sent, which can be emails, tokens and hashes.
• Headers, query strings and bodies of requests.
• The paths of Better Auth’s endpoints, which can hold a token: they are all /api/auth/*.
• /api/health, and the relay looking for jobs to hand to BullMQ every second.
• Queries made outside a request or a job.

Record your own

Spans go through @opentelemetry/api, and cost almost nothing while telemetry is off:

import { trace } from '@opentelemetry/api';
import { TRACER } from '../telemetry/tracer.js';

await trace.getTracer(TRACER).startActiveSpan('report.build', async (span) => {
  try {
    span.setAttribute('report.rows', rows.length);
    return await this.build(rows);
  } finally {
    span.end();
  }
});

Metrics come from the meter that Nest injects, which records nothing while telemetry is off:

constructor(@Inject(TELEMETRY) { meter }: Telemetry) {
  this.exports = meter.createCounter('reports.exported');
}

Keep attribute values few: a route or a job type, never an id or an email. Each different value is a separate series, and backends charge by the series.

Nothing is instrumented by patching packages as they load, as OpenTelemetry does on Node, because Bun does not support it. A library you add, such as an HTTP client, is not traced until you wrap its calls in a span.

Test it

createTestApp({ teams, telemetry: { spans, metrics } }) records into an InMemorySpanExporter and a MetricReader of the test’s own. See apps/api/test/telemetry.e2e.test.ts.