Observability and logging

What exists

Concern	Implementation	Package
Request id	`requestIdMiddleware`	`@perform/observability`
Structured logs	pino, `createLogger`	`@perform/logger`
HTTP access log	`pino-http`, `httpLogger`	`@perform/logger`
Logs from ported packages	consola, `logger`, `logError`	`@repo/logs`
Liveness, readiness	`healthRouter`, `readyRouter`	`@perform/observability`
Metrics	`prom-client` default metrics, `metricsRouter`	`@perform/observability`
Tracing	`initTelemetry`, a no-op placeholder	`@perform/observability`

There is no tracing and there are no custom metrics today. initTelemetry(serviceName) is called first thing in server.ts and returns immediately. It marks the place where an SDK would be initialized.

Request ids

requestIdMiddleware runs right after helmet and cors.

If the request has a non empty x-request-id header, that value is used. A caller can pass its own id to correlate across services.
Otherwise a ULID is generated.
The id is stored on req.requestId, returned in the X-Request-Id response header, included in the response envelope, and attached to the access log line.

When someone reports an error, ask for the requestId from the response body. It finds the access log line and the [error] or [client-error] line for that request.

The pino logger

import { createLogger } from '@perform/logger';

const log = createLogger('core-api');
log.info({ studioId, count }, 'task generator run');
log.error({ err, importId }, 'autofit import: job failed');

createLogger(serviceName) returns a pino logger with:

base: { service: serviceName } on every line.
ISO timestamps.
Redaction with remove: true, so redacted paths are dropped from the output, not masked.

The convention is fields first, message second. Put an error under the key err so pino serializes its type, message and stack. core-api creates one logger in server.ts and puts it on AppContext as ctx.logger. Services receive it as a dependency, usually optional (logger?: Logger), so they run in tests without one.

Redacted paths

req.headers.authorization
req.headers.cookie
req.headers["x-api-key"]
req.headers["x-smartsend-signature"]
req.body.password
req.body.currentPassword
req.body.newPassword
req.body.otp
req.body.token
req.body.apiKey
res.body.token
res.body.accessToken
res.body.refreshToken

Redaction is by exact path. It protects objects logged in the req and res shape. It does not find a token inside a field with another name. Do not log request bodies, API keys, OTP codes or session tokens under any key.

Two code paths log an OTP code on purpose, and only in dev mode: trainee OTP dev mode: skipping WhatsApp send and login OTP dev mode: skipping WhatsApp send. Both include the code. Never enable TRAINEE_OTP_DEV_MODE in production.

HTTP access log

httpLogger is a pino-http instance with its own logger named http.

Setting	Behaviour
Level	`error` for 5xx or a thrown error, `warn` for 4xx, `debug` for everything else
Skipped paths	`/healthz`, `/readyz`, `/metrics`
Request fields	`id`, `method`, `url`, `remoteAddress`, `remotePort`, and three headers: `host`, `content-type`, `x-request-id`
Response fields	`statusCode`
Extra	`requestId`

Only three headers are serialized, so signatures, nonces, API keys and cookies do not reach the access log regardless of the redaction list. Successful requests log at debug. With the default pino level of info, the access log shows only 4xx and 5xx. createLogger does not set a level and nothing in the repo reads a log level variable, so seeing successful requests means changing the logger options in code. url includes the query string. The webhook routes carry their token as ?token=..., so a webhook request that ends in 4xx or 5xx writes that token to the log. Treat access logs as sensitive.

Error logs

The error middleware in @perform/errors writes with console, not pino:

[error]        { requestId, path, method, code, name, message }          5xx
[client-error] { requestId, path, method, status, code, message }        4xx

These are plain console.error and console.warn calls, so they are not JSON lines and carry no service field. They do not include a stack. For a stack, log at the point of failure with ctx.logger.error({ err }, '...'). The automation lane logs unexpected errors as console.error('[automation-api]', err).

`@repo/logs`

The ported packages (auth, api, payments, mail, storage, notifications, database) use a second logger built on consola:

import { logger, logError, serializeError } from '@repo/logs';

logger.warn('Skipping coach provisioning', { studioId });
logError('Failed to send email', error, { to });

Export	Purpose
`logger`	A consola instance. Message first, then data.
`serializeError(error)`	`{ name, message, code?, stack? }`. The stack is cut to four frames. A string becomes `{ name: 'Error', message }`. Anything else becomes `{ name: 'UnknownError', message: JSON }`.
`logError(message, error, context?)`	`logger.error(message, { error: serializeError(error), ...context })`.

Note the argument order is the opposite of pino. consola takes the message first. @repo/database’s default client forwards Prisma events to this logger. The client core-api builds through createPrismaClient({ logger: log }) forwards them to pino instead.

Which one to use

Code	Logger
`apps/core-api/src/**`	`ctx.logger` (pino)
`packages/*` in the `@perform` scope	A `Logger` passed in, or none
`packages/*` in the `@repo` scope	`logger` and `logError` from `@repo/logs`

The output of one process therefore mixes three formats: pino JSON, consola lines and raw console lines from the error middleware. Log shipping has to tolerate non JSON lines.

Prisma logging

createPrismaClient subscribes to Prisma’s warn and error events and filters them before logging:

Transient connection errors (P1001, P1002, P1008, P1017, or a message that looks like a dropped connection) are logged at debug. The client retries them up to four times with delays of 200, 500, 1000 and 2000 ms.
Unique constraint failed is logged at debug. Code that relies on a unique constraint to dedup, such as flow run starts, would otherwise fill the error log.
A pool error on an idle client is logged at error as postgres pool error (idle client).
Query logging is off unless logQueries: true is passed, in which case queries are logged at debug with their duration.

Health and readiness

Endpoint	Behaviour
`GET /healthz`	Always `200` with `{ "ok": true }`. The process is up.
`GET /readyz`	Runs the checks in parallel. `200` when no required check failed, `503` otherwise.
`GET /api/health`	Plain text `OK` from the Hono app. Proves the gateway and the Hono app are mounted.

{ "ok": true, "degraded": ["redis"] }

{ "ok": false, "failed": ["database"], "degraded": ["redis"] }

A check is a ReadinessCheck: { name, check, optional? }. A check that throws counts as failed. database is required and redis is optional. See Bootstrap and AppContext. A degraded Redis means queues, OTP, replay protection and caches are affected while the API still answers. Signed lanes will fail, because the nonce store needs Redis.

Metrics

GET /metrics serves the Prometheus text format from a dedicated Registry with collectDefaultMetrics: process CPU, memory, event loop lag, garbage collection and handles. The registry is exported as metricsRegistry, so a module can register its own counter or histogram on it. The endpoint has no authentication and is not rate limited. Restrict it at the proxy if the API is public.

Worker and job logs

Workers log a summary per run. Message strings are stable, so they can be used as search terms.

Message	Meaning
`task generator run`	`created` and `closed` counts. `info` when something changed, else `debug`.
`update form dispatcher run`, `form reminder dispatcher run`, `pending form notifier run`	Counts per pass.
`subscription reconcile run`	`evaluated` and `reconciled`.
`plan release sweep run`	`released` and `failed`.
`flow sweep run`	`resumed` and `stalled`.
`demo activity run`	Per studio results.
`... skipped: infrastructure unavailable`	A transient database or Redis error. `warn`.
`flow run: job retrying`, `flow run: job gave up`	Attempt progress for a flow run.
`automation hook delivered`, `automation hook dropped`, `automation hook gave up`	Webhook delivery outcome.
`autofit import: job failed`	With `importId`.
`trainee push: sent`, `trainee push: no registered devices, nothing sent`, `trainee push: deliveries failed after Expo accepted them`	See Notifications and push.

A run that did nothing logs at debug, so a quiet system produces a quiet log at the default level.

Process level

unhandledRejection is logged at error with the reason and the process keeps running.
uncaughtException is logged and the process exits with code 1. The supervisor restarts it.
core-api listening is logged with the port at startup.
shutdown initiated is logged with the signal, and http server closed when the listener closes.

​What exists

​Request ids

​The pino logger

​Redacted paths

​HTTP access log

​Error logs

​@repo/logs

​Which one to use

​Prisma logging

​Health and readiness

​Metrics

​Worker and job logs

​Process level

What exists

Request ids

The pino logger

Redacted paths

HTTP access log

Error logs

`@repo/logs`

Which one to use

Prisma logging

Health and readiness

Metrics

Worker and job logs

Process level