Skip to content

Telemetry ​

Overview ​

The Telemetry Module (src/modules/telemetry/) pushes the API's logs to an OpenTelemetry Collector over OTLP. The Collector filters the logs, hashes personal data and samples successful requests, then sends them to OpenObserve, where people search logs, read the dashboard and receive alerts in Mattermost.

It is off by default. When OTEL_EXPORTER_OTLP_ENDPOINT is not set (or is empty), logging works exactly as before: stdout, plus LOG_FILE when set. Stdout output is always kept; OTLP is an extra output, not a replacement.

ODP API ──OTLP──▶ OTel Collector ──OTLP──▶ OpenObserve ──webhook──▶ Mattermost
 2 pino loggers     filter · hash PII        logs · metrics           alerts
 /server/metrics ◀─ scrape (15s)            dashboard · alerts

The module has two parts:

PartLocationContent
Codeservice/api/src/modules/telemetry/OTLP transport target, redaction for the app logger, serializer for the request log
Stackdocker-images/telemetry/docker-compose.yml (Collector + OpenObserve), otel-collector.yaml, openobserve/ provisioning (alerts, dashboard, retention)

Business audit data (activity, revisions, security records in the database) is not moved to OTel. It stays the source of truth for "who changed which record".

Enabling Telemetry ​

  1. Start the stack: cd docker-images/telemetry && cp .env.example .env, fill the values, then docker compose up -d. The one-shot openobserve-init service provisions streams, alerts and the dashboard.
  2. Set the API env and restart the API:
env
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
LOG_REQUEST_BODY=errors
METRICS_ENABLED=true
METRICS_TOKEN=<same value as the stack's METRICS_TOKEN>
VariableDefaultDescription
OTEL_EXPORTER_OTLP_ENDPOINT—Collector base URL. Set → both loggers also push over OTLP (/v1/logs is appended). Unset or empty → off.
OTEL_SERVICE_NAMEodp-apiservice.name resource attribute.
LOG_REQUEST_BODYoffAdd the sanitized request body to the request completed line: off, errors (status ≥ 400) or all.
LOG_REQUEST_BODY_MAX4096Maximum length of the logged body string.

The OTel exporter also reads the standard variables directly (OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_RESOURCE_ATTRIBUTES, OTEL_BLRP_*). In tests (NODE_ENV=test or vitest) OTLP export is always off.

Recommended production values: LOG_LEVEL=info, LOG_REQUEST_BODY=errors, DB_LOG_QUERIES=false.

What Is Sent ​

Every log record carries the resource attributes service.name, service.version, deployment.environment and host.name, the pino level, the message, the logger name and the request id reqId.

KindSourceContent
HTTP requestFastify loggerincoming request (dropped by the Collector) and request completed (see below)
Security eventsecurity/events/logger.ts (logger security)event_type, severity (info / warning / critical), user_id, ip, target_user, collection, details
Module loguseLogger(name)About 90 loggers (mcp, ai, workflow, notify, database, …); errors carry err_* fields with the stack
Raw SQLDB_LOG_QUERIES=true onlysql; the Collector deletes bindings
MetricsGET /server/metricsScraped by the Collector; see Metrics

Position in the request pipeline ​

StepLog lineNote
onRequestincoming requestBefore authentication: the user is unknown. Kept on stdout, dropped in the Collector.
preHandler (authenticate)Security eventsLogin, token, MFA, rate limit, …
HandlerModule logs, 5xx errors
onResponserequest completedAfter authentication: carries the user and the parsed body.

All lines of one request share the same reqId (taken from the x-request-id header when present).

The request completed line ​

The Fastify res serializer (requestLogSerializers in request-log.ts) writes:

res: { statusCode, method, url, route, ip, user_id, request_body? }
responseTime, reqId
  • url has credential query parameters masked (redactUrl).
  • route is the matched route pattern (for example /items/:collection); it is empty for requests that match no route.
  • user_id is null for public requests.
  • request_body is one JSON string, present only when LOG_REQUEST_BODY=all, or errors and status ≥ 400, and only for JSON bodies (form strings, Buffers, streams and multipart are skipped).

This shape is a contract with otel-collector.yaml and with the OpenObserve alerts and dashboard. Change the serializer → update both.

Sensitive Data ​

DataHandled inResult in OpenObserve
Password, token, secret, OTP, API key, private key, credential, email keys in the request body (any depth)App[REDACTED]
authorization, cookie, password, token, secret, access_token, refresh_token, otp, api_key in app logs (top level and one level deep)App (pino redact)[REDACTED]
Authorization / cookie headers, credential query parametersApp[REDACTED]
Extension log data and audit details: credential keys at any depth, Error values, depth/array/string/size capsApp (sanitizeLogData)[REDACTED], { type, message, stack }, [depth]… — see Extension Guardrails §7
Extension log data: user_id, email, ip, phone, target_user deeper than the top level or detailsApp[REDACTED] (the Collector cannot hash them there)
user_id, email, ip, phone, target_user at the top level and in details; res.ip, res.user_idCollectorsha256(PII_HASH_SALT + value)
Raw SQL bindingsCollectorDeleted

Body sanitizing limits: depth 6, arrays 20 items, strings 256 characters, whole string LOG_REQUEST_BODY_MAX characters with a …[truncated] suffix.

Stdout keeps the original ip and user_id. To find one user's logs in OpenObserve, compute the hash with the same salt:

sh
printf '%s%s' "$PII_HASH_SALT" "<user-uuid>" | sha256sum

Collector Pipeline ​

  1. Filter — drop incoming request.
  2. Transform — delete bindings, hash the PII fields, copy the pino level to the level attribute.
  3. Route — copy security events (core and extension audit), error / fatal logs and the extension rate-cap lines (extension log rate cap hit, extension audit rate cap hit) to the audit stream.
  4. Sample — keep 10% of request completed lines with status 200–399, and 10% of extension trace / debug lines. Both are keyed by the OTLP trace id, so a kept request keeps all its lines. Lines without a trace (boot) pass. 4xx, 5xx and every other log are kept at 100%.

The Collector has no rate limiter for logs (contrib 0.163.1 ships none). The hard cap per extension is in the app (EXTENSION_LOG_RATE / EXTENSION_LOG_BURST).

Storage and Retention ​

StreamContentRetention
odp_apiAll logs14 days (O2_RETENTION_DAYS)
odp_api_auditSecurity events, extension audit events (name = 'ext-security'), error / fatal logs and rate-cap lines (also in odp_api)90 days (O2_AUDIT_RETENTION_DAYS)
odp_alert_notificationsAlert messages when no Mattermost webhook is set14 days
odp_*, up (metrics)API metrics14 days

OpenObserve flattens nested fields with _ and lowercases them: res.statusCode → res_statuscode, reqId → reqid, details.email → details_email, extension.id → extension_id. Filter the pino level with level; on security events the severity column holds the event's own severity. The OTLP trace id is the trace_id column.

Searching ​

Open OpenObserve → Logs → choose a stream → SQL mode:

sql
-- Failed requests by route
SELECT res_route, res_statuscode, count(*) AS n FROM "odp_api"
WHERE body = 'request completed' AND res_statuscode >= 400
GROUP BY res_route, res_statuscode ORDER BY n DESC

-- Every line of one request
SELECT _timestamp, name, level, body FROM "odp_api" WHERE reqid = '<reqId>' ORDER BY _timestamp

-- Failed logins by hashed IP
SELECT ip, count(*) AS failed FROM "odp_api_audit"
WHERE event_type = 'LOGIN_FAILED' GROUP BY ip ORDER BY failed DESC

-- One extension's lines, by kind
SELECT _timestamp, extension_kind, level, body FROM "odp_api"
WHERE extension_id = '<extension-id>' ORDER BY _timestamp DESC

-- Every line of one trace (inbound traceparent or a schedule run)
SELECT _timestamp, name, extension_id, body FROM "odp_api" WHERE trace_id = '<32-hex>' ORDER BY _timestamp

-- One user's requests (hash computed as above)
SELECT _timestamp, res_method, res_route, res_statuscode FROM "odp_api"
WHERE res_user_id = '<hash>' ORDER BY _timestamp DESC

The ODP API Overview dashboard shows API up, requests/s, 5xx ratio, p95 latency, memory, failed logins, requests by status, p95 by route, security events by type, warn/error logs by logger, memory and event-loop lag, and tables of failed requests, security events and errors, plus extension log lines by extension_id and extension lines dropped by the rate cap.

Alerts ​

Alerts are evaluated every minute and sent to Mattermost through an incoming webhook (MATTERMOST_WEBHOOK_URL). Without a webhook, messages are written to the odp_alert_notifications stream instead.

AlertFires whenAction
odp_api_5xx_ratio_high5xx > 5% of requests over 5 min, at least 20 requestsRead the Errors table, filter level = 'error', check err_message
odp_api_login_failed_burst> 20 LOGIN_FAILED from one hashed IP in 5 minFind the source in odp_api_audit; check for RATE_LIMIT_TRIGGERED; block if needed
odp_api_security_criticalAny security event with severity = 'critical' in 1 minAct at once (audit chain broken, malware, drift)
odp_api_memory_highRSS > 1.5 GiB for 10 minCheck the memory panel for a steady climb (leak)
odp_api_target_down/server/metrics cannot be scraped for 2 minCheck the API container and network
odp_api_ext_security_criticalAny extension audit event with severity = 'critical' in 1 min (message [EXT <id>] ext.<id>.<type>)Ask the extension's owner; the event is the extension's own claim, not a core event
odp_api_ai_guardrail_block_spike≥ 20 AI guardrail refuse / replace for one app_id in 5 min (rule image not counted; shown as images). Message [AI] app <app_id>: <n> refuse/replace in 5 min (<m> unchecked images)Read the app's guardrail lines in odp_api_audit (name = 'ai-guardrail') by guardrail_rule: an attack, or a rule that is too strict
odp_api_ai_guardrail_unavailable≥ 5 AI guardrail unavailable for one integration_id in 5 min. Message [AI] decision integration <id> unavailable: <n> in 5 minCheck the decision model integration; guardrail_fail_mode says whether turns passed (open) or were refused (closed)

The core security alerts (odp_api_login_failed_burst, odp_api_security_critical) and the core security panels count only lines without extension_id, so no extension line can fire them.

Thresholds live in docker-images/telemetry/openobserve/alerts/*.json. Edit the file, then run docker compose run --rm openobserve-init. Changes made in the OpenObserve UI are overwritten by the next init run.

Run init.sh before relying on alerts ​

OpenObserve refuses a query that names a column the stream has never seen (400 unknown field), and it then never fires that alert. init.sh pre-declares every column the alerts and the dashboard filter on (extension_id, event_type, trace_id…).

  • Run openobserve-init on every new install and after every upgrade of this stack (new alerts and columns come with it). It is idempotent.
  • On a fresh install the metrics alerts (odp_api_5xx_ratio_high, odp_api_memory_high, odp_api_target_down) name metric streams that exist only after the Collector has scraped the API once. init.sh then skips them, still creates the log alerts and the dashboard, and exits with code 3 and WARN skipped alerts. Run it again once metrics arrive; the exit code is then 0.

Limitations ​

  • Only @odp/api is covered. The admin webapp (BFF) does not push logs yet, and it does not forward a request id or trace context to the API.
  • Light trace context only: every log line carries trace_id / span_id. The only spans exported are those of the AI SDK calls, through a private tracer (nothing registered globally) and an attribute allowlist that drops prompt and response content (src/modules/telemetry/llm-telemetry.ts); they join the request on trace_id. No other spans are exported (no per-request or per-operation timing).
  • OpenObserve OSS makes every user an Admin. Read-only access (Viewer role) and SSO need OpenObserve Enterprise.

AI Guardrail Audit ​

Each finding of the AI guardrails (warn, mask, refuse, replace, note, unavailable, unchecked_image) writes one log line on the logger ai-guardrail (aiGuardrailAudit() in src/modules/telemetry/ai-guardrail-audit.ts). It is never a row in odp_security_events and never on the logger security, so it cannot fire the core security alerts.

FieldValue
nameai-guardrail
categorysecurity (copied to odp_api_audit)
event_typeai.guardrail
severitywarning for refuse, replace, unavailable; info for the others
guardrail_directionin / out
guardrail_ruleI1…I9, O1…O9; decision (a decision call is unavailable); image (a user image the decision model cannot check); size (a text longer than the checked windows: refuse in / replace out when closed, warn when open)
guardrail_actionthe action above
guardrail_surfacechat / test_run / engine / agentic
guardrail_fail_modeopen / closed
guardrail_pdecision probability in [0, 1], 3 decimals; absent when there is none
app_id, integration_idids, [A-Za-z0-9_-]{1,64}; absent when null
user_iduser of the turn, hashed by the Collector; absent for a public turn
message[AI] ai.guardrail <action> <rule>
  • Every field has a closed format, so the line cannot carry user text. An event with a field out of format is dropped, with one warning per 10 s on the logger ai.
  • Rate cap 50/s, burst 500 per process. Dropped events are counted in one ai guardrail audit rate cap hit line per window (category = 'security').
  • info lines are written only when LOG_LEVEL is info or lower.
  • The columns are declared by init.sh (run it after upgrading, see above).

Extensions ​

Extension backend code (hook, endpoint, schedule, MCP tool) writes through context.logger and context.audit. Its lines go into the same pipeline:

ItemWhere it lands
context.logger linesodp_api, filtered by extension_id; error / fatal also copied to odp_api_audit
context.audit.event()odp_api_audit (and odp_api), name = 'ext-security', event_type = ext.<id>.<type>; critical fires odp_api_ext_security_critical. Never a row in odp_security_events
Rate-cap linesodp_api and odp_api_audit; counter odp_extension_log_dropped_total{extension_id} on /server/metrics

The logger contract, the reserved keys, the PII keys and the audit rules are in Extension Guardrails §7. The env keys are EXTENSION_LOG_LEVELS, EXTENSION_LOG_RATE, EXTENSION_LOG_BURST, EXTENSION_AUDIT_RATE, EXTENSION_AUDIT_CRITICAL_PER_MIN (Environment variables).

Trace Context ​

  • Inbound. A valid W3C traceparent gives the request its trace_id and flags; the request gets a new span_id, and the caller's span id is parent_span_id on the Fastify request lines. An absent or invalid header starts a new trace. x-request-id / reqId are unchanged and stay on every line.
  • Lines. App logger lines (mixin) and Fastify request lines (request logger bindings) carry trace_id, span_id, trace_flags. A schedule run gets its own reqId (run:<uuid>) and trace.
  • Outbound. Core never adds traceparent to calls to third parties (OAuth/OpenID providers, Infobip, Mattermost, LLM vendors, remote MCP servers): the trace id stays inside the trust boundary. safeFetch has an opt-in trace: true; no core caller sets it, and auth drivers must never set it. An extension uses context.trace.headers() only for services it owns.

Programmatic API ​

Exported from src/modules/telemetry/index.ts:

ExportPurpose
buildOtelTargets(level)Pino transport targets for OTLP export; [] when off or in tests. Used by src/logger.ts and src/server.ts.
appLogRedactPino redact options for the app logger.
requestLogSerializersFastify logger serializer for res on the request completed line.
sanitizeBody(body, max)The body sanitizer, exported for tests.
sanitizeLogData(data, options?), sanitizeDetailsData(details, options?)Sanitizer for extension log data and audit details.
prepareLogLine(args), prepareLogBindings(bindings), isReservedLogKey(key, value?), normalizeLogKey(key)pino call forms → sanitized data + message; reserved keys on OpenObserve's column form. Used by src/extensions/extension-logger.ts.
createTokenBucket(options), createRateCap(options)Per-extension caps with one drop report per window.
createTraceContext(inbound), outboundTraceHeaders(ctx), traceLogFields(ctx), parseTraceparent(header)W3C trace context helpers.

src/modules/telemetry/ext-audit.ts (createExtensionAudit) is imported directly by src/extensions, not through the index (it imports logger.ts).

Source Files ​

FileResponsibility
src/modules/telemetry/config.tsRead the telemetry env, build resource attributes
src/modules/telemetry/transport.tsbuildOtelTargets()
src/modules/telemetry/redact.tsRedact paths for the app logger
src/modules/telemetry/request-log.tsres serializer and body sanitizer
src/modules/telemetry/sanitize.tsOne walker for bodies, extension log data and audit details
src/modules/telemetry/log-format.tsCall forms, printf, reserved keys
src/modules/telemetry/rate-cap.tsToken bucket and rate cap
src/modules/telemetry/trace-context.tstraceparent parse/format, outbound headers, log fields
src/modules/telemetry/ext-audit.tscontext.audit.event()
docker-images/telemetry/otel-collector.yamlCollector pipeline
docker-images/telemetry/openobserve/init.sh, alerts, dashboard, Mattermost template

ODP Internal API Documentation