Appearance
Telemetry
Overview
The Telemetry Module (src/modules/telemetry/) pushes the API's logs to an OpenTelemetry Collector over OTLP. The Collector filters the logs, hashes personal data and samples successful requests, then sends them to OpenObserve, where people search logs, read the dashboard and receive alerts in Mattermost.
It is off by default. When OTEL_EXPORTER_OTLP_ENDPOINT is not set (or is empty), logging works exactly as before: stdout, plus LOG_FILE when set. Stdout output is always kept; OTLP is an extra output, not a replacement.
ODP API ──OTLP──▶ OTel Collector ──OTLP──▶ OpenObserve ──webhook──▶ Mattermost
2 pino loggers filter · hash PII logs · metrics alerts
/server/metrics ◀─ scrape (15s) dashboard · alertsThe module has two parts:
| Part | Location | Content |
|---|---|---|
| Code | service/api/src/modules/telemetry/ | OTLP transport target, redaction for the app logger, serializer for the request log |
| Stack | docker-images/telemetry/ | docker-compose.yml (Collector + OpenObserve), otel-collector.yaml, openobserve/ provisioning (alerts, dashboard, retention) |
Business audit data (activity, revisions, security records in the database) is not moved to OTel. It stays the source of truth for "who changed which record".
Enabling Telemetry
- Start the stack:
cd docker-images/telemetry && cp .env.example .env, fill the values, thendocker compose up -d. The one-shotopenobserve-initservice provisions streams, alerts and the dashboard. - Set the API env and restart the API:
env
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
LOG_REQUEST_BODY=errors
METRICS_ENABLED=true
METRICS_TOKEN=<same value as the stack's METRICS_TOKEN>| Variable | Default | Description |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | — | Collector base URL. Set → both loggers also push over OTLP (/v1/logs is appended). Unset or empty → off. |
OTEL_SERVICE_NAME | odp-api | service.name resource attribute. |
LOG_REQUEST_BODY | off | Add the sanitized request body to the request completed line: off, errors (status ≥ 400) or all. |
LOG_REQUEST_BODY_MAX | 4096 | Maximum length of the logged body string. |
The OTel exporter also reads the standard variables directly (OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_RESOURCE_ATTRIBUTES, OTEL_BLRP_*). In tests (NODE_ENV=test or vitest) OTLP export is always off.
Recommended production values: LOG_LEVEL=info, LOG_REQUEST_BODY=errors, DB_LOG_QUERIES=false.
What Is Sent
Every log record carries the resource attributes service.name, service.version, deployment.environment and host.name, the pino level, the message, the logger name and the request id reqId.
| Kind | Source | Content |
|---|---|---|
| HTTP request | Fastify logger | incoming request (dropped by the Collector) and request completed (see below) |
| Security event | security/events/logger.ts (logger security) | event_type, severity (info / warning / critical), user_id, ip, target_user, collection, details |
| Module log | useLogger(name) | About 90 loggers (mcp, ai, workflow, notify, database, …); errors carry err_* fields with the stack |
| Raw SQL | DB_LOG_QUERIES=true only | sql; the Collector deletes bindings |
| Metrics | GET /server/metrics | Scraped by the Collector; see Metrics |
Position in the request pipeline
| Step | Log line | Note |
|---|---|---|
onRequest | incoming request | Before authentication: the user is unknown. Kept on stdout, dropped in the Collector. |
preHandler (authenticate) | Security events | Login, token, MFA, rate limit, … |
| Handler | Module logs, 5xx errors | |
onResponse | request completed | After authentication: carries the user and the parsed body. |
All lines of one request share the same reqId (taken from the x-request-id header when present).
The request completed line
The Fastify res serializer (requestLogSerializers in request-log.ts) writes:
res: { statusCode, method, url, route, ip, user_id, request_body? }
responseTime, reqIdurlhas credential query parameters masked (redactUrl).routeis the matched route pattern (for example/items/:collection); it is empty for requests that match no route.user_idisnullfor public requests.request_bodyis one JSON string, present only whenLOG_REQUEST_BODY=all, orerrorsand status ≥ 400, and only for JSON bodies (form strings, Buffers, streams and multipart are skipped).
This shape is a contract with otel-collector.yaml and with the OpenObserve alerts and dashboard. Change the serializer → update both.
Sensitive Data
| Data | Handled in | Result in OpenObserve |
|---|---|---|
| Password, token, secret, OTP, API key, private key, credential, email keys in the request body (any depth) | App | [REDACTED] |
authorization, cookie, password, token, secret, access_token, refresh_token, otp, api_key in app logs (top level and one level deep) | App (pino redact) | [REDACTED] |
| Authorization / cookie headers, credential query parameters | App | [REDACTED] |
Extension log data and audit details: credential keys at any depth, Error values, depth/array/string/size caps | App (sanitizeLogData) | [REDACTED], { type, message, stack }, [depth]… — see Extension Guardrails §7 |
Extension log data: user_id, email, ip, phone, target_user deeper than the top level or details | App | [REDACTED] (the Collector cannot hash them there) |
user_id, email, ip, phone, target_user at the top level and in details; res.ip, res.user_id | Collector | sha256(PII_HASH_SALT + value) |
Raw SQL bindings | Collector | Deleted |
Body sanitizing limits: depth 6, arrays 20 items, strings 256 characters, whole string LOG_REQUEST_BODY_MAX characters with a …[truncated] suffix.
Stdout keeps the original ip and user_id. To find one user's logs in OpenObserve, compute the hash with the same salt:
sh
printf '%s%s' "$PII_HASH_SALT" "<user-uuid>" | sha256sumCollector Pipeline
- Filter — drop
incoming request. - Transform — delete
bindings, hash the PII fields, copy the pino level to thelevelattribute. - Route — copy security events (core and extension audit),
error/fatallogs and the extension rate-cap lines (extension log rate cap hit,extension audit rate cap hit) to the audit stream. - Sample — keep 10% of
request completedlines with status 200–399, and 10% of extensiontrace/debuglines. Both are keyed by the OTLP trace id, so a kept request keeps all its lines. Lines without a trace (boot) pass. 4xx, 5xx and every other log are kept at 100%.
The Collector has no rate limiter for logs (contrib 0.163.1 ships none). The hard cap per extension is in the app (EXTENSION_LOG_RATE / EXTENSION_LOG_BURST).
Storage and Retention
| Stream | Content | Retention |
|---|---|---|
odp_api | All logs | 14 days (O2_RETENTION_DAYS) |
odp_api_audit | Security events, extension audit events (name = 'ext-security'), error / fatal logs and rate-cap lines (also in odp_api) | 90 days (O2_AUDIT_RETENTION_DAYS) |
odp_alert_notifications | Alert messages when no Mattermost webhook is set | 14 days |
odp_*, up (metrics) | API metrics | 14 days |
OpenObserve flattens nested fields with _ and lowercases them: res.statusCode → res_statuscode, reqId → reqid, details.email → details_email, extension.id → extension_id. Filter the pino level with level; on security events the severity column holds the event's own severity. The OTLP trace id is the trace_id column.
Searching
Open OpenObserve → Logs → choose a stream → SQL mode:
sql
-- Failed requests by route
SELECT res_route, res_statuscode, count(*) AS n FROM "odp_api"
WHERE body = 'request completed' AND res_statuscode >= 400
GROUP BY res_route, res_statuscode ORDER BY n DESC
-- Every line of one request
SELECT _timestamp, name, level, body FROM "odp_api" WHERE reqid = '<reqId>' ORDER BY _timestamp
-- Failed logins by hashed IP
SELECT ip, count(*) AS failed FROM "odp_api_audit"
WHERE event_type = 'LOGIN_FAILED' GROUP BY ip ORDER BY failed DESC
-- One extension's lines, by kind
SELECT _timestamp, extension_kind, level, body FROM "odp_api"
WHERE extension_id = '<extension-id>' ORDER BY _timestamp DESC
-- Every line of one trace (inbound traceparent or a schedule run)
SELECT _timestamp, name, extension_id, body FROM "odp_api" WHERE trace_id = '<32-hex>' ORDER BY _timestamp
-- One user's requests (hash computed as above)
SELECT _timestamp, res_method, res_route, res_statuscode FROM "odp_api"
WHERE res_user_id = '<hash>' ORDER BY _timestamp DESCThe ODP API Overview dashboard shows API up, requests/s, 5xx ratio, p95 latency, memory, failed logins, requests by status, p95 by route, security events by type, warn/error logs by logger, memory and event-loop lag, and tables of failed requests, security events and errors, plus extension log lines by extension_id and extension lines dropped by the rate cap.
Alerts
Alerts are evaluated every minute and sent to Mattermost through an incoming webhook (MATTERMOST_WEBHOOK_URL). Without a webhook, messages are written to the odp_alert_notifications stream instead.
| Alert | Fires when | Action |
|---|---|---|
odp_api_5xx_ratio_high | 5xx > 5% of requests over 5 min, at least 20 requests | Read the Errors table, filter level = 'error', check err_message |
odp_api_login_failed_burst | > 20 LOGIN_FAILED from one hashed IP in 5 min | Find the source in odp_api_audit; check for RATE_LIMIT_TRIGGERED; block if needed |
odp_api_security_critical | Any security event with severity = 'critical' in 1 min | Act at once (audit chain broken, malware, drift) |
odp_api_memory_high | RSS > 1.5 GiB for 10 min | Check the memory panel for a steady climb (leak) |
odp_api_target_down | /server/metrics cannot be scraped for 2 min | Check the API container and network |
odp_api_ext_security_critical | Any extension audit event with severity = 'critical' in 1 min (message [EXT <id>] ext.<id>.<type>) | Ask the extension's owner; the event is the extension's own claim, not a core event |
odp_api_ai_guardrail_block_spike | ≥ 20 AI guardrail refuse / replace for one app_id in 5 min (rule image not counted; shown as images). Message [AI] app <app_id>: <n> refuse/replace in 5 min (<m> unchecked images) | Read the app's guardrail lines in odp_api_audit (name = 'ai-guardrail') by guardrail_rule: an attack, or a rule that is too strict |
odp_api_ai_guardrail_unavailable | ≥ 5 AI guardrail unavailable for one integration_id in 5 min. Message [AI] decision integration <id> unavailable: <n> in 5 min | Check the decision model integration; guardrail_fail_mode says whether turns passed (open) or were refused (closed) |
The core security alerts (odp_api_login_failed_burst, odp_api_security_critical) and the core security panels count only lines without extension_id, so no extension line can fire them.
Thresholds live in docker-images/telemetry/openobserve/alerts/*.json. Edit the file, then run docker compose run --rm openobserve-init. Changes made in the OpenObserve UI are overwritten by the next init run.
Run init.sh before relying on alerts
OpenObserve refuses a query that names a column the stream has never seen (400 unknown field), and it then never fires that alert. init.sh pre-declares every column the alerts and the dashboard filter on (extension_id, event_type, trace_id…).
- Run
openobserve-initon every new install and after every upgrade of this stack (new alerts and columns come with it). It is idempotent. - On a fresh install the metrics alerts (
odp_api_5xx_ratio_high,odp_api_memory_high,odp_api_target_down) name metric streams that exist only after the Collector has scraped the API once.init.shthen skips them, still creates the log alerts and the dashboard, and exits with code3andWARN skipped alerts. Run it again once metrics arrive; the exit code is then0.
Limitations
- Only
@odp/apiis covered. The admin webapp (BFF) does not push logs yet, and it does not forward a request id or trace context to the API. - Light trace context only: every log line carries
trace_id/span_id. The only spans exported are those of the AI SDK calls, through a private tracer (nothing registered globally) and an attribute allowlist that drops prompt and response content (src/modules/telemetry/llm-telemetry.ts); they join the request ontrace_id. No other spans are exported (no per-request or per-operation timing). - OpenObserve OSS makes every user an Admin. Read-only access (Viewer role) and SSO need OpenObserve Enterprise.
AI Guardrail Audit
Each finding of the AI guardrails (warn, mask, refuse, replace, note, unavailable, unchecked_image) writes one log line on the logger ai-guardrail (aiGuardrailAudit() in src/modules/telemetry/ai-guardrail-audit.ts). It is never a row in odp_security_events and never on the logger security, so it cannot fire the core security alerts.
| Field | Value |
|---|---|
name | ai-guardrail |
category | security (copied to odp_api_audit) |
event_type | ai.guardrail |
severity | warning for refuse, replace, unavailable; info for the others |
guardrail_direction | in / out |
guardrail_rule | I1…I9, O1…O9; decision (a decision call is unavailable); image (a user image the decision model cannot check); size (a text longer than the checked windows: refuse in / replace out when closed, warn when open) |
guardrail_action | the action above |
guardrail_surface | chat / test_run / engine / agentic |
guardrail_fail_mode | open / closed |
guardrail_p | decision probability in [0, 1], 3 decimals; absent when there is none |
app_id, integration_id | ids, [A-Za-z0-9_-]{1,64}; absent when null |
user_id | user of the turn, hashed by the Collector; absent for a public turn |
| message | [AI] ai.guardrail <action> <rule> |
- Every field has a closed format, so the line cannot carry user text. An event with a field out of format is dropped, with one warning per 10 s on the logger
ai. - Rate cap 50/s, burst 500 per process. Dropped events are counted in one
ai guardrail audit rate cap hitline per window (category = 'security'). infolines are written only whenLOG_LEVELisinfoor lower.- The columns are declared by
init.sh(run it after upgrading, see above).
Extensions
Extension backend code (hook, endpoint, schedule, MCP tool) writes through context.logger and context.audit. Its lines go into the same pipeline:
| Item | Where it lands |
|---|---|
context.logger lines | odp_api, filtered by extension_id; error / fatal also copied to odp_api_audit |
context.audit.event() | odp_api_audit (and odp_api), name = 'ext-security', event_type = ext.<id>.<type>; critical fires odp_api_ext_security_critical. Never a row in odp_security_events |
| Rate-cap lines | odp_api and odp_api_audit; counter odp_extension_log_dropped_total{extension_id} on /server/metrics |
The logger contract, the reserved keys, the PII keys and the audit rules are in Extension Guardrails §7. The env keys are EXTENSION_LOG_LEVELS, EXTENSION_LOG_RATE, EXTENSION_LOG_BURST, EXTENSION_AUDIT_RATE, EXTENSION_AUDIT_CRITICAL_PER_MIN (Environment variables).
Trace Context
- Inbound. A valid W3C
traceparentgives the request itstrace_idand flags; the request gets a newspan_id, and the caller's span id isparent_span_idon the Fastify request lines. An absent or invalid header starts a new trace.x-request-id/reqIdare unchanged and stay on every line. - Lines. App logger lines (mixin) and Fastify request lines (request logger bindings) carry
trace_id,span_id,trace_flags. A schedule run gets its ownreqId(run:<uuid>) and trace. - Outbound. Core never adds
traceparentto calls to third parties (OAuth/OpenID providers, Infobip, Mattermost, LLM vendors, remote MCP servers): the trace id stays inside the trust boundary.safeFetchhas an opt-intrace: true; no core caller sets it, and auth drivers must never set it. An extension usescontext.trace.headers()only for services it owns.
Programmatic API
Exported from src/modules/telemetry/index.ts:
| Export | Purpose |
|---|---|
buildOtelTargets(level) | Pino transport targets for OTLP export; [] when off or in tests. Used by src/logger.ts and src/server.ts. |
appLogRedact | Pino redact options for the app logger. |
requestLogSerializers | Fastify logger serializer for res on the request completed line. |
sanitizeBody(body, max) | The body sanitizer, exported for tests. |
sanitizeLogData(data, options?), sanitizeDetailsData(details, options?) | Sanitizer for extension log data and audit details. |
prepareLogLine(args), prepareLogBindings(bindings), isReservedLogKey(key, value?), normalizeLogKey(key) | pino call forms → sanitized data + message; reserved keys on OpenObserve's column form. Used by src/extensions/extension-logger.ts. |
createTokenBucket(options), createRateCap(options) | Per-extension caps with one drop report per window. |
createTraceContext(inbound), outboundTraceHeaders(ctx), traceLogFields(ctx), parseTraceparent(header) | W3C trace context helpers. |
src/modules/telemetry/ext-audit.ts (createExtensionAudit) is imported directly by src/extensions, not through the index (it imports logger.ts).
Source Files
| File | Responsibility |
|---|---|
src/modules/telemetry/config.ts | Read the telemetry env, build resource attributes |
src/modules/telemetry/transport.ts | buildOtelTargets() |
src/modules/telemetry/redact.ts | Redact paths for the app logger |
src/modules/telemetry/request-log.ts | res serializer and body sanitizer |
src/modules/telemetry/sanitize.ts | One walker for bodies, extension log data and audit details |
src/modules/telemetry/log-format.ts | Call forms, printf, reserved keys |
src/modules/telemetry/rate-cap.ts | Token bucket and rate cap |
src/modules/telemetry/trace-context.ts | traceparent parse/format, outbound headers, log fields |
src/modules/telemetry/ext-audit.ts | context.audit.event() |
docker-images/telemetry/otel-collector.yaml | Collector pipeline |
docker-images/telemetry/openobserve/ | init.sh, alerts, dashboard, Mattermost template |