Connect Cordum to Datadog

Export distributed traces and metrics from Cordum to Datadog using the OpenTelemetry Collector.

:::info Two distinct Datadog paths Cordum has a native, built-in Datadog integration for the audit/SIEM event stream: set CORDUM_AUDIT_EXPORT_TYPE=datadog plus CORDUM_AUDIT_EXPORT_DD_API_KEY, CORDUM_AUDIT_EXPORT_DD_SITE, and optional CORDUM_AUDIT_EXPORT_DD_TAGS on the API gateway. That ships hash-chained governance audit events straight to Datadog Logs with no extra infrastructure — see Connect the audit stream to Datadog below.

This page's main walkthrough covers a separate, additive path: traces and Prometheus-style metrics via an OpenTelemetry Collector that you run yourself. The OTEL Collector is not built into Cordum — you deploy it alongside the stack. The two paths are independent and can run together. :::

Prerequisites

Cordum running (Docker Compose or Kubernetes)
Datadog account with an API key
Datadog site URL (e.g., datadoghq.com, datadoghq.eu)

Step 1: Enable OTEL in Cordum

Set the following environment variables on all Cordum services:

# Enable OTEL tracing and metrics
OTEL_ENABLED=true
OTEL_METRICS_ENABLED=true

# Point to the OTEL collector sidecar (or standalone collector)
OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317

# Sampling rate (0.1 = 10% of traces, adjust for your volume)
OTEL_TRACES_SAMPLER_ARG=0.1

For Docker Compose, add these to your .env file. For Kubernetes, set them in the deployment env vars or ConfigMap.

Step 2: Configure the OTEL Collector

Create or update the collector configuration to export to Datadog.

Docker Compose

Add an OTEL collector service to your docker-compose.yml:

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.115.0
    command: ["--config=/etc/otel/config.yaml"]
    ports:
      - "4317:4317"   # OTLP gRPC
    volumes:
      - ./config/otel-collector.yaml:/etc/otel/config.yaml:ro
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=${DD_SITE:-datadoghq.com}

Create config/otel-collector.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  datadog:
    api:
      key: ${DD_API_KEY}
      site: ${DD_SITE}
    traces:
      span_name_as_resource_name: true
    metrics:
      resource_attributes_as_tags: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

Kubernetes (Kustomize)

Uncomment the OTEL resources in deploy/k8s/production/kustomization.yaml:

resources:
  # ... existing resources ...
  - otel-collector-config.yaml  # uncomment this

patches:
  # ... existing patches ...
  - path: patches/otel-sidecar.yaml  # uncomment this

Update the collector ConfigMap to use the Datadog exporter (replace the default OTLP exporter section).

Step 3: Set Datadog Credentials

# Docker Compose
export DD_API_KEY="your-datadog-api-key"
export DD_SITE="datadoghq.com"  # or datadoghq.eu

# Kubernetes
kubectl create secret generic datadog-api-key \
  --from-literal=api-key="your-datadog-api-key" \
  -n cordum

Step 4: Restart and Verify

# Docker Compose
docker compose up -d

# Kubernetes
kubectl rollout restart deployment -n cordum

Verify traces in Datadog APM

Open Datadog APM > Traces
Filter by service: cordum-api-gateway, cordum-scheduler, cordum-safety-kernel
You should see traces for:
- HTTP requests to the API gateway
- Safety kernel policy evaluations
- Job dispatch and result processing

Verify metrics in Datadog Metrics Explorer

Open Datadog Metrics > Explorer
The metric names depend on which pipeline ingests them:
- Prometheus scrape (gateway :9092/metrics, scheduler :9090) uses dotless, _total-suffixed names:
  - cordum_scheduler_jobs_received_total — jobs received by topic
  - cordum_scheduler_jobs_completed_total — jobs completed by status
  - cordum_api_gateway_http_requests_total — API request rate
  - cordum_api_gateway_http_request_duration_seconds — API latency
- OTEL Collector (the path on this page) uses Cordum's OTEL instrument names, which are dot-separated and have no _total suffix:
  - cordum.scheduler.jobs_received
  - cordum.scheduler.jobs_completed
  - cordum.api_gateway.http_requests
  - cordum.api_gateway.http_request_duration
Datadog's OTLP ingestion may rewrite dots to underscores depending on your collector/exporter config — confirm the exact name in the Metrics Explorer rather than assuming the Prometheus form.

Expected Datadog Views

APM Service Map: Cordum services appear as nodes with request/error/latency metrics. The gateway connects to the safety kernel and scheduler, which connect to NATS and Redis.

Trace Waterfall: A single job submission shows: gateway HTTP handler > safety kernel policy check > scheduler dispatch > worker execution > result processing.

Metrics Dashboard: Import the pre-built Grafana dashboard (deploy/grafana/cordum-overview.json) into Datadog using the Prometheus integration, or build a Datadog dashboard using the same metric names.

Connect the audit stream (native)

Separately from the OTEL traces/metrics path above, Cordum can ship its hash-chained governance audit events to Datadog Logs directly — no collector required. This is the built-in SIEM exporter, configured on the API gateway with CORDUM_AUDIT_EXPORT_* environment variables (see docker-compose.yml and core/audit):

# On the api-gateway service:
CORDUM_AUDIT_EXPORT_TYPE=datadog
CORDUM_AUDIT_EXPORT_DD_API_KEY=your-datadog-api-key
CORDUM_AUDIT_EXPORT_DD_SITE=datadoghq.com        # or datadoghq.eu, etc.
CORDUM_AUDIT_EXPORT_DD_TAGS=env:prod,team:platform  # optional, comma-separated

The gateway refuses to start the Datadog exporter without CORDUM_AUDIT_EXPORT_DD_API_KEY. The audit Merkle chain runs unconditionally at gateway boot; CORDUM_AUDIT_EXPORT_TYPE only controls the external SIEM destination (webhook|syslog|datadog|cloudwatch, or null|discard|chain-only for chain-only with no export). These are Cordum environment variables — distinct from the DD_API_KEY / DD_SITE variables the OTEL Collector container reads in the traces/metrics path above.

Troubleshooting

Issue	Solution
No traces in Datadog	Check `OTEL_ENABLED=true` is set on all services. Verify collector logs: `docker compose logs otel-collector`
Missing metrics	Verify `OTEL_METRICS_ENABLED=true`. Check collector config has a `metrics` pipeline
High cardinality	Lower sampling rate: `OTEL_TRACES_SAMPLER_ARG=0.01` (1%)
Collector OOM	Increase memory limit or lower batch size in collector config
Existing Prometheus broken	OTEL metrics are additive. Prometheus on `:9092/metrics` should be unchanged. If missing, check the Prometheus endpoint independently

Prerequisites​

Step 1: Enable OTEL in Cordum​

Step 2: Configure the OTEL Collector​

Docker Compose​

Kubernetes (Kustomize)​

Step 3: Set Datadog Credentials​

Step 4: Restart and Verify​

Verify traces in Datadog APM​

Verify metrics in Datadog Metrics Explorer​

Expected Datadog Views​

Connect the audit stream (native)​

Troubleshooting​